[mythtv] [PATCH] DataDirect service -- updated to match CVS xmltv tv_grab_na_dd

David Shay david at shay.net
Sat Mar 27 00:47:50 EST 2004


Attached is an *updated* patch to enable usage of the new XMLTV grabber for 
the datadirect service.  Please ignore the previous patch.  This patch 
contains all of the same functionality as the last, but is now tied to the 
CVS version of XMLTV tv_grab_na_dd script (not yet committed to an official 
XMLTV release). I have been working with Robert Eden on the XMLTV list to 
incorporate the changes required to the script in order to automate 
configuration from the MythTV setup GUI.

A couple of notes:

The datadirect service (labs.zap2it.com) is "free", but you must signup on 
their website, creating a userid and password.  According to comments from 
the administrators of the site, this service is not really experimental but 
is a "final product" that will remain free -- the only requirement being 
filling out a survey every 3 months.  Their gain from this is to stem the 
screen-scraping of their sites and provide a highly efficient/cached 
mechanism to get the same data to non-commercial users.  In fact, XMLTV will 
eventually drop support of tv_grab_na and completely replace it with 
tv_grab_na_dd.

There are changes to the database, in order to add a datadirect userID, 
password, and lineup ID.  These are directly tied to the videosource table. 

During configuration, you must specify a datadirect userID and password and 
then click on the "Retrieve Lineups" button.  The combo box will then be 
filled with the lineups that you have created on the datadirect site.  When 
you save the videosource, an XMLTV configuration file gets created in the 
standard location.  This configuration file includes your datadirect userid, 
password, selected lineup, and list of channels.

mythfilldatabase has also been updated to use the new grabber and set the 
number of retrieval days to 13 to support the increased amount of available 
data.

Because the role, actor, and director information is now completely available, 
this data is getting filled in as well, although I'm not sure where this is 
really used inside Myth yet...

An overall run is significantly faster than tv_grab_na, but still longer than 
I would like it to be.  This is largely because the data from datadirect is 
formatted in a quite different way -- i.e. much more relational-database 
structured.  For instance, there is a "schedule" which contains channel, time 
information, and a program ID field.  Then, there is a list of programs which 
contain the title of the show, etc.  tv_grab_na_dd essentially denormalizes 
this whole structure to match that of tv_grab_na.  This allows me to make 
minimal changes to the parsing code, but at the loss of performance of a 
complete XML transformation inside tv_grab_na_dd.

The XMLTV maintainers indicate that they are intending to add seriesID and 
programID from the datadirect service, which would be of some use to the 
scheduling algorithms, but they haven't indicated a date yet.

I am still considering writing a completely internal grabber. The main 
challenge is that I don't have any real good examples of QHttp which include 
authentication.  Without a base class that provides authentication, it would 
be extremely tedious to get back the requested authentication realm, build an 
MD5 digest/response, etc.  Anyone seen any QT code that does this?  One 
possibility is to shell out to a perl program to do the data grabbing part -- 
libwww-perl includes easy access to the authentication stuff.  Isaac, would 
including a perl script to perform this be acceptable as a patch, and be able 
to get worked into the install mechanism, etc.?

If so, the rest of it could be done quite easily.  It then really comes down 
to two choices internally: 
* Use memory to load up the complete list of channels, programs, schedules, 
actors, etc.
* As you traverse the XML document, insert a record into a temporary database 
table.  You could then load the tables using a direct SQL call off of these 
temporary tables.
Anyone have a preference here?  I'm leaning towards the temporary tables

Feedback?  Isaac?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dd.diff
Type: text/x-diff
Size: 12576 bytes
Desc: not available
Url : http://mythtv.org/pipermail/mythtv-dev/attachments/20040327/ee105776/dd.bin


More information about the mythtv-dev mailing list