[mythtv] [PATCH] DataDirect service -- updated to match CVS xmltv
tv_grab_na_dd
David Shay
david at shay.net
Sat Mar 27 00:47:50 EST 2004
Attached is an *updated* patch to enable usage of the new XMLTV grabber for
the datadirect service. Please ignore the previous patch. This patch
contains all of the same functionality as the last, but is now tied to the
CVS version of XMLTV tv_grab_na_dd script (not yet committed to an official
XMLTV release). I have been working with Robert Eden on the XMLTV list to
incorporate the changes required to the script in order to automate
configuration from the MythTV setup GUI.
A couple of notes:
The datadirect service (labs.zap2it.com) is "free", but you must signup on
their website, creating a userid and password. According to comments from
the administrators of the site, this service is not really experimental but
is a "final product" that will remain free -- the only requirement being
filling out a survey every 3 months. Their gain from this is to stem the
screen-scraping of their sites and provide a highly efficient/cached
mechanism to get the same data to non-commercial users. In fact, XMLTV will
eventually drop support of tv_grab_na and completely replace it with
tv_grab_na_dd.
There are changes to the database, in order to add a datadirect userID,
password, and lineup ID. These are directly tied to the videosource table.
During configuration, you must specify a datadirect userID and password and
then click on the "Retrieve Lineups" button. The combo box will then be
filled with the lineups that you have created on the datadirect site. When
you save the videosource, an XMLTV configuration file gets created in the
standard location. This configuration file includes your datadirect userid,
password, selected lineup, and list of channels.
mythfilldatabase has also been updated to use the new grabber and set the
number of retrieval days to 13 to support the increased amount of available
data.
Because the role, actor, and director information is now completely available,
this data is getting filled in as well, although I'm not sure where this is
really used inside Myth yet...
An overall run is significantly faster than tv_grab_na, but still longer than
I would like it to be. This is largely because the data from datadirect is
formatted in a quite different way -- i.e. much more relational-database
structured. For instance, there is a "schedule" which contains channel, time
information, and a program ID field. Then, there is a list of programs which
contain the title of the show, etc. tv_grab_na_dd essentially denormalizes
this whole structure to match that of tv_grab_na. This allows me to make
minimal changes to the parsing code, but at the loss of performance of a
complete XML transformation inside tv_grab_na_dd.
The XMLTV maintainers indicate that they are intending to add seriesID and
programID from the datadirect service, which would be of some use to the
scheduling algorithms, but they haven't indicated a date yet.
I am still considering writing a completely internal grabber. The main
challenge is that I don't have any real good examples of QHttp which include
authentication. Without a base class that provides authentication, it would
be extremely tedious to get back the requested authentication realm, build an
MD5 digest/response, etc. Anyone seen any QT code that does this? One
possibility is to shell out to a perl program to do the data grabbing part --
libwww-perl includes easy access to the authentication stuff. Isaac, would
including a perl script to perform this be acceptable as a patch, and be able
to get worked into the install mechanism, etc.?
If so, the rest of it could be done quite easily. It then really comes down
to two choices internally:
* Use memory to load up the complete list of channels, programs, schedules,
actors, etc.
* As you traverse the XML document, insert a record into a temporary database
table. You could then load the tables using a direct SQL call off of these
temporary tables.
Anyone have a preference here? I'm leaning towards the temporary tables
Feedback? Isaac?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dd.diff
Type: text/x-diff
Size: 12576 bytes
Desc: not available
Url : http://mythtv.org/pipermail/mythtv-dev/attachments/20040327/ee105776/dd.bin
More information about the mythtv-dev
mailing list