[mythtv-users] tv_grab_na_dd

William Powers wepprop at sbcglobal.net
Wed May 5 23:42:39 EDT 2004

If someone would be kind enough to post a checklist or a mini-HowTo on 
how to migrate from tv_grab_na to tv_grab_na_dd for those running 0.14 
(or previous, I suppose) I for one would be very glad to do so.

I have already conclusively demonstrated in the past that running CVS is 
not a good choice for me.

I tried (more than once) to migrate over using the script given below. 
I was able to get guide data downloaded and reprocessed, but the 
mythfilldatabase step resulted in beaucoup mysql errors and only 4 days 
worth of uncertain guide data.  That was enough cause for me to restore 
the database from backup and consider my next move.

Here is what I have done so far:

1)  Registered at http://labs.zap2it.com using code 'TGYM-ZKOC-BUTV'
     (Letter O, not zero)

2)  Recorded my new username and password.

3)  Entered my zip code and then made sure I got the same channel
     selections I was using with tv_grab_na

4)  Computed my offset from GMT (UTC) as -0500 to CDT.

5)  Updated/installed XML::Writer and SOAP::Lite from CPAN.

6)  Downloaded, unpacked, compiled and installed the latest version of
     XMLTV including tv_grab_na_dd

7)  With username, password and GMT offset at hand, did:
     > tv_grab_na_dd --configure

8)  Backed up the database per section 21.5 of the HOWTO

9)  Installed Mike's script (below) as:

10) Temporarily commented out the last two steps in the script
     (the two 'rm' steps)

11) Ran the script by hand and examined the results

Suggestions would be very much appreciated.


Michael T. Dean wrote:

> It's not a drop-in replacement.  It uses an XML web service instead of 
> screen scraping HTML pages (which makes it extremely fast--i.e. 5 
> seconds for a day's listings instead of 2 1/2 minutes), so it was 
> redesigned for the service.  Therefore, it has different command-line 
> options.
> Also, and more importantly for Myth users, it uses different channel 
> ID's, so either a) the database needs to be updated to use the new 
> channel ID's or b) the "--old-chan-id" command-line option must be 
> specified (I don't know if the option will be supported indefinitely, 
> though, and I don't know which is being used for the CVS version of 
> MythTV).  Also, when configuring the grabber, you must specify a numeric 
> timezone offset (i.e. -0400 for US Eastern Daylight Time), username, 
> password, and lineup; and you don't have to specify a provider (that's 
> done through the website when signing up for the account).
> Doing a mythfilldatabase without "--old-chan-id" will create "near" 
> duplicates of all channels (and the duplicates won't be associated with 
> any sources/recordings).  Also, if the timezone offset is incorrect, the 
> shows will appear in the EPG at the wrong times.
> We should probably get something up that helps MythTV users who haven't 
> upgraded to CVS (or who may decide not to immediately upgrade to 0.15 
> when released) start using the new grabber because Tribune Media is 
> "anxious" to stop the screen scraping that's eating up their bandwidth 
> (and were kind enough to supply something better instead of shutting out 
> XMLTV users).  They've already asked the XMLTV developers why people are 
> still using the old grabber, and notified them that some changes that 
> they need to make to the site in the very near future (new data) will 
> break the old grabber.  The XMLTV developers will not fix the old 
> grabber since it is no longer supported.  Therefore, depending on the 
> timing of this change and the release of 0.15, many Myth users may get 
> caught unaware.
> To get things rolling, I'm using the below script to update my 
> listings.  It's a good start, but there's plenty of room for 
> improvement.  Note that the script does not take into account multiple 
> DataDirect lineups or multiple MythTV source ID's and it always 
> downloads/updates all the available data (which is about 2.5MB total for 
> my 16 channels) instead of using "--days" and "--offset" options (like 
> it should).  Oh, and it's vulnerable to a symlink attack thanks to the 
> totally predictable temporary filenames.  :)
> Finally, we'll want to tell everyone that the new service is extremely 
> fast (WIIFM) and assure them that it is still available at no charge.  
> The three-month subscription will be renewed by filling out another 
> survey (similar to the one filled/fill out during registration).
> Mike
> #!/bin/sh
> # temporary filenames
> dirname='/tmp'
> filename="tv-`date +%Y%m%d`"
> dd_file=${dirname}/${filename}.dd
> xmltv_file=${dirname}/${filename}.xmltv
> # use card input 1
> sourceid=1
> # replace all data, up to 10 days
> offset=-1
> # Save the downloaded DataDirect listings to a file
> #  - This file contains the data for all lineups and may be
> #    "reprocessed" multiple times (so there is no need to re-download
> #    for different lineups)
> tv_grab_na_dd --dd-data ${dd_file} --download-only
> # Process the downloaded DataDirect listings and output XMLTV listings
> tv_grab_na_dd --dd-data ${dd_file} --reprocess \
>              --output ${xmltv_file} --old-chan-id
> # If using multiple lineups, create a separate configuration file for
> # each one and specify the appropriate configuration file for the
> # lineup such as:
> #tv_grab_na_dd --config-file=lineup1.dat \
> #              --dd-data ${dd_file} --reprocess \
> #              --output ${xmltv_file} --old-chan-id
> # Fill the database with the downloaded data
> mythfilldatabase --file ${sourceid} ${offset} ${xmltv_file}
> # Remove the temporary files
> rm ${xmltv_file}
> rm ${dd_file}

More information about the mythtv-users mailing list