[mythtv-users] tv_grab_na_dd backport data to .14 system?

[William Powers]

Jeff C wrote:
> I was wondering if anyone has written a filter to take the zap2it data
> direct listings and back port them in to a .14 system.  I run KnoppMyth and
> would like to stay on it as everything is stable and functioning well (and
> my wifi card isn't supported in newer kernels as the driver is closed
> source).

I think I may have something working for me.  Actually, I just got it 
working so I may still find more problems as I go along, but so far the 
only problem I've seen are some minor format glitches on MythWeb.

I pieced this together from other postings to this list and from the 
tv_grab_na_dd code.  There's nothing new here; I didn't invent any of 
it.  I'm simply posting it all here in one place in the hope that it 
might help tide people over who don't want to run from CVS until the 
official 0.15 release comes out.  If anyone has corrections, 
suggestions, improvements, etc., just post them as replies.  If you are 
reading this sometime in the future, I suggest reading the entire chain 
before implementing to make sure you get the best version.

Note: The following assumes you are building XMLTV from source.  If 
you're getting it via ATRPMS, then skip steps 3 and 4 and just apt-get 
the newest version of XMLTV.

1)  Register at http://labs.zap2it.com using code 'TGYM-ZKOC-BUTV'
     (Letter O, not zero)

2)  Record your new username and password and choose the proper
     lineup and channels.

3)  Update/install all of the XMLTV prereq's:

     $ perl -MCPAN -e shell
     cpan> install XML::Twig
     cpan> install Date::Manip
     cpan> install LWP
     cpan> install XML::Writer
     cpan> install SOAP::Lite
     cpan> exit

4)  Download, unpack, build, and install the lastest version of XMLTV.
     Be sure to build tv_grab_na_dd.

5)  Compute your local offset from GMT (UTC), ie:  -0400 from GMT to EDT

6)  With username, password, and GMT offset in hand, and while logged
     in as the same user that will make the datadirect requests, do:

     $ tv_grab_na_dd --configure

     and answer the questions.

7)  Execute the following parts of (the soon to be published)
     section 21.19 of the HOWTO, "Migrating from XMLTV to
     DataDirect".  Specifically (all one line):

     $ mysqldump -u mythtv -pmythtv mythconverg -c >
       /tmp/mythtv_backup.sql

     Look at mythtv_backup.sql and verify it is good before proceeding.

     $ mysql -u root mythconverg

     mysql>delete from program;
     mysql>delete from channel;
     mysql>truncate table videosource;
     mysql>exit

     Don't bother rerunning the MythTV setup program if you are
     using 0.14 or previous.

8)  Install the following script in place of 'mythfilldatabasecron'
     or whatever you use to pull down the listings.  Make sure that
     it is run under the same userid as did the configuration in
     step 6.  Otherwise it won't be able to find the configuration
     file.

     Note:  This script contains bits and pieces stolen from a lot
     of different places, but most of it is from from Mike Dean's
     post.  If you have multiple lineups to deal with, refer back
     to his post for a method of handling them.  There is also an
     XMLTV utility called tv_cat that can be used to concatenate
     multiple files together after downloading them separately.
     Check the XMLTV source for details.

     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     #!/bin/sh
     #/usr/local/bin/mythfilldatabasecron_dd

     # temporary filename
     dirname='/tmp'
     filename="tv-${RANDOM}${RANDOM}"
     xml_file=${dirname}/${filename}.xml

     # Should be 1 after step 7...
     sourceid=1

     # replace all data
     offset=-1

     # How many days worth of guide data to download
     numdays=10

     /usr/bin/tv_grab_na_dd --output ${xml_file} --days ${numdays} && \
     /usr/bin/mythfilldatabase --file ${sourceid} ${offset} ${xml_file}

     # Remove the temporary file
     rm ${xml_file}
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you're happy with the results, don't ever run tv_grab_na again.

However, if the whole thing blows up in your face and you want to back 
out of it, execute section 21.6 of the HOWTO to drop the database, and 
then run the restore database part of section 21.5.

I encountered one very funky problem that is probably due to a unique 
glitch in my system and likely won't bother anyone else.  However, just 
in case, I'll mention it.  Initially, tv_grab_na_dd returned four days 
worth of guide data no matter how much was asked for, be it 0 or 10 or 
whatever.  Inspection of the downloaded dd data revealed the following 
message:

<message>The endTime you specified -05-17T05:00:00Z is not in a valid 
format, and has been corrected to a default value.</message>
<message>The endTime you specified was invalid and has been corrected to 
2004-05-12T05:00:00Z.</message>

For some reason, the year is missing off of the front of the endTime. 
The relevant line in tv_grab_na_dd is:

$dd_data->print($soap->download("<startTime>$dd_start</startTime><endTime>$dd_stop</endTime>"));

Printing the contents of $dd_stop revealed a perfectly formed stop time, 
so there's some problem in the module that handles the transmission. 
However, just for grins, I changed that line to read:

$dd_data->print($soap->download("<startTime>$dd_start</startTime><endTime>2004$dd_stop</endTime>"));

After that, everything worked properly for me.  I have no idea why.  If 
I have time, I'll look into it, but hopefully 0.15 will be out before 
2005 rolls around...