[mythtv-users] Rex: a new tv_grab_au script

Michael Cheshire (Mailing Lists) michael-mlists at cheshire.id.au
Sat Aug 12 07:06:51 UTC 2006


Max,

It was probably a log from several runs - I log it all to one file so I can 
watch over it from day to day.

Interestingly enough it seems to have issues working out a home dir variable 
when I run it from cron. Works fine when run from the command line.

In the second run below I've explicitly specified the cache file so it works 
OK.

However I am not getting data for all the foxtel channels - there are whole 
channels missing - any suggestions? FTA appears to be working fine..

Other than that - Fantastic work - its great to see that someone is still 
maintaning a ripper with good data!

(I have created a second version of rex on my system called rex_fta for the 
free to air stuff) Not sure if this is the best way to do it but it works 
for me, becuase even though the foxtel data contains all the FTA channels, 
if I use that with my FTA source id, it fills up my database with 
non-existant channels.

Notice how small the fox guide is at the bottom - it has more channels so I 
would expect it to be bigger.

Also I have just removed the channels I dont want data for from rex.conf - 
is this the correct way to do it?

(lots of detail for you below) - Cheers! :)

My crontab :

# grab foxtel channels
10 1 * * * root /usr/local/bin/rex --days 5 --config-file 
/root/.xmltv/rex.conf --output /var/local/foxguide.xml --cache-file 
/root/.rex/cache.dat >>/var/log/rex.log 2>>/var/log/rex.log
# stick em in the database
50 10 * * * root /usr/bin/mythfilldatabase --file 1 -1 
/var/local/foxguide.xml >> /var/log/tv_grab_au.log
# grab extra channels for digital tv
10 4 * * * root /usr/local/bin/rex_fta --days 5 --config-file 
/root/.xmltv/rex_fta.conf --output /var/local/ftaguide.xml --cache-file 
/root/.rex_fta/cache.dat >>/var/log/rex_fta.log 2>>/var/log/rex_fta.log
# put em in the databse
55 10 * * * root /usr/bin/mythfilldatabase --file 2 -1 
/var/local/ftaguide.xml >> /var/log/tv_grab_au.log

output from rex :(/var/log/rex.log)

can't create //.rex/cache.dat: No such file or directory at 
/usr/local/bin/rex line 278
Rex 1.2.1
Reading configuration file: /root/.xmltv/rex.conf
Unable to read cache file: //.rex/cache.dat.
Not using cache.
Grabbing data for days 1 - 5 into /var/local/foxguide.xml.
Shows found: 2019.
Saving cache.
write_programme() expects programme hashref at /usr/local/bin/rex line 315
Rex 1.2.1
Reading configuration file: /root/.xmltv/rex.conf
Retrieved 1721 cached items from file.
Grabbing data for days 1 - 5 into /var/local/foxguide.xml.
Shows found: 1985.
Saving cache.
Writing XML.

[root at mythtv local]# ls -Flah /var/local/
total 888K
drwxr-xr-x   2 root root 4.0K Aug 11 11:52 ./
drwxr-xr-x  23 root root 4.0K Jun  9 21:53 ../
-rw-r--r--   1 root root 342K Aug 12 02:57 foxguide.xml
-rw-r--r--   1 root root 519K Aug 12 04:35 ftaguide.xml

my rex.conf

$opt = {
         'payregionid' => 168,
         'days' => 5,
         'TZ' => '+0930',
         'regionid' => 81
       };
$channels = {
              'CNBC' => 'cnbc.foxtel.com.au',
              'BBC World' => 'bbcworld.foxtel.com.au',
              'FOX8+2' => 'fox82.foxtel.com.au',
              'ABC2' => 'abc2.abc.gov.au',
              'The LifeStyle Channel' => 'lifestyle.foxtel.com.au',
              'Seven' => 'channelsevenadelaide.free.au',
              'UKTV+2' => 'uktv2.foxtel.com.au',
              'The LifeStyle Channel+2' => 'lifestyle2.foxtel.com.au',
              'Channel [V]' => 'v.foxtel.com.au',
              'Sky News' => 'skynews.foxtel.com.au',
              'SBS' => 'sbssa.free.au',
              'UKTV' => 'sbssa.free.au',
              'TV1' => 'sbssa.free.au',
              'National Geographic' => 'natgeo.foxtel.com.au',
              'TV1+2' => 'tv12.foxtel.com.au',
              'Discovery Channel' => 'discovery.foxtel.com.au',
              'FOX Classics+2' => 'classics2.foxtel.com.au',
              'ABC' => 'abcsa.free.au',
              'The Weather Channel' => 'weather.foxtel.com.au',
              'Aurora' => 'aurora.foxtel.com.au',
              'MAX' => 'max.foxtel.com.au',
              'ARENA TV' => 'arena.foxtel.com.au',
              'FOX Classics' => 'classics.foxtel.com.au',
              'ARENA TV+2' => 'arena2.foxtel.com.au',
              'TCM' => 'tcm.foxtel.com.au',
              'FOX8' => 'fox8.foxtel.com.au',
              'Nickelodeon' => 'nickelodeon.foxtel.com.au',
              'FOX News' => 'foxnews.foxtel.com.au',
              'CNN' => 'cnn.foxtel.com.au',
              'Bloomberg' => 'bloomberg.foxtel.com.au',
              'SBS News' => 'news.sbs.com.au',
              'Cartoon Network' => 'cartoon.foxtel.com.au',
              'TEN' => 'networktenadelaide.free.au',
              'Nine' => 'channelnineadelaide.free.au',
              'CMC' => 'cmc.foxtel.com.au'
            };


----- Original Message ----- 
From: "Max Barry" <mythtv at maxbarry.com>
To: "Discussion about mythtv" <mythtv-users at mythtv.org>
Sent: Monday, August 07, 2006 11:21 AM
Subject: Re: [mythtv-users] Rex: a new tv_grab_au script


> Michael Cheshire wrote:
>> Max,
>>
>> Great work on the script! Its good to see that it all continues :)
>>
>> I'm getting this wierd error :
>>
>> (in regards to the cache file?)
>>
>> Rex 1.2.0
>> Unable to read cache file: /root/.rex/cache.dat.
>> Not using cache.
>> Grabbing data for days 1 - 5 into /var/local/foxguide.xml.
>> Shows found: 1540.
>> Saving cache.
>> Writing XML.
>> Rex 1.2.0 finished:
>> 1540 shows grabbed
>> 1713 downloads, including 1540 detail pages
>> 0 cache hits, 0 failed detail pages, 64 stop times adjusted
>> Time elapsed: 2 hr 46 min 38 sec
>> can't create //.rex/cache.dat: No such file or directory at
>> /usr/local/bin/rex line 277
>> Rex 1.2.0
>> Unable to read cache file: //.rex/cache.dat.
>> Not using cache.
>> Grabbing data for days 1 - 5 into /var/local/foxguide.xml.
>> Shows found: 2152.
>> ERROR! Failed to retrieve page:
>> http://www.yourtv.com.au/guide/index.cfm?action=session_info&event_id=14957760&sid=6017789&loc=grid.
>> Saving cache.
>>
>> Help?
>
> Lots of weird stuff in there. For starters, Rex has run twice, so I
> would look at what is kicking off your script. Maybe you're running it
> too often, so it doesn't get a chance to finish before the new process
> begins. Because everything up to and including "Time elapsed:" is what
> you'd expect when running Rex for the first time.
>
> Second, the first time it has run as root and the second time with no
> environment (i.e. $ENV{HOME} is not set).  I don't know how that can
> happen, but you'll have a nice cache file in /root/.rex/cache.dat which
> the second process has ignored, causing lots of unnecessary downloads.
> This also causes a failure to save the cache in the second process.
>
> Third, it's weird that the first process found 1540 subscribed shows and
> the second, which presumably ran at roughly the same time, found 2152.
> So perhaps they are being pointed at different config files. Run version
> 1.2.1 to find out.
>
> The "ERROR!" means one detail page didn't download. Not a big deal. A
> transient network error, perhaps. That shouldn't have killed the script.
>
> Despite all this you should have got your data, since none of the above
> is fatal. Although if "Saving cache." was indeed the last output you
> saw, that's weird, too.
>
> Max.
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>
> !DSPAM:44d69cd0127131605119131!
>
> 



More information about the mythtv-users mailing list