[mythtv-users] UK - my listings are disappearing

[Mike Perkins]

On 02/11/11 15:34, Nick Morrott wrote:
> On 2 November 2011 15:08, Mike Perkins<mikep at randomtraveller.org.uk>  wrote:
>> I have realised that I am no longer getting listings updates from the Radio
>> Times feed (via xmltv). I have tried to fault find the problem but without much
>> success.
>>
>> If I run tv_grab_uk_rt manually it appears to succeed, but the output file only
>> contains partial schedule data. If I then attempt to load this using
>> mythfilldatabase /nothing/ changes, probably because the data replicates what is
>> already there from when it worked.
>>
>> I can't ping xmltv.radiotimes.com, which means I can't even check if the GETs
>> are being intercepted by my firewall.
>>
>> If I attempt to web browse to xmltv.radiotimes.com I get a web page which notes:
>>
>>
>> Old RadioTimes.com
>>
>> Please note we are only updating listings on this site until 30th Sept
>>
>>
>> Any ideas?
>
> The grabber is working fine here, and the Radio Times currently
> carries data out to 2011-11-16. No issues are reported on the nightly
> XMLTV validator either:
>
> http://debian.crustynet.org.uk/~xmltv-tester/squeeze/release/
>
> Run the grabber manually with debugging enabled (pass the --debug
> option) and capture the debug output to STDERR. This should show what
> is happening and reveal whether fresh data is being received from
> radiotimes.com or whether cached data is being used (which will
> eventually run out).
>
I have solved this one, and I thought that I should post my conclusions in case 
someone else falls down a similar hole someday.

Apologies for the delay, Real Life intervened and I wasn't able to devote a lot 
of time to it until yesterday.

I did as Nick suggested, and ran the grabber as specified. I have two sources 
called "Freeview" and "Cable". The "Cable" one is a subset of the available 
(Virgin Media) channels, with all the tat and the HD edited out (I can't do HD 
yet). The only item of note in the logs was:

[5] Will check for inconsistent title 'Have I Got News For You?'
Wrong number of fields in XMLTV prog_titles_to_process entry:
         5|Hay Sessions 2011~2|Hay Sessions
[5] Will check for inconsistent title 'Hollyoaks Music Show'

Needless to say, this wasn't relevant to my problem, which turned out to be in 
two distinct parts.

Firstly, we suffered a power cut two weeks ago for an hour, and I carefully 
brought up all my servers afterwards under manual control to ensure I had no 
disk corruption, and all seemed normal.

When I subsequently ran the grabber it pulled all the listings from cache, not 
once going to the website. However, I noticed that the "Freeview" source 
completed successfully but the "Cable" run didn't, it stopped abruptly after 
"Filmflex Preview Channel", all following channels (in rough alphabetic order) 
were completely blank. I deduced from this that the cache was bad, nuked the 
directory and re-ran.

Nothing. I got an "Error 256" from the grabber. Checking further, I discovered 
that my firewall had blocked the website. I unblocked it and everything ran to 
completion. 14 days listings again!

Problem 2: About three weeks ago I upgraded my firewall, to pfSense 2.0 
(Release) with Snort 2.9. It was a snort rule that was tripping me up:

snort[6943]: [1:3079:9] WEB-CLIENT Microsoft ANI file parsing overflow 
[Classification: Attempted User Privilege Gain] [Priority: 1] {TCP} 
78.136.19.14:80 -> x.x.x.x:30072

WTF? I'm calling from a Linux box, stupid, let my data through!

Every time I would reset the block and the grabber would run, it would trip 
again. Whitelisting the Radio Times website address made *no difference*. In the 
end I was forced to disable that specific rule (3079).

Moral is: check your firewall rules if you encounter grabber problems.

This /only/ happens for the "Cable" source. Is there something specific in that 
file which could trip that rule? Does anybody know or care?

-- 

Mike Perkins