[mythtv-users] UK - my listings are disappearing

[Nick Morrott]

On 8 November 2011 10:58, Mike Perkins <mikep at randomtraveller.org.uk> wrote:
> On 02/11/11 15:34, Nick Morrott wrote:
>> On 2 November 2011 15:08, Mike Perkins<mikep at randomtraveller.org.uk>  wrote:
>>> I have realised that I am no longer getting listings updates from the Radio
>>> Times feed (via xmltv). I have tried to fault find the problem but without much
>>> success.
>>>
>>> If I run tv_grab_uk_rt manually it appears to succeed, but the output file only
>>> contains partial schedule data. If I then attempt to load this using
>>> mythfilldatabase /nothing/ changes, probably because the data replicates what is
>>> already there from when it worked.
>>>
>>> I can't ping xmltv.radiotimes.com, which means I can't even check if the GETs
>>> are being intercepted by my firewall.
>>>
>>> If I attempt to web browse to xmltv.radiotimes.com I get a web page which notes:
>>>
>>>
>>> Old RadioTimes.com
>>>
>>> Please note we are only updating listings on this site until 30th Sept
>>>
>>>
>>> Any ideas?
>>
>> The grabber is working fine here, and the Radio Times currently
>> carries data out to 2011-11-16. No issues are reported on the nightly
>> XMLTV validator either:
>>
>> http://debian.crustynet.org.uk/~xmltv-tester/squeeze/release/
>>
>> Run the grabber manually with debugging enabled (pass the --debug
>> option) and capture the debug output to STDERR. This should show what
>> is happening and reveal whether fresh data is being received from
>> radiotimes.com or whether cached data is being used (which will
>> eventually run out).
>>
> I have solved this one, and I thought that I should post my conclusions in case
> someone else falls down a similar hole someday.
>
> Apologies for the delay, Real Life intervened and I wasn't able to devote a lot
> of time to it until yesterday.
>
> I did as Nick suggested, and ran the grabber as specified. I have two sources
> called "Freeview" and "Cable". The "Cable" one is a subset of the available
> (Virgin Media) channels, with all the tat and the HD edited out (I can't do HD
> yet). The only item of note in the logs was:
>
> [5] Will check for inconsistent title 'Have I Got News For You?'
> Wrong number of fields in XMLTV prog_titles_to_process entry:
>         5|Hay Sessions 2011~2|Hay Sessions
> [5] Will check for inconsistent title 'Hollyoaks Music Show'

Fixed (there was an extraneous "2|" in the fixup).

> Needless to say, this wasn't relevant to my problem, which turned out to be in
> two distinct parts.
>
> Firstly, we suffered a power cut two weeks ago for an hour, and I carefully
> brought up all my servers afterwards under manual control to ensure I had no
> disk corruption, and all seemed normal.
>
> When I subsequently ran the grabber it pulled all the listings from cache, not
> once going to the website. However, I noticed that the "Freeview" source
> completed successfully but the "Cable" run didn't, it stopped abruptly after
> "Filmflex Preview Channel", all following channels (in rough alphabetic order)
> were completely blank. I deduced from this that the cache was bad, nuked the
> directory and re-ran.
>
> Nothing. I got an "Error 256" from the grabber. Checking further, I discovered
> that my firewall had blocked the website. I unblocked it and everything ran to
> completion. 14 days listings again!
>
> Problem 2: About three weeks ago I upgraded my firewall, to pfSense 2.0
> (Release) with Snort 2.9. It was a snort rule that was tripping me up:
>
> snort[6943]: [1:3079:9] WEB-CLIENT Microsoft ANI file parsing overflow
> [Classification: Attempted User Privilege Gain] [Priority: 1] {TCP}
> 78.136.19.14:80 -> x.x.x.x:30072
>
> WTF? I'm calling from a Linux box, stupid, let my data through!
>
> Every time I would reset the block and the grabber would run, it would trip
> again. Whitelisting the Radio Times website address made *no difference*. In the
> end I was forced to disable that specific rule (3079).
>
> Moral is: check your firewall rules if you encounter grabber problems.
>
> This /only/ happens for the "Cable" source. Is there something specific in that
> file which could trip that rule? Does anybody know or care?

Glad you've worked out where the problem was/is.

After a little detective wok, I think I may have worked out why Snort
was blocking the grabber run (and triggering a false positive block):

i) Looking at http://isc.sans.edu/diary.html?storyid=2540 suggests
that the Snort rule causing the block has contents:

alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"WEB-CLIENT
Microsoft ANI file parsing overflow"; flow:established,from_server;
content:"RIFF"; nocase; content:"anih"; nocase;
byte_test:4,>,36,0,relative,little; reference:cve,2004-1049;
classtype:attempted-user; sid:3079; rev:3;)

ii) As you mentioned that the run stopped immediately after retrieving
the Filmflex Previews listings, I grabbed a copy:

http://xmltv.radiotimes.com/xmltv/2505.dat

iii) The Snort rule above is looking for content that includes the
string "anih" - looking through the Filmflex listings reveals an entry
for the film "Saint" which features actress Escha Tanihatu.

iv) My theory (as I have no knowledge of how Snort rules work fully)
therefore is that the listings data for Filmflex is parsed as it comes
in across the wire, triggers the Snort rule due to the presence of
"Tanihatu", and blocks all future retrievals from
xmltv.radiotimes.com.


To remedy this situation, remove (or comment out) the Filmflex entry
in your tv_grab_uk_rt config file and retry your regular grabber runs.

Additionally, as this appears to be a genuine false positive
detection, should it be reported to the Snort team?

Cheers,
Nick

-- 
Nick Morrott

MythTV Official wiki: http://mythtv.org/wiki/
MythTV users list archive: http://www.gossamer-threads.com/lists/mythtv/users

"An investment in knowledge always pays the best interest." - Benjamin Franklin