[mythtv-users] tv_grab_huro regularly fails

[Hika van den Hoven]

Hoi Alistair,

Friday, January 2, 2015, 5:57:56 PM, you wrote:

> On Fri, Jan 2, 2015 at 11:59 AM, Hika van den Hoven <hikavdh at gmail.com> wrote:
>> Hoi Alistair,
>>
>> Friday, January 2, 2015, 11:28:48 AM, you wrote:
>>
>>> Hi Hika,
>>
>>> On Fri, Jan 2, 2015 at 11:10 AM, Hika van den Hoven <hikavdh at gmail.com> wrote:
>>>>> I'm currently using tv_grab_huro to populate the free-to-air listings
>>>>> in the Czech Republic, however more often than not it fails with a
>>>>> message similar to:
>>>>
>>>>> could not fetch
>>>>> http://www.port.cz/pls/tv/tv.channel?i_ch=20006&i_date=20150110&i_xday=5&i_where=1,
>>>>> error: 500 Can't connect to www.port.cz:80 (Connection timed out),
>>>>> aborting
>>>>> INFO: tv_grab exit code: 110
>>>>
>>>>> I'm not familiar with this in Czech (shepherd in Australia "just
>>>>> works"), so am not sure whether this is common behaviour or not.
>>>>
>>>>> mythfilldatabase is being run without any parameters (default configuration).
>>>>
>>>>> Can someone let me know if there is something I should do to improve
>>>>> the reliability (I'm not aware of any other internet problems here),
>>>>> or whether I should start diving in to the code to see if I can make
>>>>> this a bit more reliable (I *really* hate perl :-().
>>>>
>>>> The site www.port.cz does exist. I can open it in my browser, can you?
>>>> If not you have to look into your machines internet settings.
>>>> Do you perchance use a proxy or anything like that or the most
>>>> elementary question: 'Is the machine connected to the internet?'
>>
>>> Yep, tv_grab_huro does work sometimes (that's what I meant by "fails
>>> more often than not" above), I've been able to open www.port.cz in my
>>> browser (different machine, but same internet connection) and can curl
>>> the URL from the myth machine.
>>
>>> No proxy anywhere on the local network, and I can ssh / scp GB of data
>>> between machines without problem.  While I can't rule out problems
>>> with the internet connection entirely, I haven't had any other issues
>>> with the connection.
>>
>>> Thanks,
>>> Alistair
>>> _______________________________________________
>>
>> If you're capable of diving into the code, you could look for the
>> time-out setting and raise that. This is the time it waits for
>> response, after which it fails.

> Yep.  Based on observing the behaviour of the grabber, I assume that
> it is fetching numerous pages, and that if any one fails, it gives up.
> It would also be worthwhile having a retry on each page (with a delay
> in between).

> mythbuntu 14.04 installs tv_grab_huro v1.48, while
> http://xmltv.cvs.sourceforge.net/viewvc/xmltv/ contains a version
> 1.53.

> I'd prefer to work on the latest version.  Can anyone confirm that
> this is the correct source repository?

> Thanks again,
> Alistair
> _______________________________________________

I know nothing about your grabber, but I'm working on the dutch
grabber in python. For most sites you can grab data for one day for
one channel. So if you have 20 channels and 7 days, its 140 grabs.
With some sites (mainly those that provide json) you can grab the 7
days in one. Luckily the availability of json seems to increase. This
to provide for mobile apps.
Next some site only give the basic info and you have to do a detail
grab for every program. Basically that's when clicking on a program
opens a new page, as opposed to just showing some extra info on the
same page.
So I don't think it fails all on one fail. It could be that it simply
doesn't wait long enough to get result. And if the site is slow... The
time-out for my grabber is 10 seconds! You could try grabbing at slow
hours, say at 4 O'clock at night.

Tot mails,
  Hika                            mailto:hikavdh at gmail.com

"Zonder hoop kun je niet leven
Zonder leven is er geen hoop
Het eeuwige dilemma
Zeker als je hoop moet vernietigen om te kunnen overleven!"

De lerende Mens