[mythtv-users] Schedules Direct DataDirect replacement service testing

[Robert Eden]

On 10/18/2014 3:43 PM, Jay Foster wrote:
> On 10/17/2014 08:43 PM, Bill Meek wrote:
>> On 10/17/2014 12:29 PM, Jay Foster wrote:
>> ...
>> > I seem to be having problems with the SD listings.  When I run MFDB, I see the following:
>>>
>>> Resolving webservices.schedulesdirect.tmsdatadirect.com... 54.85.117.227
>>> Connecting to webservices.schedulesdirect.tmsdatadirect.com|54.85.117.227|:80... connected.
>>> HTTP request sent, awaiting response... 401 Authorization Required
>>> Connecting to webservices.schedulesdirect.tmsdatadirect.com|54.85.117.227|:80... connected.
>>> HTTP request sent, awaiting response... 402 Account Expired
>>> 2014-10-17 10:19:26 ERROR 402: Account Expired.
>>>
>>> 2014-10-17 10:19:26.981 DataDirect, Error: Failed to save DD cache! redownloading data...
>>> --2014-10-17 10:19:26-- http://webservices.schedulesdirect.tmsdatadirect.com/schedulesdirect/tvlistings/xtvdService
>>>
>>> My account is NOT expired (confirmed on the SD website).  I have not seen this happen with the previous service.  Looks like something odd going
>>> on.
>> ...
>>
>> I believe you were/are running 0.21.
>>
>> Note that in 0.25 (Apr. 2012), MythTV began using the internal
>> MythDownloadManager. It no longer uses the wget command to
>> download SD files (and many others.) There's also discussions
>> on the SD forum about similar problems:
>>
>> http://forums.schedulesdirect.org/viewtopic.php?f=8&t=2599&start=20#p7888
>>
>> Maybe trying: man wget and searching for --http-passwd will explain
>> what type of authentication is supported. My wget uses --http-password
>> and will encode the response in either digest or basic format depending
>> on the challenge issued. Currently, the report is that only basic is
>> being used.
>>
> I think I was not clear (my bad).  I am not ALWAYS getting 402 Account Expired, just sometimes within the same MFDB run.  Most MFDB runs 
> authenticate each connection successfully.
>
> I am sometimes seeing a different issue too.  Most MFDB connections fetch  about 200K bytes of data.  For example:
>
> HTTP request sent, awaiting response... 200 OK
> Length: unspecified [text/xml]
> Saving to: `STDOUT'
>
>      0K .......... .......... .......... .......... .......... 120K
>     50K .......... .......... .......... .......... .......... 135K
>    100K .......... .......... .......... .......... .......... 39.9K
>    150K .......... .......... .......... .......... .......... 96.8K
>    200K .......... .......... ...... 92.1K=2.8s
>
> 2014-10-14 02:33:18 (79.6 KB/s) - `-' saved [231515]
>
> 2014-10-14 02:33:18.089 DataDirect: Your subscription expires on 03/22/2015 01:59:05 PM
> 2014-10-14 02:33:20.477 Grab complete.  Actual data from Sat Oct 25 07:00:00 2014 to Sun Oct 26 07:00:00 2014 (UTC)
>
> I have seen the following 500 Internal Server Error too:
>
> HTTP request sent, awaiting response... 401 Authorization Required
> Connecting to webservices.schedulesdirect.tmsdatadirect.com|54.85.117.227|:80... connected.
> HTTP request sent, awaiting response... 500 Internal Server Error
> 2014-10-14 02:33:26 ERROR 500: Internal Server Error.
>
> 2014-10-14 02:33:26.547 DataDirect, Error: Failed to save DD cache! redownloading data...
> --2014-10-14 02:33:26-- http://webservices.schedulesdirect.tmsdatadirect.com/schedulesdirect/tvlistings/xtvdService
>
> But sometimes (usually the last connection?) it gets something strange, such as:
>
> HTTP request sent, awaiting response... 200 OK
> Length: 5022 (4.9K) [text/xml]
> Saving to: `STDOUT'
>
>      0K ....                                                  100% 343M=0s
>
> 2014-10-14 02:33:35 (343 MB/s) - `-' saved [5022/5022]
>
> 2014-10-14 02:33:35.114 DataDirect: Your subscription expires on 03/22/2015 01:59:05 PM
> 2014-10-14 02:33:35.158 Grab complete.  Actual data from Mon Oct 27 07:00:00 2014 to Tue Oct 28 07:00:00 2014 (UTC)
> 2014-10-14 02:33:35.172 Main temp tables populated.
> 2014-10-14 02:33:35.173 Did not find any new program data.
>
> This may also be a reason for the "Failed to fetch some program info" message at the end.  Note that this download HAS a reported Length field 
> (5022) in the HTTP header (seems a bit short for listings data).
>
> I see that most of the HTTP responses do not have a Length field in the header (Length: unspecified [text/xml]).  I seem to recall RobertE stating 
> that he needed to gather the listings data together first in order to fill in the Length header, and that is why there was an additional delay with 
> the replacement service. Has that changed?  I AM getting listings data okay, or as far as I can tell. Just pointing out some anomalies going on 
> behind the scenes in the logs.

I squished a pretty big authentication bug last night.  Hopefully that solves the problem.
I don't actually set the headers directly, I let CGI do it, but I don't see how the length header could be set because it deson't know how long the 
data will be when the headers have to go out.

Robert