[mythtv-users] mythfilldatabase problems

Wed Aug 31 16:19:06 UTC 2011

I recently moved my mythtv setup from Time-Warner cable to Dish Network 
(we moved out into the country).

With the Dish Network lineup configured on Schedules Direct, I get 
mythfilldatabase failures. --dd-grab-all never works (timeouts). 
Individual daily grabs sometimes work, but fail more often than not. 
Confusingly, some daily grabs seem to have worked, but a subsequent 
mythfilldatabase run claims that no data is present and repeats the grab 
again (which still doesn't seem to have worked).

Here is a log from a seemingly successful grab (I set 
--dont-refresh-tomorrow to bypass that 20-minute grab that had 
previously been done):

**** Start of first log *****

$ mythfilldatabase --dont-refresh-tomorrow
2011-08-31 10:25:08.551 Using runtime prefix = /usr
2011-08-31 10:25:08.551 Using configuration directory = 
/home/dwilson/.mythtv
2011-08-31 10:25:08.551 Unable to read configuration file mysql.txt
2011-08-31 10:25:08.551 Empty LocalHostName.
2011-08-31 10:25:08.551 Using localhost value of dan
2011-08-31 10:25:08.590 New DB connection, total: 1
2011-08-31 10:25:08.597 Connected to database 'mythconverg' at host: 
localhost
2011-08-31 10:25:08.604 Closing DB connection named 'DBManager0'
2011-08-31 10:25:08.605 Connected to database 'mythconverg' at host: 
localhost
2011-08-31 10:25:08.614 Current locale EN_US
2011-08-31 10:25:08.614 Reading locale defaults from 
/usr/share/mythtv//locales/en_us.xml
2011-08-31 10:25:08.632 Loading en_us translation for module mythfrontend
2011-08-31 10:25:08.634 Current MythTV Schema Version (DBSchemaVer): 1264
2011-08-31 10:25:08.637 New DB connection, total: 2
2011-08-31 10:25:08.637 Connected to database 'mythconverg' at host: 
localhost
2011-08-31 10:25:08.878 Updating source #1 (Dish Network) with grabber 
schedulesdirect1
2011-08-31 10:25:08.879 Found 519 channels for source 1 which use grabber
2011-08-31 10:25:08.879
2011-08-31 10:25:08.879 Checking day @ offset 0, date: Wed Aug 31 2011
2011-08-31 10:25:08.984 Data is already present for Wed Aug 31 2011, 
skipping
2011-08-31 10:25:08.984
2011-08-31 10:25:08.984 Checking day @ offset 1, date: Thu Sep 1 2011
2011-08-31 10:25:09.079 Data is already present for Thu Sep 1 2011, skipping
2011-08-31 10:25:09.079
2011-08-31 10:25:09.079 Checking day @ offset 2, date: Fri Sep 2 2011
2011-08-31 10:25:09.174 Data is already present for Fri Sep 2 2011, skipping
2011-08-31 10:25:09.174
2011-08-31 10:25:09.174 Checking day @ offset 3, date: Sat Sep 3 2011
2011-08-31 10:25:09.265 Data Refresh needed because offset day 3 has 
less than 3 programs per channel for the 8PM - midnight time window for 
channels that normally have data. We want at least 1545 programs, but 
only found 1444
2011-08-31 10:25:09.265 Refreshing data for Sat Sep 3 2011
2011-08-31 10:25:09.265 New DB DataDirect connection
2011-08-31 10:25:09.266 Connected to database 'mythconverg' at host: 
localhost
2011-08-31 10:25:09.267 Retrieving datadirect data.
2011-08-31 10:25:09.267 Grabbing data for Wed Aug 31 2011 offset 3
2011-08-31 10:25:09.267 From Sat Sep 3 05:00:00 2011 to Sun Sep 4 
05:00:00 2011 (UTC)
2011-08-31 10:25:09.267 Grabbing listing data
--2011-08-31 10:25:09-- 
http://webservices.schedulesdirect.tmsdatadirect.com/schedulesdirect/tvlistings/xtvdService
Resolving webservices.schedulesdirect.tmsdatadirect.com... 144.142.232.53
Connecting to 
webservices.schedulesdirect.tmsdatadirect.com|144.142.232.53|:80... 
connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to 
webservices.schedulesdirect.tmsdatadirect.com|144.142.232.53|:80... 
connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/xml]
Saving to: “STDOUT”

[ <=> ] 38,718 8.25K/s 2011-08-31 10:25:15.254 New DB connection, total: 3
2011-08-31 10:25:15.255 Connected to database 'mythconverg' at host: 
localhost
2011-08-31 10:25:15.257 DataDirect: Your subscription expires on Wed Sep 
28 11:23 PM
[ <=> ] 60,818 9.49K/s 2011-08-31 10:25:44.353 sourceid 1 has lineup 
type: Satellite
[ <=> ] 501,728 815B/s in 10m 16s

2011-08-31 10:37:38 (815 B/s) - written to stdout [501728]

2011-08-31 10:43:31.413 Grab complete. Actual data from Sat Sep 3 
05:00:00 2011 to Sun Sep 4 05:00:00 2011 (UTC)
2011-08-31 10:43:31.417 Main temp tables populated.
2011-08-31 10:43:31.417 Updating MythTV channels.
2011-08-31 10:43:31.578 IconData: Updating icons for sourceid: 1
2011-08-31 10:43:31.580 Channels updated.
2011-08-31 10:43:32.158 Clearing data for source.
2011-08-31 10:43:32.158 Clearing from Sat Sep 3 00:00:00 2011 to Sun Sep 
4 00:00:00 2011 (localtime)
2011-08-31 10:43:36.174 Data for source cleared.
2011-08-31 10:43:36.174 Updating programs.
2011-08-31 10:43:41.116 Program table update complete.
2011-08-31 10:43:41.116

**** End of First log ****

Here is an example of a timeout:

**** Start of Second Log ****

2011-08-31 10:43:41.116 Checking day @ offset 4, date: Sun Sep 4 2011
2011-08-31 10:43:41.183 Data refresh needed because only 0 out of 519 
channels have at least one program listed for day @ offset 4 from 8PM - 
midnight. Previous day had 515 channels with data in that time period.
2011-08-31 10:43:41.183 Refreshing data for Sun Sep 4 2011
2011-08-31 10:43:41.184 Retrieving datadirect data.
2011-08-31 10:43:41.184 Grabbing data for Wed Aug 31 2011 offset 4
2011-08-31 10:43:41.184 From Sun Sep 4 05:00:00 2011 to Mon Sep 5 
05:00:00 2011 (UTC)
2011-08-31 10:43:41.184 Grabbing listing data
--2011-08-31 10:43:41-- 
http://webservices.schedulesdirect.tmsdatadirect.com/schedulesdirect/tvlistings/xtvdService
Resolving webservices.schedulesdirect.tmsdatadirect.com... 144.142.232.53
Connecting to 
webservices.schedulesdirect.tmsdatadirect.com|144.142.232.53|:80... 
connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to 
webservices.schedulesdirect.tmsdatadirect.com|144.142.232.53|:80... 
connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/xml]
Saving to: “STDOUT”

[ <=> ] 38,720 7.85K/s 2011-08-31 10:43:47.010 DataDirect: Your 
subscription expires on Wed Sep 28 11:23 PM
[ <=> ] 237,994 --.-K/s in 19m 47s

2011-08-31 11:03:29 (200 B/s) - Read error at byte 237994 (Connection 
timed out).Retrying.

--2011-08-31 11:03:30-- (try: 2) 
http://webservices.schedulesdirect.tmsdatadirect.com/schedulesdirect/tvlistings/xtvdService
Connecting to 
webservices.schedulesdirect.tmsdatadirect.com|144.142.232.53|:80... 
connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to 
webservices.schedulesdirect.tmsdatadirect.com|144.142.232.53|:80... 
connected.
HTTP request sent, awaiting response... 200 OK

The file is already fully retrieved; nothing to do.

gzip: stdin: unexpected end of file
2011-08-31 11:03:35.703 DataDirect, Error: DataDirect XML failed to 
properly parse, downloaded listings were probably corrupt.
2011-08-31 11:03:35.716 Encountered error in grabbing data.
2011-08-31 11:03:35.716

**** End of Second Log *****

Whenever I get the timeout, the retry always fails, claiming that the 
file was already completely downloaded when that is clearly not the case.

I wish there was a way to force mythfilldatabase to only download a 
single day. It seems that even if you use --refresh-day 4, for instance, 
mythfilldatabase will look at day 3, decide it needs that day's data, 
and goes to get it. For diagnosis of problems, it would be great to be 
able to force mythfilldatabase to only do what was asked, not try to 
decide on it's own what is needed.

Any ideas on what I could do to fix these grab failures?

Thanks!

Dan.