[mythtv-users] mysql errors, logging stopped, then lost all recordings

[David Lasker]

I am running a combined FE/BE using stock Mythbuntu 0.23+fixes 24158. It has
been running perfectly for about 2 months. Then yesterday, I noticed that
all of my recordings were gone. Both the database entries and the mpg files
had disappeared.

When I tried mythweb, it would display settings, but hang on Listings and
Recorded Programs, presumably due to lack of response from the backend.
However, the mythtv frontend was able to communicate with the backend with
no problems.

I looked at the backend log, and the most recent message at the end of the
log was from the night before and looked like this:

2010-09-23 23:53:38.119 Closing DB connection named
'DBM........................................................................
.....

Looks like some invalid data hosed the logging. 

I restarted the backend, and there were still no backend logs generated. So
I rebooted the box, and all was OK again, although my recordings were still
gone. 

Looking in the backend log prior to the logging stopping, I found these
errors that have never occurred before:

2010-09-23 19:00:57.982 DB Error (StorageGroup::StorageGroup()):
Query was:
SELECT DISTINCT dirname FROM storagegroup WHERE groupname = ?
Bindings were:
:GROUP=Default
Driver error was [2/1030]:
QMYSQL3: Unable to execute statement
Database error was:
Got error 28 from storage engine
2010-09-23 19:00:58.037 SG(Default) Error: Unable to find any Storage Group
Directories.  Using old 'RecordFilePrefix' value of
'/var/lib/mythtv/recordings'

2010-09-23 19:00:59.939 DB Error (Creating sched_temp_record table):
Query was:
CREATE TEMPORARY TABLE sched_temp_record LIKE record;
Driver error was [2/1004]:
QMYS (rest of error message not  logged)

I don't see anything suspicious in /var/log/syslog or /var/log/messages. 

After the system rebooted, I found many errors in /var/log/mysql/error.log
of the form:
100924 18:03:25 [ERROR] /usr/sbin/mysqld: Table './mythconverg/channel' is
marked as crashed and should be repaired

There were no mysql errors logged when the problem first occurred.

I am guessing that as a result of the mysql errors, mythtv decided to expire
all of my recordings, but I don't the expires in the backend log because the
logging was hosed.

My Googling indicates that these errors occur when mysql runs out of /tmp or
other disk space. However, my single-partition system has 428GB out of a
total of 463GB free; only 35GB is in use. Or maybe a recording used up all
the disk space, and then got expired?

The system was not being used when the errors occurred, and my recording
schedule is such that the most space ever used for recordings should be well
under 100GB.

Does anyone have any ideas what might have caused this, and/or what I should
look for if it happens again?

Thanks for the help!

Dave