[mythtv-users] Reviving a myth system after hardware failure on /var

[UB40D]

I hadn't checked the myth box in a few days; when I did, I found that there
was no guide data at all. I did a manual mythfilldatabase and it gave an
unusual error:

table './mythconverg/program' is marked as crashed and last (automatic?)
repair failed

I googled a bit and then attempted

mysql -u mythtv -pXXXXXX -D mythconverg -e "repair TABLE program"

but this gave more errors (formatted as an sql table) and could not
complete. One of the entries in the table of errors said

Error writing file '/var/lib/mysql/mythconverg/program.MYI' (Errcode: 5)

I checked whether I had run out of space on /var, but df -h /var gave 26 GB
available and df -ih /var gave 2 million inodes free.

I attempted to copy the above file somewhere else before attempting a more
substantial database repair with myisamchk, but while doing so I got an
input/output error from cp. Argh! I checked dmesg and, sure enough, there
was an unrecovered read error on sdb.

Of course, as Murphy would have it, sdb is a 3 TB drive that has /, /boot,
/var, /home and a couple of big data partitions, so it's the most annoying
one to unmount, fix and replicate.

I'm kind of surprised/annoyed that the drive failed because it's fairly
new---at most a year old; but on the other hand the box, with its 14
terabytes over 5 drives (= practically impossible to backup fully), does
stay on 24/7 so something has to fail at some point.

I think in the interest of reliability I'll replace the drive rather than
just attempt to repair the /var partition and wait for the next hardware
failure somewhere else. However the question is: what's the safest way to
copy over what works and bring the system back to life? Note I have 12+ TB
of videos on the other drives that I'd still like to access...

I understand the best way would be to start from a known-good backup of the
database, perhaps reinstalling mythbuntu from zero. I have an sql.gz backup
from mythconverg_backup.pl which is about a week old, so that's not too
bad, but I am not 100% sure that the db was still good back then.

Noting that the machine has 5 multi-tb drives and that the sdb drive to be
replaces has itself 7 partitions, what's the most reliable way of replacing
the drive and preserving access to the over 2000 recordings (and their
metadata) currently on the system?


STRATEGY 1:
dd the old drive onto a new drive, ignoring any errors on /var; then boot
from the new drive and run a database repair tool such as myisamchk (will
it work?)

STRATEGY 2:
partition the new drive with the same 7 partitions as the old one; install
mythbuntu from distribution media onto the new drive; mess around endlessly
to restore the previous configuration (/etc/fstab can't just be copied over
because the uuids are different etc etc, and in any case WHAT FILES would I
be copying over, besides /etc/fstab, without forgetting anything
crucial??); then restore the database with mythconverg_restore.pl (will
this be enough for myth to find all the videos spread over the various
drives?)

None of the above sounds terribly satisfactory. What's a better way to
proceed?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mythtv.org/pipermail/mythtv-users/attachments/20141104/fe5890cd/attachment.html>