[mythtv-users] Reviving a myth system after hardware failure on /var

[Jay Foster]

On 11/4/2014 3:06 PM, UB40D wrote:
> I hadn't checked the myth box in a few days; when I did, I found that 
> there was no guide data at all. I did a manual mythfilldatabase and it 
> gave an unusual error:
>
> table './mythconverg/program' is marked as crashed and last 
> (automatic?) repair failed
>
> I googled a bit and then attempted
>
> mysql -u mythtv -pXXXXXX -D mythconverg -e "repair TABLE program"
>
> but this gave more errors (formatted as an sql table) and could not 
> complete. One of the entries in the table of errors said
>
> Error writing file '/var/lib/mysql/mythconverg/program.MYI' (Errcode: 5)
>
> I checked whether I had run out of space on /var, but df -h /var gave 
> 26 GB available and df -ih /var gave 2 million inodes free.
>
> I attempted to copy the above file somewhere else before attempting a 
> more substantial database repair with myisamchk, but while doing so I 
> got an input/output error from cp. Argh! I checked dmesg and, sure 
> enough, there was an unrecovered read error on sdb.
>
> Of course, as Murphy would have it, sdb is a 3 TB drive that has /, 
> /boot, /var, /home and a couple of big data partitions, so it's the 
> most annoying one to unmount, fix and replicate.
>
> I'm kind of surprised/annoyed that the drive failed because it's 
> fairly new---at most a year old; but on the other hand the box, with 
> its 14 terabytes over 5 drives (= practically impossible to backup 
> fully), does stay on 24/7 so something has to fail at some point.
>
> I think in the interest of reliability I'll replace the drive rather 
> than just attempt to repair the /var partition and wait for the next 
> hardware failure somewhere else. However the question is: what's the 
> safest way to copy over what works and bring the system back to life? 
> Note I have 12+ TB of videos on the other drives that I'd still like 
> to access...
>
> I understand the best way would be to start from a known-good backup 
> of the database, perhaps reinstalling mythbuntu from zero. I have an 
> sql.gz backup from mythconverg_backup.pl 
> <http://mythconverg_backup.pl> which is about a week old, so that's 
> not too bad, but I am not 100% sure that the db was still good back then.
>
> Noting that the machine has 5 multi-tb drives and that the sdb drive 
> to be replaces has itself 7 partitions, what's the most reliable way 
> of replacing the drive and preserving access to the over 2000 
> recordings (and their metadata) currently on the system?
>
>
> STRATEGY 1:
> dd the old drive onto a new drive, ignoring any errors on /var; then 
> boot from the new drive and run a database repair tool such as 
> myisamchk (will it work?)
>
> STRATEGY 2:
> partition the new drive with the same 7 partitions as the old one; 
> install mythbuntu from distribution media onto the new drive; mess 
> around endlessly to restore the previous configuration (/etc/fstab 
> can't just be copied over because the uuids are different etc etc, and 
> in any case WHAT FILES would I be copying over, besides /etc/fstab, 
> without forgetting anything crucial??); then restore the database with 
> mythconverg_restore.pl <http://mythconverg_restore.pl> (will this be 
> enough for myth to find all the videos spread over the various drives?)
>
> None of the above sounds terribly satisfactory. What's a better way to 
> proceed?
>
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://www.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
I might partition the new drive the same (or similarly) as the old 
drive, and format each partition as desired.  I would then use rsync to 
transfer the data from each partition (file system) from the old drive 
to the new drive.  I would then try to repair any known broken files on 
the new drive.  You might need to adjust UUID values for the 
drive/partitions for mounting.  I tend to prefer rsync over dd, because 
it only copies the parts of the disk that have actual data and it does 
respect bad blocks on the new drive that partitioning/formatting might 
find and mark.
Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mythtv.org/pipermail/mythtv-users/attachments/20141104/2c7673f7/attachment.html>