[mythtv-users] Is locking tables important when backing up mythconverg?

[f-myth-users at media.mit.edu]

    > Date: Sat, 07 Apr 2007 10:55:58 -0400
    > From: "Michael T. Dean" <mtdean at thirdcontact.com>

[Simple (!) feature suggestion that might save some work, about halfway down...]

    > On 04/06/2007 10:33 PM, Yeechang Lee wrote:
    > > So I have this script
    > > (<URL:http://www.gossamer-threads.com/lists/mythtv/users/247868#247868>)
    > > that I run every two hours to back up mythconverg.

    > As I had previously discussed--and then decided against--I am now, once 
    > again, planning to add automatic repair, optimize, analyze, and backup 
    > into mythbackend's housekeeper.

You presented a fairly good case for why not to do this.  I'm curious
(having no particular opinion either way)---what changed your mind?

    >				     It should be there by the time 0.21 
    > comes out (unless someone gets super motivated and puts out 0.21 very 
    > quickly...).  (Thanks to input from Chris Pinkham, I think we now have a 
    > reliable and safe plan for implementation.)

    > Oh, and for those who have moved to SVN trunk because of the, "I 
    > accidentally upgraded my system and can't go back because I didn't back 
    > up the database first," thing, the database upgrade will also backup 
    > your database automatically before upgrading the database.  :)

This is excellent news!  It's not a complete solution for the "oops!"
problems I've been talking about, but it's certainly part.  (And I
will probably write up & submit a patch for the rest of it sometime
after the first week of May, given current other deadlines---even with
your auto-backup stuff, it's still way too easy to accidentally blow
currently-running recordings out of the water with an accidental
upgrade and/or accidentally upgrade and not realize it for quite
a while, hence making rollback difficult because now there are new
recordings in the new schema.  We just saw mail from someone this week
talking about having to take a machine off his net to run a test safely;
I know exactly how he feels, since that's what I've had to do as well.)

This also simplifies my patch, since part of it had an "auto-backup"
step if you proceed w/a schema upgrade.

    > Sure, this doesn't help you, yet, but I just thought you might like to 
    > know it's on the horizon so you could take the "easy" way out (i.e. just 
    > back up at times when your box is very unlikely to be in use) for now.

I assume that the daily-database-backup code uses (some) definition
of "enough time" the same way that your "mfdb if there's enough time"
patch would?  This general idea ("do this action only if there's
enough time before the next recording") could be widely useful all
over Myth, and so I have what I hope might be a trivial addition to it:

Allow wrapping the housekeeper so a user-supplied script (with no
arguments?) can be run both -before- and -after- the mfdb and backup
runs.  (I don't know if backup & mfdb are supposed to run with each
other, or separately, or in some particular order, but the use case
-I- have in mind is specific to the DB backup and not mfdb per se.)
(No UI---just make the user change a field in the DB from NULL to the
pathname to the external program.)

For the "before" script, check its return value and DO NOT RUN the
housekeeper stuff if it doesn't return 0.  For the "after" script,
just run it all the time (unless DB backup and/or mfdb can return
errors, in which case don't run it?  unsure).

Why allow wrapping?  Because -my- backups do extra work afterwards:
they copy the backup to a second disk besides the "backup" disk, and
they prune extra copies, and they do some other work as well.

Why the before part of the wrapper?  Why not?  If you've got one, it's
trivial to add the other, and who knows, there might be some situation
where the user wants to make sure backups (or mfdb?) -don't- happen if
some other situation is true (like it would cause glitches we didn't
somehow anticipate, or whatever).  Or maybe their backups are on a
filesystem kept read-only or offline by default (to prevent accidents
with rm and the like) and they need to put the storage back online so
the backup system can write to it.  [I said "no args" only 'cause I
can't think of what args it might make sense to supply.]

    Oh, and it will likely be a daily backup--rather than a 2-hourly 
    backup.  But, if you're really that concerned, you could always change 
    the source in your copy.

(I'd change mine to 4 times/day---the issue here is not having a huge
PITA of new recordings to somehow re-merge if I lose the DB up to 24
hours after its last backup.  If this was configurable as "backup
every n hours", that'd be great---or some might argue I should just use
binary logging to do this, though I turned it off because I thought it
was one of the things that was causing scheduler updates to cause ivtv
to hang.  Now that that's fixed, I haven't turned it back on because
scheduler runs are still annoyingly long (they hang the FE UI) and I'm
afraid keeping binary logs will just exacerbate the problem.)

THE GENERALIZATION---there's a general problem here where we want to
run stuff that pounds on the DB but we don't want to do it while any
recordings are in progress.  So why not just have a general hook,
which runs after scheduling is completed (hence after a recording
ends, -or- after the user -changes- something, -or- after mfdb) which
can run a user-supplied external program whenever Myth will be idle?
(See my next message about what "idle" means and how we tell.)

Presumably the user can use this to run a program many times a day;
I imagine typical cases will exec the program and it will instantly
decide "I ran too recently & have nothing to do" but every so often it
will decide "okay, time to do <whatever>".  This is a generalization
of the whole housekeeping idea that can run user-supplied code.  (If
Myth were written in Lisp, I'd just define some :AROUND advice and be
done with it, but it's not. :)

(Real fanciness would have two versions:  "run user-supplied code
after recording completes no matter whether another is ongoing; and
run USC iff we're idle".  You could use the former to run, e.g.,
jobs that rename files to have pretty names (why poll via monitoring
the filesystem or running via cron when you can be event-driven
instead?) and use the latter for, e.g., DB backups and other things
that pound the FS or the DB.  The former might even be a more general
version of kicking off transcoding, although that's probably not
useful given that the jobqueue already exists.  But who knows what
clever uses some users might dream up if the functionality is there?)

Hmm---I just realized I suggested this post-scheduling hook 6 weeks
ago in a different context---sharing one STB output across two tuners
in a way the Myth scheduler couldn't cope with.  If I could run a
sanity-checker after every reschedule, it could detect when Myth was
trying to use the same STB for two overlapping recordings on different
channels and warn me the instant the schedule was created (probably by
me).  And then there are sanity-checkers to check that a recording
really worked by looking at length & blackness & rescheduling
instantly if it didn't...  the list goes on and on for ways this
functionality could be used.