[mythtv] [mythtv-commits] Ticket #1791: Stop transcode/commflag from hogging the CPU while recording/playback is in progress.

[f-myth-users at media.mit.edu]

This actually may be relevant to both #1791 and 2194 in a sideways
sort of way.  I'm forwarding below a message I sent to -users that
sank without a trace; its general idea is that the scheduler kickoff
at the end of a recording is trashing in-progress recordings unless
I'm -very, very- careful not to have any other disk I/O in progress,
despite running on powerful hardware and keeping the DB and recordings
on separate spindles and buses.  The right solution is to (somehow)
make the scheduler less aggressive or cause MySQL to stop hitting the
disk so hard, and that's what I'd like.  But if that can't happen:
(a) Can commflagging simply be suspended in a window around any
    scheduler operations?  On -all- commflagging hosts?
(b) Randomizing when mythfilldatabase runs will introduce the
    potential for lossage as above; it'd be nice if there was a way to
    say "Don't run mythfilldatabase if a program is due to -end- in
    the next n minutes [for me, ~15]", so that way the scheduler
    doesn't get kicked while MFD is running as well.  [Or perhaps
    just "don't run MFD until no recordings are in progress.]

[Failing that, the larger hammer of #1791 could work for me, because
it would guarantee no commflagging -at all- during any recordings;
then I could allow a zillion commflagging jobs in parallel when -not-
recording and catch up, since one job runs at >6 times real-time even
over a 100Mbps network and might run even faster locally (I haven't
carefully checked how much runtime a single job takes).  But it'd
sure be nicer to fix the underlying problem and allow commflagging
during recording -and- in parallel, so I could jump into a stream
currently being recorded and still skip commercials.]

It's a pity that scheduler interactions are doing this, since
otherwise the boxes have plenty of I/O and horsepower to record
half a dozen streams and commflag them simultaneously, and even
to play one, with no problem at all.  It's -just- the thrashing
when the scheduler gets run that swamps the disk badly enough to
cause IOBOUND errors.  (To avoid it, I have to be very careful
about when and how much comflagging [and other I/O] happens,
or I'd have to put the DB on a whole separate host.)

- - - Begin forwarded message - - -

Date: Wed, 13 Sep 2006 02:44:52 -0400 (EDT)
From: f-myth-users at media.mit.edu
Subject: [mythtv-users] Making the scheduler and/or MySQL thrash the disk less

The scheduler is trashing my recordings, and I'd like advice on how to
make it stop.  I've taken some rather extreme steps and they've only
helped a little.

When a recording ends, it triggers the scheduler to run.  That run
thrashes the disk hard enough that it leads to IOBOUND errors that
drop 10-30 seconds from all streams being captured at the time.
Since almost everything I record runs with 1-2 minutes of postroll,
this pretty much guarantees that any recording that ends will trash
an in-progress recording 1-2 minutes in.  I'm running with "-v all"
logging and it's really, really obvious that the scheduler runs all
start about 10 seconds before IOBOUNDs start cropping up.

(Amazingly, checking, optimizing, and mysqldumping the database does
-not- thrash the disk hard enough to cause this---but the scheduler
query -does-.)

If 0.20 thrashes the disk less, that'd be good to know, but somehow I
doubt it.  Can I retune MySQL somehow?  Do something else?

Configuration:  MBE w/5 PVR-250's on an MSI K7N2 Delta-L w/AMD 2800+
CPU, 1 GB RAM, and 2 Seagate 200 GB PATA IDE drives on opposite buses
(e.g., /dev/hda and /dev/hdc).  SBE/FE w/1 PVR-350, identical memory,
RAM, and drives.  100Mbps switched net between them.  Running Ubuntu
Breezy & Myth 0.18.1; root filesystem (also used by MySQL) is ext3fs;
media filesystems are JFS.  The main recording directory is on the
JFS on the MBE's hdc; the SBE streams over the network via NFS to
that directory.  The OS & MySQL DB are on MBE's hda.

Here's what I've done:
o  Yes, the database is frequently checked & optimized.
o  Maximal buffering (see the wiki, e.g., "options ivtv yuv_buffers=32
   mpg_buffers=16 vbi_buffers=16 pcm_buffers=16 dec_osd_buffers=2"),
   which chews up a rather enormous amount of RAM but still leaves me
   with ~150meg free RAM on the MBE (I sent a message a week or two
   ago asking -why- the memory usage was so high, but got no response).
o  The OS (and MySQL) are currently using hda, while the media
   filesystem is on a separate spindle -and- bus (hdc).

If the machine is otherwise unloaded but recording a single stream,
the scheduler query isn't bad enough to get an IOBOUND.  Even if it's
recording all 6 possible streams (but doing nothing else), the
scheduler won't get an IOBOUND.  (OTOH, before I went to maximal
buffering -and- put the DB on its own spindle, I could get IOBOUNDs by
doing innocent scheduling operations [e.g., in MythWeb, going from
dup-check of "All Recordings" to "Only New Episodes" if that caused
the un- or re- scheduling of a dozen episodes of something], so doing
both the buffering and the second spindle -did- help.)

If, on the other hand, I try copying a file from hdc to hda (I've got
a JFS on each spindle [the hda one isn't used directly by Myth]),
that's enough stress that a scheduler run will cause IOBOUNDs on
running streams.

If enough commflagging jobs are running, they will -also- cause this
problem.  (There -could- be 5 running at once, if most of the tuners
are/were recently in use when the scheduler runs, and apparently
that's enough disk load that it's comparable to a a direct hdc-to-hda
copy, even though the commflaggers are mostly running on the SBE, not
the MBE.)

Here's why I'm confused:  Clearly, adding lots and lots of ivtv
buffering isn't quite enough to deal with the peaks.  Fine.  But it
was my impression that putting the DB on a spindle -and- IDE bus (hda)
that was -not- in any way being used for recording (hdc) meant that
hda could -not- starve hdc for disk access!  Yet that seems to be
exactly what's going on here.  It's -better- than when everything was
on hda, but I would have expected it to be entirely gone, and it's
not.

When I spent a whole bunch of time last week increasing buffering,
adding a spindle, and running tests, I didn't immediately see this
problem, because my testing regime was (a) start 5 manual recordings
of 10 minutes; (b) start 1 manual recording of 5 minutes, and (c)
check to see if the 5-minute recording's end caused glitches in the
other 5 streams.  Unfortunately, the commflagger doesn't -start- until
5 minutes in, and staggers each start by a minute, so I wasn't provoking
enough disk I/O to get hit by the misbehavior (while I figured I
wouldn't be deliberately doing huge hda<->hdc copies by hand while the
scheduler might be running, I'd forgotten about the commflagger's
behavior).  But under a real load, in which most of the streams get
commflagged, I just got bitten by this---in fact, one recording got
bitten by it at about 1, 1.5, and 2 minutes in, from several
successive scheduler runs as postrolls of 1 and 2 minutes ended!

Things I could try, in increasing order of undesireability:
o  Run only 1 simultaneous commflagger, not 5.  This is annoying
   because when the machine is making lots of simultaneous recordings,
   commflagging can fall -way- behind.  This machine is used for
   research and is often in this position; -at the moment- it probably
   doesn't record so much that a single commflagger couldn't
   eventually keep up, but it might in the future.  And if it weren't
   for the scheduler-caused glitches, it's got plenty of disk, net,
   and CPU power to run those 5 jobs and record on 6 streams just
   fine.  (This also means that, even if it -is- keeping up, it might
   be hours before any particular recording is flagged, whereas right
   now I'm telling it to start commflagging "as soon as possible" (not
   at the end of a recording) and doing a bunch in parallel.)
o  Put the DB (and hence the MBE) on a machine that doesn't have any
   tuners at all [or at least doesn't own the filesystem those tuners
   are writing to]---or perhaps flip things around so the MBE only has
   1 tuner of its own and the SBE has the other 5.  This solution is
   really fragile, because it means losing either of these machines
   kills both dead, and I -do- occasionally have wedges (generally
   caused by the PVR-350 on the FE).  The nice thing about the current
   setup (MBE also has most of the tuners -and- the disk that the data
   is going to) is that a crash of the FE/SBE only takes out a single
   tuner and doesn't affect the rest of the system.  And having to put
   the DB on a -third- machine would be even worse---expensive, hot,
   lots of network traffic, etc etc---all because of that single MySQL
   query.

Assuming that there's nothing I can do to either tune MySQL -or-
change the scheduler query so it doesn't push MySQL so hard:

Can I throttle the flagger's rate somehow?  But I might have to
throttle it so much that I might as well just one 1, or even run
1 that only starts -after- the recording ends (still bites me if
there are many recordings back-to-back, which is usually the case).

How feasible would it be to, e.g., -suspend- commflagging for a couple
of minutes around each scheduler run?  I don't know if there's any way
for the scheduler (or whatever calls that query) to talk to all the
commflaggers to tell them to hold off for a bit---especially since
those flaggers are mostly running on the SBE, not the MBE.

Any other ideas?

Thanks.

- - - End forwarded message - -