[mythtv] Scheduler needs table keys?

f-myth-users at media.mit.edu f-myth-users at media.mit.edu
Thu Mar 1 05:32:09 UTC 2007


    > Date: Wed, 28 Feb 2007 23:55:52 -0500
    > From: Chris Pinkham <cpinkham at bc2va.org>

    > * On Tue Feb 27, 2007 at 03:31:12AM -0500, f-myth-users at media.mit.edu wrote:
    > >     > be doing one final curRecording->SetFilesize() when the recording is
    > >     > done anyway.  The filesize is updated during recording for a couple
    > > 
    > > In my test system, it's not getting set at the end, since you
    > > suggested commenting out -both- occurrences of SetFilesize; one

    > Yes, it was a test, not really meant for production.  I think I suggested
    > that before I actually confirmed in the other thread that it was the
    > scheduler effectively locking the recorded table which prevented the
    > filesize from being updated.

Ok.  I still haven't been able to run a full-up test with my
production system, but a quick trial on my recently-built devo
system doesn't show any problems (and it won't show any corruption
because it's on a faster machine with a smaller database).  [When
I do, questions about useful tests to run are below.]

    > > is in ProcessData, but the other is in FinishRecording.  Did you
    > > mean to totally inhibit the latter, or did you just not notice
    > > that one of the calls was there?

    > There is code in the backend to update the filesize in the DB if a recording
    > has a filesize = 0.  I think this code has been in there since I first
    > added the filesize field to the database because we needed to populate the
    > filesize for old recordings.  I just set all the filesizes on my dev system
    > to 0 and went into the Watch Recordings screen and now the recordings have
    > their filesize populated in the database again so this code appears to be
    > working correctly in trunk as well.

Yes---the first time the FE showed Watch Recordings, that field was
populated, and then stayed constant.

So sometime last night I backed out my patch so it only comments-out
the update in ProcessData and leaves the update in FinishRecording,
and, as expected, at least that way it's updated at the end of the
recording.

    > > In an unmodified system, recorded.filesize gets updated every 15
    > > seconds, which makes sense if it happens every 30 keyframes.

    > Yes.

Btw, realizing this the other day made me realize one reason why this
corruption is so unpredictable---not only does the scheduler lock that
table for widely varying amounts of time for no reason I've seen (I'm
assuming it's a combo of FS caching & other things), but there's
probably some dither on -exactly- when keyframes happen in each stream
(e.g., based on exactly when a tuner starts recording), and if the
30th keyframe arrives just -before- a long scheduler-updated-induced
lock (and hence recorded.filesize got updated), then the first 15
seconds of that lock are "free" in that they can't cause a problem
---the table has to stay locked 15 seconds -plus- the ivtv buffering
value in order to drop data.  OTOH, if you're unlucky and that
keyframe arrives just -after- the table gets locked, you're going to
lose a -lot- of data.  This quantitization of the thread's behavior
probably explains why even in simultaneous streams, sometimes I saw a
little dropped from all of them, and sometimes I saw nothing dropped
from one or two and a huge chunk missing from another.  And since
recordings both start and end on one-minute boundaries unless you're
doing something truly strange with start-times and pre/post-roll, that
tends to ensure that keyframes will arrive -just- as a recording ends,
meaning you're quite sensitive to which side exactly of that scheduler
query you land on---it's unlikely to be randomly distributed, for a
particular system configuration.  [Yeah, I know, the tuner may not
start up exactly on the minute, but neither does the scheduler query,
etc.  For any given configuration, it's probably fairly deterministic.]

    > > In my test system, recorded.filesize starts out 0 when a recording
    > > commences, and is modified to the current filesize the instant the
    > > frontend looks at it (e.g., the first time I go to the listing of
    > > what's been recorded---I did this a minute or so into a 5-minute
    > > manual recording).  At that point, nothing I do, including skipping
    > > around in the recording, leaving and reentering that frontend page, or
    > > even terminated and restarting the frontend, causes the value to
    > > change.  [This is really easy to demonstrate; I just put the relevant
    > > SQL query in a one-second loop in a shell and watched it in another
    > > window while playing around.]  Obviously the frontend sees a value
    > > there and never tries to update it.

    > OK, so this is a negative side-effect of removing those calls to
    > SetFilesize().  The frontend doesn't actually update the filesize,
    > the backend does, but once it updates it once for a running program,
    > it will never update it again since the value is now non-zero.
    > If you want to work around this in your tree, you can look in
    > mythtv/programs/mainserver.cpp and look for calls to SetFilesize and
    > comment those out as well.  This will cause the backend to look up
    > the current filesize on the filesystem, but it won't write it out to
    > the database.  This is at the expense of load time on the Watch
    > Recordings screen though because the backend will have to stat every
    > file which is the way things used to work a few years ago.

-My- Watch Recordings screen doesn't seem to need to know the filesize
-anyway-, since there's nothing there that mentions it!  I use G.A.N.T.
as my theme (what -does- this stand for? :)---is it possible that this
is some number that some other theme(s) would be showing me but isn't
even displayed in G.A.N.T.?  (If so, I'll be -really- pissed that
I've gone through all this pain---and a year of trashed recordings---
for a value that Myth was faithfully updating but was completely
useless except for a display I don't even see.  *sigh* :)

[Is there an opportunity here to only compute this data if the theme
requires it?  Such lazy evaluation might be handy for others.  OTOH,
as long as keeping it updated doesn't cause these buffer-thread/scheduler
deadlocks, I don't really have an opinion on this...  Hmm, and I guess
MythWeb needs this info somehow; I'm assuming it's using that field.]

I'm going to leave mainserver.cpp alone for the time being (once I get
the chance, I'll run my production system with -just- that single line
in ProcessData commented out) and see the results; I'm guessing that
there's no way that mainserver could lock up the table long enough to
cause dropped data, but I'll test it by rebooting a frontend and
selecting Watch Recordings while something is recording, and see
if that corrupts it.  [Can you think of a more-strenuous test I should
try instead?]  If not, I'll leave it alone.

    > > I noticed that -something- split out an error flagging my test
    > > video about "MPEG motion vector out of boundary" and "concealing 45
    > > DC, 45 AC, 45 MV errors" (and some other warnings in between).  I

    > Data corruption, nothing to do with the filesize in the DB.  Very
    > little code actually looks at the filesize field, it is mainly used
    > for the Watch Recordings screen.

...and apparently not even by mine, if I'm seeing this correctly.

    > > Note:  if not having the filesize set -is- causing commflagging to
    > > barf, I need to know in order to tell commflagging not to start until
    > > the recording is finished.  (Right now, it runs concurrently.)

    > Shouldn't be, a "find ./ -name '*.cpp' | xargs grep filesize" shows
    > very little places using this value.

Ok.  I didn't happen to run that grep 'cause I simply assumed that it
was some critical value or it wouldn't be getting updated so religiously. :)

    > > So the big question:  Do I care that recorded.filesize won't -ever-
    > > get updated to the "final" value?  What looks at it?  Or should I
    > > only remove the use in ProcessData and leave FinishRecording alone?
    > > [I'm assuming that, by the time that's running, we've read all the
    > > ivtv buffers we ever will, so it doesn't matter if -that- hangs for
    > > 30+ seconds if the scheduler is already running; is this correct?]

    > It should be safe (and better) to only remove the one in ProcessData,
    > that way even if the backend sets the filesize in the DB, the recorder
    > will update the value to the final filesize when the recording finishes.

Yeah, that's what I did.  I hope to be able to run some definitive
tests either late tonight or (more likely) tomorrow afternoon, given
when I can take the prod backend down during a maintenance interval.

Thanks!


More information about the mythtv-dev mailing list