[mythtv-users] Problems in load-balancing commflagging

f-myth-users at media.mit.edu f-myth-users at media.mit.edu
Tue Dec 6 22:25:43 EST 2005


    Date: Tue, 6 Dec 2005 19:02:09 -0500 (EST)
    From: "Chris Pinkham" <cpinkham at bc2va.org>

    > (a) The setup page that deals with running jobs tops out at 5, e.g., I
    >     can't tell myth to schedule more than 5 jobs in parallel because
    >     the control just won't go any higher.  Why's it got this limit?

    The Max of 5 simultaneous jobs (per backend) was set prior to the
    creation of realtime flagging.  With realtime flagging and hardware
    encoding cards I can see how someone might want more than 5 jobs running
    at a time on a host.  I've bumped this number up to 10 in my source tree
    so it'll be in SVN with my next commit.

Ok.  (I'm running 1.18.1 'cause SVN is way too unstable for me, but I
might be willing to try recompiling from 1.81.1 sources if enough
things accumulate that it seems worthwhile.  Which file defines that
parameter?)

    The default check frequency is 60 seconds, but it goes down to 10 or
    up to 300 seconds in 5-second increments.  I don't think 10 seconds is
    too long to wait for a queue run, but don't see harm in setting this
    as low as 5 seconds.  Going all the way to 1 could cause problems for
    some people though for a couple reasons.  The constant database hits
    querying for new jobs and the added system load from firing off 5-10
    jobs at 1-second intervals could cause hickups in recording so I didn't
    want to go that low.  I also didn't want to fire off multiple jobs in
    the same cycle because that would have the same effect.  Thinking about
    it now, I can see a alternative in the middle though.  I could make it
    so that the JobQueue sleeps for JobQueueCheckFrequency seconds in between
    runs normally, but when it fires off a job, it could be modified to
    only sleep for 5 or 10 seconds so it could pickup another job quicker.

That would certainly work, and I agree there's no point going lower
than 10 seconds or so if it'll cause glitches.  I'll leave mine at 10s
for the moment.

    > (b) Possibly because of (a), if I do a test recording of 6 things
    >     simultaneously, commflagging only happens on 5 of them at once.
    >     The 6th waits until the others are done, and then runs.  They
    >     all run on the MBE, even though I don't have "run jobs only on
    >     original recording host" set, but OTOH, there's that cap of 5
    >     jobs total, so would I ever see that sixth job on the SBE?

    One of them should have run on the SBE as long as you have the SBE
    configured to allow running flagging jobs.  Turn on JobQueue debugging
    with "-v jobqueue" on the backend to see if it tells you why it isn't
    firing off the 6th job on the SBE.  With "-v jobqueue" enabled, it
    prints out information about every job everytime through the loop
    so you can see what's running, what's queued, what's finished, etc..

I tried this, and -everything- changed.  More in next message.

    > An even cleverer thing (but which I wouldn't be surprised can't happen
    > in the current Myth architecture) would be to load-balance the commflagging,
    > so that (if I'm recording with 6 tuners), 3 of them happen on the MBE,
    > and 3 happen on the SBE.  Myth would have to be reasonably intelligent

    This is on my TODO list, but I haven't sat down and looked at it much
    because it hasn't been that high of a priority.  I'd like to make it so
    that the Queue could prefer a less-busy backend (based on number of items
    recording and jobs running).  I have a few other things on my list to
    help improve it as well such as creating a concept of a "rush" job so
    if you start watching a recording that isn't flagged it could queue a
    flagging job and push it to the head of the queue and wakeup the sleeping
    ProcessQueue thread.

That's a cute idea.


More information about the mythtv-users mailing list