[mythtv-users] Problems in load-balancing commflagging

Chris Pinkham cpinkham at bc2va.org
Tue Dec 6 19:02:09 EST 2005

> (a) The setup page that deals with running jobs tops out at 5, e.g., I
>     can't tell myth to schedule more than 5 jobs in parallel because
>     the control just won't go any higher.  Why's it got this limit?

The Max of 5 simultaneous jobs (per backend) was set prior to the
creation of realtime flagging.  With realtime flagging and hardware
encoding cards I can see how someone might want more than 5 jobs running
at a time on a host.  I've bumped this number up to 10 in my source tree
so it'll be in SVN with my next commit.

The default check frequency is 60 seconds, but it goes down to 10 or
up to 300 seconds in 5-second increments.  I don't think 10 seconds is
too long to wait for a queue run, but don't see harm in setting this
as low as 5 seconds.  Going all the way to 1 could cause problems for
some people though for a couple reasons.  The constant database hits
querying for new jobs and the added system load from firing off 5-10
jobs at 1-second intervals could cause hickups in recording so I didn't
want to go that low.  I also didn't want to fire off multiple jobs in
the same cycle because that would have the same effect.  Thinking about
it now, I can see a alternative in the middle though.  I could make it
so that the JobQueue sleeps for JobQueueCheckFrequency seconds in between
runs normally, but when it fires off a job, it could be modified to
only sleep for 5 or 10 seconds so it could pickup another job quicker.

> (b) Possibly because of (a), if I do a test recording of 6 things
>     simultaneously, commflagging only happens on 5 of them at once.
>     The 6th waits until the others are done, and then runs.  They
>     all run on the MBE, even though I don't have "run jobs only on
>     original recording host" set, but OTOH, there's that cap of 5
>     jobs total, so would I ever see that sixth job on the SBE?

One of them should have run on the SBE as long as you have the SBE
configured to allow running flagging jobs.  Turn on JobQueue debugging
with "-v jobqueue" on the backend to see if it tells you why it isn't
firing off the 6th job on the SBE.  With "-v jobqueue" enabled, it
prints out information about every job everytime through the loop
so you can see what's running, what's queued, what's finished, etc..

> An even cleverer thing (but which I wouldn't be surprised can't happen
> in the current Myth architecture) would be to load-balance the commflagging,
> so that (if I'm recording with 6 tuners), 3 of them happen on the MBE,
> and 3 happen on the SBE.  Myth would have to be reasonably intelligent

This is on my TODO list, but I haven't sat down and looked at it much
because it hasn't been that high of a priority.  I'd like to make it so
that the Queue could prefer a less-busy backend (based on number of items
recording and jobs running).  I have a few other things on my list to
help improve it as well such as creating a concept of a "rush" job so
if you start watching a recording that isn't flagged it could queue a
flagging job and push it to the head of the queue and wakeup the sleeping
ProcessQueue thread.


More information about the mythtv-users mailing list