[mythtv] [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling

[osma ahvenlampi]

On 12/7/06, Chris Pinkham <cpinkham at bc2va.org> wrote:
> cards now, this is more of a possibility, but there are other things that
> we need to consider than just CPU.   Just because my system can run X jobs
> before the CPU usage gets above Y% doesn't mean I want to run X jobs because
> that will affect things like storage, potentially network throughput, etc..

The way I wrote the patch, it will only start a job if 25% of CPU is
"idle". That 25% is hardcoded because not only did I want to make this
automatic, I also really wanted not to introduce any more config
options (believing that software should "do the right thing right"
without configuring). 25% should also be 1) easily enough for
recording and playback, especially since the jobs will be run niced
anyway, and 2) actually relatively rare, especially on a 1-2 core
machine since most candidate jobs would consume all resources
available.

It also considers ONLY time reported truly "idle", ie. not used by
sys, user, nice OR iowait, so it should cover for the cases I think
you're referring to, although truly considering network load would be
very tricky, it not being entirely measurable on the host. Anyway, in
my environment I've only seen two kinds of situations that have been
problems with the old way, but both of these are very common: two
CPU-intensive concurrent processes, ie MPEG-2 -> MPEG-4 transcodes or
transcode plus delayed commflag, each finishing later than they would
have had only one been running at once, and two IO-intensive
concurrent processes, ie MPEG-2 -> MPEG-2 lossless transcodes, also
finishing (much) later than if run sequentially, AND hurting
concurrent recording and playback performance.

The patch fixes both of these situations "perfectly" (well, it's
worked fine for me for several days, while I record up to two DVB
streams at once on a combined FE/BE), and yet allows also two
real-time commflags (but only if no transcodes are running when the
commflags are candidates for starting - that's something that could
potentially be improved but would complicate the jobqueue scheduler
quite a bit).

The biggest drawback is that the "util" part of figuring out whether
idle time is available or not only supports Linux (it reads from
/proc/stat), so support for other OSs would need to be contributed by
others. The way it's written, it will allow jobs to start if it can't
figure out whether CPU is available.

> Users will probably always prefer a late commercial flag or transcode rather
> than a corrupted recording.

This has never been an issue for me, but I use DVB receivers instead
of framegrabbers.

> All that being said, I haven't even looked at the patch yet, I just wanted
> to put it in my queue since it is JobQueue related.  Even if it is perfect,
> don't expect it to go in over the next few weeks because my holidays are a
> bit busy.  That is one of the reasons I wanted to get Storage Groups committed
> a few weeks ago to give it time to stabilize.

Cool. Let me know if you have any questions or critique, I'd be happy
to work on that. However, I don't expect to upgrade from 0.20-fixes
any time soon, my install needs to remain WAF, so I hope that won't be
a problem.

-- 
Osma Ahvenlampi   <oa at iki.fi>       http://www.fishpool.org