[mythtv] [mythtv-commits] Ticket #2782: automatic simultaneous jobs scaling

[osma ahvenlampi]

On 12/9/06, Daniel Kristjansson <danielk at cuymedia.net> wrote:
> When recording an MPEG stream MythTV needs to write the
> keyframe locations to the DB. This allows for quick seeking.
> If the DB blocks, MythTV waits until the DB has has finished
> before writing any more A/V data to the disk.

I don't think MySQL will block under this scenario if using InnoDB. I
haven't tested that, though. Even if it will, InnoDB commit is a 4096
byte sequential write to a ringbuffer, much faster than MyISAM's
seek-write-seek-write-index. FWIW, libsqlite is better with this than
MyISAM as well.

> Most of us are using Linux, but I'm weary of adding Linux
> dependencies. If the only possible fix on some other operating
> system fixes it for Linux too, then we probably want to go
> with the more portable fix.

Sure. I could not find a way to get to this information without being
OS-specific, *but* not getting to it doesn't hurt the system any
longer - the new version of the patch just won't start more than one
background job without having idle time.

> I hadn't considered I/O wait, that might be the simplest way to
> retrofit this metric.. The I/O wait stat includes both disk and
> network, right?

It should, but I'm not that deep of an expert on the kernel. This is
where I got the idea for the patch from - Linux 2.6 is the first OS
I've seen to separate iowait state out of system or user time, though
I'm sure there are others (haven't looked whether Solaris does it
these days). But I'm not depending on seeing the iowait, I'm looking
at idle vs all the rest, so I'm only depending on idle not including
iowait. As far as I know, that is true for most other Unix-like
systems.

> Generally, if it is small re-implement, if it is optional and
> the library is generally available link, if it is big and not
> optional import the library. But Issac is the final arbiter on
> this issue since we all can easily disagree on where a library
> fits on this continuum.

Well, libgtop is the best portable system info library I know of, but
it does a lot more than is needed here (full process lists and memory
maps etc) and is not installed by default with anything but GNOME
(AFAIK). KSysGuard hasn't been updated to look at I/O wait so it will
actually miscalculate total CPU (and resultingly the percentage of
idle time vs total time). I'd recommend using libgtop as a reference
to implementing support for other OSes, but I couldn't find any that
were actually advertised as alternative platforms for mythbackend.

> It's impossible to keep all resources 100% utilized with an
> application like MythTV with some soft real-time requirements.
> It is better to err on under-committing the resources we have.

Fully agree. I believe this patch will do exactly this, as it will now
permit *one* background job when they are enabled, but will not permit
more if idle time looks to be close to zero (well, less than 25% which
is in my empiric testing as good a threshold as any). That is, the old
functionality would have been to start up-to maxJobs under the same
circumstances, and potentially overcommit resources.

And once again, if the idle time can not be determined (as would be
the case for non-Linux platforms), it falls back to old behaviour.

Please review the patch (jobqueue_dynamic) -- the part about stripping
away settings is separate and doesn't need to be considered at this
point.

-- 
Osma Ahvenlampi   <oa at iki.fi>       http://www.fishpool.org