[mythtv-users] Memory needs

[Anthony Giggins]

On 19 October 2016 at 21:05, Simon Hobson <linux at thehobsons.co.uk> wrote:

> Anthony Giggins <seven at seven.dorksville.net> wrote:
>
> > I was running a Combined frontend/Backend with 4GB RAM (rarely used
> Frontend) until recently when I was given some secondhand RAM and I'm now
> running 10GB, what I found was with 2 concurrent user jobs running
> trans-coding would cause masses of I/O and I could only queue about 3 jobs
> (2 running) before the system would freeze while it caught up. Now I can
> continue queuing trans-coding jobs until the cows come home :)
>
> Which is exactly one use case I gave earlier for more ram - extend the
> size of the disk cache such that commflagging/transcoding jobs (and the
> like) can run using the data that's still in cache from when it was
> recorded instead of having to re-read it from disk.
> Transcoding is worse than commflagging. Commflagging is basically
> read-only - it reads the data stream, looks for features within it, and
> adds a few database entries. Transcoding is also writing an output stream -
> and so that output stream will cause input data to be dropped from the disk
> cache.
>
> To illustrate that last point. Suppose you have 2G of ram available for
> the cache, and record a program which is exactly that size - for simplicity
> we'll ignore everything else. You start transcoding a recording as it
> starts and can do it in real time. All the data the transcode job needs
> will always be in ram - I/O is minimised.
> Now suppose you don't (for whatever reason) start the job until the
> recording is complete. The transcode process starts, and the first few
> blocks it needs are in memory. The process writes it's output stream to
> disk - and this causes the oldest blocks in cache to get flushed. Within a
> few blocks, the ones being flushed are actually the ones we will need next
> ! So now we have to re-read blocks that we've only just dropped from the
> cache - and doing this will make other blocks (which we'll need very soon)
> get flushed out even quicker.
> So suddenly the system has switched from "everything we need is in memory"
> to "nothing we need is in memory and we need to read it from disk". And for
> good measure, the disk (assuming the transcoded version goes on the same
> drive) now has to thrash about between reading the input stream and writing
> the output stream - thus making both operations slower than "the sum of the
> parts".
> And of course, all this stuff getting flushed to make way for other stuff
> we only need once means that other stuff we may want (eg database file
> blocks, or video files we are trying to watch) get flushed, and so these
> other operations also slow down. This latter one is why it's a good idea to
> tune the database to keep stuff in it's own reserved memory as far as
> possible - the very reason for Myth (handling large video files) means that
> relying on the OS cache for the database is a bad idea.
>
>
Exactly Simon,

The other thing to note with an low RAM system and especially with a
combined frontend/backend that is rarely used as a frontend is when you
goto use it, it has to swapped the mythfrontend process memory to swap and
has to read it back so its very slow to respond until this has completed.

I believe you can pin certain processes to RAM with cgroups but I haven't
investigated this much.

Cheers,

Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-users/attachments/20161020/512c480f/attachment.html>