[mythtv-users] Memory needs

[Simon Hobson]

Anthony Giggins <seven at seven.dorksville.net> wrote:

> I was running a Combined frontend/Backend with 4GB RAM (rarely used Frontend) until recently when I was given some secondhand RAM and I'm now running 10GB, what I found was with 2 concurrent user jobs running trans-coding would cause masses of I/O and I could only queue about 3 jobs (2 running) before the system would freeze while it caught up. Now I can continue queuing trans-coding jobs until the cows come home :)

Which is exactly one use case I gave earlier for more ram - extend the size of the disk cache such that commflagging/transcoding jobs (and the like) can run using the data that's still in cache from when it was recorded instead of having to re-read it from disk.
Transcoding is worse than commflagging. Commflagging is basically read-only - it reads the data stream, looks for features within it, and adds a few database entries. Transcoding is also writing an output stream - and so that output stream will cause input data to be dropped from the disk cache.

To illustrate that last point. Suppose you have 2G of ram available for the cache, and record a program which is exactly that size - for simplicity we'll ignore everything else. You start transcoding a recording as it starts and can do it in real time. All the data the transcode job needs will always be in ram - I/O is minimised.
Now suppose you don't (for whatever reason) start the job until the recording is complete. The transcode process starts, and the first few blocks it needs are in memory. The process writes it's output stream to disk - and this causes the oldest blocks in cache to get flushed. Within a few blocks, the ones being flushed are actually the ones we will need next ! So now we have to re-read blocks that we've only just dropped from the cache - and doing this will make other blocks (which we'll need very soon) get flushed out even quicker.
So suddenly the system has switched from "everything we need is in memory" to "nothing we need is in memory and we need to read it from disk". And for good measure, the disk (assuming the transcoded version goes on the same drive) now has to thrash about between reading the input stream and writing the output stream - thus making both operations slower than "the sum of the parts".
And of course, all this stuff getting flushed to make way for other stuff we only need once means that other stuff we may want (eg database file blocks, or video files we are trying to watch) get flushed, and so these other operations also slow down. This latter one is why it's a good idea to tune the database to keep stuff in it's own reserved memory as far as possible - the very reason for Myth (handling large video files) means that relying on the OS cache for the database is a bad idea.