[mythtv-users] Howto prevent MythTV flushing/Syncing disk writes (LiveTV/Recordings) [ SOLVED ]

[David Rees]

On Wed, Mar 19, 2008 at 2:07 PM, Albert Graham <agraham at g-b.net> wrote:
>  Here are some of my /proc/sys/vm settings for reference, (I know they
>  are extreme for now, but for testing only).

Yeah, those are fairly extreme!

>  dirty_ratio = 99
>  dirty_background_ratio = 99
>  dirty_expire_centisecs = 1000000 <--- Basically wait a long long time...

Yeah, you are basically saying don't bother writing any data to disk
until it is a couple hours old!

>  dirty_writeback_centisecs = 0

This basically disables writeback, so the previous settings shouldn't
even be used.

>  When reviewing the source I could see why things are done the way they
>  are, I'm thinking about writing a patch that would be called "The Night
>  Time Patch", the idea is to combine my "no_sync patch" detailed below,
>  but also track if any frontends are connected, if they are, everything
>  operates as normal, but when they are all disconnected, switch to
>  "no_sync mode", in other words when you stop watching TV, it records in
>  silent mode until you start watching TV again, or until it finishes
>  recording a program :) (in addition to a simple daemon script mentioned
>  later).

I don't see why you would need to bother with tracking frontends,
unless you want to use your extreme VM settings. I can't see many
people going as far as you have to reduce the frequency of writes, so
I think doing so will have a very limited audience. But hey, if you
want to... ;-) Just don't expect to get such a patch upstream.

>  The myth admin would need to be warned that that this feature may cause
>  data loss if the backend crashes or has power loss during the night
>  time, however, the admin "should :)" be asleep so the backend should be
>  safe :)

Again, I think that by simply disabling the explicit syncs, you aren't
exposing yourself to any significant amount of data loss - not any
more than usual unless you are tweaking the VM writeout settings. At
which point you should know what you are getting yourself in to.

>  +/**  just before the file is closed, I actually call Sync() **/
>  +        if (no_sync == 1) {
>  +          no_sync = 0;
>  +           Sync();
>  +        }

I don't see any point in syncing when closing the file. This will
cause a large writeout which could cause other activity to lag. It
will be better to just let the OS flush the data out to disk as usual.
The large writeout will be even worse when you have extreme VM
settings like you do. Luckily, you have the hardware to support this
type of activity which is probably why you haven't seen any ill
effects.

>  Thats all the changes I've made and it works a treat, however, I noticed
>  when the Linux cache fills up, say 6G in my case, Linux slowly starts to
>  trickle the data from the Linux cache to disk, which in a sense puts you
>  back to square one :(, however it much less noticeable and you do get
>  the initial 6G write free.

Not really. The behavior you see once memory has filled up is about as
good as you'd expect. You could reduce the frequency of these writes
in a steady state writing case by re-enabling the
dirty_writeback_centisecs setting and setting it to 5-10 seconds or
so.

>  To solve this last problem, I have a little script which could run from
>  cron or just an endless loop while the backend is runnig (I prefer the
>  latter) that monitors the amount of cache memory used e.g. (free -m|grep
>  "Mem"|awk {'print $7'}) or  (better still) use /proc etc.. and when it
>  reaches a threshold (in my case 6G) do the following:
>
>  echo "1" > /proc/sys/vm/laptop_mode
>  sync
>  echo "3" > /proc/sys/vm/drop_caches
>  sync
>  tw_cli /c0 flush
>  echo "0" > /proc/sys/vm/laptop_mode

Eek! Why write out all that data to disk that you just worked so hard
to buffer up? And then after syncing it all to disk, you throw it all
away?

>  This is a really nice use of the "laptop_mode" feature, because Linux
>  will flush everything on the next sync, I then drop the Linux caches
>  (which should always be preceded by a sync anyway - but at this point
>  all out data is safely on disk, (thanks to laptop_mode), now, in my case

The only reason a sync is required before dropping the caches is
because dropping the caches will only drop data that has been synced
to disk. So when you have aggressive VM settings which don't ever
flush things to disk, trying to drop the caches will not actually drop
anything. You won't lose any data by dropping caches without syncing.

-Dave