[mythtv-users] Howto prevent MythTV flushing/Syncing disk writes (LiveTV/Recordings) [ SOLVED ]

[Albert Graham]

David Rees wrote:
> On Tue, Mar 18, 2008 at 3:21 PM, Albert Graham <agraham at g-b.net> wrote:
>   
>> David Rees wrote:
>>     
>>> If you want to fix this, you're going to have to dig into the source code.
>>>       
>>  I'm not a C programmer (spent 20 years programming in assembler) so I
>>  may just learn C sometime soon and start
>>   to contribute to MythTV.
>>     
>
> Look at the other post I made shortly after the one you replied to,
> here's a link just to make sure:
>
> http://www.gossamer-threads.com/lists/mythtv/users/324108#324108
>
> Specifically, look at this post:
>
> http://www.gossamer-threads.com/lists/mythtv/users/270965#270965
>
> It tells you exactly what needs to be done to disable data syncing in
> MythTV to let Linux control all the data flushing.
>
> -Dave
>   

Hi Dave,

Firstly, thanks for the links.

I patched the source, and now it's perfect, actually it's fantastic, it 
only starts writing to the array when the program/recording ends or when 
it closes the file in question, another spin off of this is Live TV 
never accesses the disk now - that was a bonus I had not even thought 
about at this stage, :), but basically the write cache is of course the 
read  cache, futhermore, XFS filesystem really shows it true colors 
because when it sees more of the data it can really optimize the writes 
:) - and even never flush the file to disk if you delete the recording - 
It's like the front and the backend are running from a ram disk, this 
means that my backend is a lot cooler and silent :)

Here are some of my /proc/sys/vm settings for reference, (I know they 
are extreme for now, but for testing only).

dirty_ratio = 99
dirty_background_ratio = 99
dirty_expire_centisecs = 1000000 <--- Basically wait a long long time...
dirty_writeback_centisecs = 0
vfs_cache_pressure = 1
swappiness = 99

When reviewing the source I could see why things are done the way they 
are, I'm thinking about writing a patch that would be called "The Night 
Time Patch", the idea is to combine my "no_sync patch" detailed below, 
but also track if any frontends are connected, if they are, everything 
operates as normal, but when they are all disconnected, switch to 
"no_sync mode", in other words when you stop watching TV, it records in 
silent mode until you start watching TV again, or until it finishes 
recording a program :) (in addition to a simple daemon script mentioned 
later).

The patch would need to be an "on/off switch" in the backend 
configuration (mythtv-setup) frontends would not know any difference. 
The myth admin would need to be warned that that this feature may cause 
data loss if the backend crashes or has power loss during the night 
time, however, the admin "should :)" be asleep so the backend should be 
safe :)

I tried many approaches, firstly increasing the size of internal buffer 
from 2M to 256M (TFW_DEF_BUF_SIZE), and that in it's self was an 
improvement, for example, it wrote about every 10/20 seconds or so 
larger chunks, but that is not part of the real problem, if fact its a 
bad idea to change this buffer size, because the frontend needs to stay 
that much data behind the backend, which means you have to pause/stop 
the frontend so it does not think it's reached the real EOF, whereas the 
frontend has no problem waiting for 2M of buffered data before it resets 
its timeout counters (I assume) - i.e. waiting for 256M, the frontend 
just gives up! So, we leave buffers alone.

What I did was something like the following:

bool ThreadedFileWriter::Open(void)
{
+    no_sync = 1; /* This switch would be set from the backend 
configuration - on/off - but for now its on)

    Blah....

}

and

ThreadedFileWriter::~ThreadedFileWriter()
{
    no_writes = true;

    if (fd >= 0)
    {

        Flush();
        in_dtor = true; /* tells child thread to exit */

        bufferSyncWait.wakeAll();
        pthread_join(syncer, NULL);
        bufferHasData.wakeAll();
        pthread_join(writer, NULL);

+/**  just before the file is closed, I actually call Sync() **/
+        if (no_sync == 1) {
+          no_sync = 0;
+           Sync();
+        }

        close(fd);
        fd = -1;
    }

    if (buf)
    {
        delete [] buf;
        buf = NULL;
    }
}

and

In the Flush() function I simply return if no_sync switch is enabled 
(ensuring flush=false for compatibly)

void ThreadedFileWriter::Flush(void)
{
     flush = false;
+    if (no_sync == 1)
+        return;
+
+/** important not to execute this the following or we'll wait forever, 
and lockup the thread! **/

    while (BufUsed() > 0)
    {
        if (!bufferEmpty.wait(2000))
          VERBOSE(VB_IMPORTANT, LOC + "Taking a long time to flush..");
    }

    flush = false;
}


At the start of Sync(), I return if sync=1, notice above when I close 
the file, I must disable no_sync=0 switch so I can force a Sync() on close.

void ThreadedFileWriter::Sync(void)
{

+    if (no_sync == 1)  /** This is the reall magic! - i.e. let the OS 
decide what to do about syncing **/
+        return;

blah..
}

Thats all the changes I've made and it works a treat, however, I noticed 
when the Linux cache fills up, say 6G in my case, Linux slowly starts to 
trickle the data from the Linux cache to disk, which in a sense puts you 
back to square one :(, however it much less noticeable and you do get 
the initial 6G write free.

To solve this last problem, I have a little script which could run from 
cron or just an endless loop while the backend is runnig (I prefer the 
latter) that monitors the amount of cache memory used e.g. (free -m|grep 
"Mem"|awk {'print $7'}) or  (better still) use /proc etc.. and when it 
reaches a threshold (in my case 6G) do the following:


echo "1" > /proc/sys/vm/laptop_mode
sync
echo "3" > /proc/sys/vm/drop_caches
sync
tw_cli /c0 flush
echo "0" > /proc/sys/vm/laptop_mode

This is a really nice use of the "laptop_mode" feature, because Linux 
will flush everything on the next sync, I then drop the Linux caches 
(which should always be preceded by a sync anyway - but at this point 
all out data is safely on disk, (thanks to laptop_mode), now, in my case 
I flush the raid controllers cache to disk too (I also have a Raid BBU), 
then we disable laptop mode again, at this point we have to wait for 
another 6G of data (or a file close) before we hear from our noisy disks 
again :).

I know I've gone on a bit, but I wanted to share the experience to save 
others interested in the same issues.

So what you think ?

Albert.

ps. I lot of people have made it very clear :) - they run MythTV on old 
junk hardware! which most seem to have acquired for free! - Well, you 
too can use the above method to extend the life of your junk even 
further :).