[mythtv-users] Combined FE/BE using USB for all I/O?

[Eric Sharkey]

On Sat, Aug 16, 2014 at 2:41 PM, Simon Hobson <linux at thehobsons.co.uk> wrote:
> Eric Sharkey <eric at lisaneric.org> wrote:
>
>> Well, there's your problem.
>>
>> The software I write professionally mostly does memory mapped I/O.
>> Files that are being written out are mmapped, written, then unmapped.
>> A separate thread syncs the unmapped regions (sync_file_range()) and
>> then does POSIX_FADV_DONTNEED to encourage these pages to be evicted
>> from the cache.  You can get very high output with streaming writes to
>> many files simultaneously this way and you won't lose data.  Circular
>> buffers are just asking for trouble.
>
> Well I'm sure the devs that spent a lot of time making the system work reliably
> and with good performance under all situations will be delighted to know they
> were wasting their time.

You know I never said anything like that.  MythTV is a tremendously
successful accomplishment, but that doesn't mean it's perfect in every
way and that there aren't parts which can't be made better.  If you
talk to any developer, I'm sure you'll find dissatisfaction with this,
that, or the other thing, and some regrets about decisions made long
ago.  That's the nature of software development and is why there's
always a new version being worked on.

> But as I read what you'd written, you're doing pretty much the
> same thing - only adding a lot of overhead mapping and releasing memory.

The critical difference is that there's no synchronization to block
the writing thread from getting arbitrarily far ahead of the syncing
thread.  There's no fixed size, compile time, same on every system
limit on the amount of memory used.  With this kind of design, one
thread reads from the recording device as fast as possible, and
another thread syncs to disk as fast as possible.  The OS is free to
schedule the actual disk I/O in ways that it sees fit, even if that
means delaying writes for a particular block by multiple seconds
(which would have caused a fixed size buffer to fill).  In other
words, the buffer size scales dynamically with need, never larger nor
smaller than necessary.

And memory mapping doesn't add overhead.  In fact, just the opposite.

http://en.wikipedia.org/wiki/Memory-mapped_file

"The primary benefit of memory mapping a file is increasing I/O
performance, especially when used on large files."

The article covers some reasons why this is the case.

> Out of interest, what happens with your system if the OS pauses while freeing
> up some memory for you to map, and then you lose a chunk of data which
> comes relentlessly from the hardware with minimal (if any) hardware buffer ?

If data is coming from the device faster than the system can allocate
RAM pages to hold that data, then you're truly sunk and will lose data
in that case.  But at that point, you've really hit a limit of your
hardware.  It means you don't have enough ram to properly buffer the
writes to your I/O system.  It's not a software problem at that point.

> PS - you frequently don't want the data evicted from cache.

Yes, I know.  The posix_fadvise functions don't actually evict
anything, they just move items closer to the front of the eviction
queue.  If RAM is plentiful, these pages will be kept cached for quite
some time.  They'll only be evicted if ram is tight.

I think Mike Perkins said it very well when he said "the OS can manage
such things much better than any program ever could, having access to
the bare metal".

The MythTV devs have worked hard to get MythTV working well under many
varied situations, but the Linux kernel developers have done the same.
There are more kernel developers and the kernel code is more heavily
stressed and tested than any particular application code.  Relying on
the OS to do OS-like things like figuring out when to schedule
particular disk writes is often the right call.

Eric