[mythtv-users] Combined FE/BE using USB for all I/O?

[Simon Hobson]

Eric Sharkey <eric at lisaneric.org> wrote:

> Well, there's your problem.
> 
> The software I write professionally mostly does memory mapped I/O.
> Files that are being written out are mmapped, written, then unmapped.
> A separate thread syncs the unmapped regions (sync_file_range()) and
> then does POSIX_FADV_DONTNEED to encourage these pages to be evicted
> from the cache.  You can get very high output with streaming writes to
> many files simultaneously this way and you won't lose data.  Circular
> buffers are just asking for trouble.

Well I'm sure the devs that spent a lot of time making the system work reliably and with good performance under all situations will be delighted to know they were wasting their time. But as I read what you'd written, you're doing pretty much the same thing - only adding a lot of overhead mapping and releasing memory.
Out of interest, what happens with your system if the OS pauses while freeing up some memory for you to map, and then you lose a chunk of data which comes relentlessly from the hardware with minimal (if any) hardware buffer ? Bear in mind that some hardware really doesn't have any buffer worth mentioning - so you take the data *NOW* or lose it - that's part of the reason for the method the devs implemented (the data input code is a tight "read, write into pre-assigned buffer" loop).

As I said, I did contemplate playing with larger buffers - the default ones aren't very big, especially with some of the higher bit rates now transmitted. I have a feeling this would fix the data loss issue for recording. Increasing the loop timer would reduce the number of seeks (write smaller number of larger chunks). Something else on my long list of "didn't get round to it" things :-(

PS - you frequently don't want the data evicted from cache. As mentioned, if your hardware can do commflagging and/or transcoding in real time, then it's often possible to do that while recording using the data in the written cache - saving the reads (and seeks that go with them).