[mythtv-users] Combined FE/BE using USB for all I/O?

[Mike Perkins]

On 16/08/14 19:41, Simon Hobson wrote:
> Eric Sharkey <eric at lisaneric.org> wrote:
>
>> Well, there's your problem.
>>
>> The software I write professionally mostly does memory mapped I/O. Files
>> that are being written out are mmapped, written, then unmapped. A separate
>> thread syncs the unmapped regions (sync_file_range()) and then does
>> POSIX_FADV_DONTNEED to encourage these pages to be evicted from the cache.
>> You can get very high output with streaming writes to many files
>> simultaneously this way and you won't lose data.  Circular buffers are just
>> asking for trouble.
>
> Well I'm sure the devs that spent a lot of time making the system work
> reliably and with good performance under all situations will be delighted to
> know they were wasting their time. But as I read what you'd written, you're
> doing pretty much the same thing - only adding a lot of overhead mapping and
> releasing memory. Out of interest, what happens with your system if the OS
> pauses while freeing up some memory for you to map, and then you lose a chunk
> of data which comes relentlessly from the hardware with minimal (if any)
> hardware buffer ? Bear in mind that some hardware really doesn't have any
> buffer worth mentioning - so you take the data *NOW* or lose it - that's part
> of the reason for the method the devs implemented (the data input code is a
> tight "read, write into pre-assigned buffer" loop).
>
> As I said, I did contemplate playing with larger buffers - the default ones
> aren't very big, especially with some of the higher bit rates now
> transmitted. I have a feeling this would fix the data loss issue for
> recording. Increasing the loop timer would reduce the number of seeks (write
> smaller number of larger chunks). Something else on my long list of "didn't
> get round to it" things :-(
>
The problem you mention has been known about ever since computers were, well, 
computers. How did you think I/O happened in the old days? OK, everything was 
slower back then but all the difficulties still existed and the answers are 
well-known and long-developed.

I worked on mainframes back in seventy-<mumble> and the disk and comms I/O was 
handled by memory allocation and release just fine. The procedures for 
preallocating read-ahead buffers, queuing them and releasing them back to the 
pool are techniques well understood, and I am sure the OS can manage such things 
much better than any program ever could, having access to the bare metal.

-- 

Mike Perkins