[mythtv-users] File System Recommendation?

[Brian Wood]

On Apr 2, 2006, at 10:10 PM, f-myth-users at media.mit.edu wrote:

>     Date: Mon, 3 Apr 2006 13:01:21 +1000
>     From: "Richard Dale" <richard at interlink.com.au>
>
>> I'm using ext3 on a RAID-0 system. Who the heck cares if it takes a
>> second or two to delete a 1-hour show ??
>
>     Here's an interesting solution - the recording could be flagged  
> for deletion
>     and then deleted in a background thread.  So, even if it takes  
> a while it's
>     the background thread that has to wait on I/O, not the  
> foreground user
>     interface.
>
> The problem isn't making the -user- wait for the disk, it's making
> -Myth- wait for the disk.
>
> While ext3fs is deleting a file, all other writers block.  If you're
> currently writing a stream (or several) to the same filesystem, -they-
> will block.  Eventually your buffers will overflow, and you'll lose
> data, and your recordings in progress will have glitches in them.
>
> I verified this experimentally with ext3fs on an ordinary disk (no
> LVM or RAID), using all SD NTSC feeds from ivtv.  Deleting a one-hour
> recording was typically okay, even if multiple streams were writing,
> but deleting (say) 10G (e.g., a 5hr recording) was not.  I don't
> recall exactly where the breakpoint was (I could look it up in my
> notes), but it also varies based on how many other streams you're
> writing, and whether the database is doing anything, etc.  Of course,
> the breakpoint might be different on your machine, and if something
> decides to delete too many 1-hour recordings at the same time (such as
> expiration), you'll be in the many-GB case and lose in the same way.
>
> This is simple to reproduce---just use dd or something to create files
> of arbitrary sizes, and time their deletion.  That'll give you an idea
> of the work the filesystem is going through.  Then (if you want) try
> recording n streams to that filesystem while doing a deletion.  There
> will come a point in ext3fs where the amount of time it takes for the
> deletion to happen will exceed your RAM buffers, and you'll lose data.
> (In JFS, the deletions are in the tens of milliseconds or less no  
> matter
> how large the file is, so that won't happen there.  Ditto XFS, I  
> believe.)
>
> If you queued all deletions until no writers were writing---or due to
> write in the next 5-10 seconds---this would be fine, but you might
> wind up waiting hours for a deletion to happen.  You'd have to be
> pretty sure you wouldn't run out of space in the meantime.
>
>

Thank you for a good explanation, now everything I have read makes  
sense, including the "bilgewater". I guess I have never noticed a  
problem because I almost never record anything longer than a 1-hour  
show, and thus never delete anything longer either. I also have a lot  
of RAM.

It seems pretty stupid for a deletion to block all writes, but I'm  
sure there is a reason this was done, I can see possible problems if  
it were otherwise.

I guess every filesystem is a compromise based on performance,  
safety, size of code, speed of code execution and other factors, and  
what is good for storing characters from a newswire is different from  
what's best for storing HD video streams.

Thanks again.