[mythtv-users] Fragmentation On Recording Disk?

[Michael T. Dean]

On 04/04/2007 03:31 AM, James Warden wrote:
> wow! I need to check this tonight!
> It's funny, I thought disk frag was not an issue with linux!
AIUI:

Fragmentation is reduced in various ways by modern filesystems (even 
NTFS is significantly better than FAT32).  Short of running a perpetual 
defrag utility (which would cause more performance loss than 
fragmentation) or having orders of magnitude more storage than is 
required (and a filesystem that's willing to waste it), there's no way 
to completely prevent fragmentation.

> guess I was wrong ...
> Seriously, are these figures reasonable ? I mean, can one trust what 
> xfs_db spits out ? I never considered checking disk frag on any of my 
> linux boxes, and I've had them for quite a while. Are disks formatted 
> with ext3 showing the same fragmentation factor ? And does one know 
> the growth rate of this factor ? (what are the relevant variables : 
> disk size ? technology ? file system ?)

Myth starts a recording at 8:00pm, it begins writing to that file.  The 
filesystem has allocated a certain amount of space for the file, and--to 
reduce fragmentation--has allocated blocks around the initial blocks for 
file growth.  What the filesystem doesn't know is that Myth will 
continue to write to that file for 30 minutes, 1 hour, 2 hours, ...  The 
filesystem doesn't allocate gigabytes of additional storage for every 
file it creates, so any other files created during that recording are 
likely to get written where they prevent the recording from being 
allocated in one contiguous block.

When you have one capture card and a large amount of contiguous space 
available on your storage and nothing else writing to the storage disk, 
fragmentation will be minimal.  When you have multiple capture 
cards--each starting a recording at the same time (or even one after the 
other but still overlapping) and writing to the same 
disk/partition/volume/whatever, there's going to be significantly more 
fragmentation.

However, things like Myth's (to be available in 0.21) Storage Groups 
(which allow writing to multiple directories) will help /if/ you 
structure your storage appropriately.  Shouldn't we fix it more?  Well, 
short of preallocating the space before a recording, there's not much 
more Myth can do.***  This means it's up to users to defrag every once 
in a while.  My preferred method is to copy files from one disk to 
another, make a new filesystem on the original, and copy them back.

Mike

*** If Myth were to preallocate, the only completely reliable way of 
doing so is to write zeros to the disk before recording begins.  To see 
how much of an issue this would be, run top in one terminal and watch 
the wa (I/O wait) value near the top of the screen as you run:

dd if=/dev/zero of=~/testlargefilepreallocation bs=1M count=2048
(for an SDTV show at 2GiB/hr)

or

dd if=/dev/zero of=~/testlargefilepreallocation bs=1M count=8192
(for an HDTV show at 8GiB/hr)

in another terminal.  Make sure you adjust the path of "of" to some 
location that has space (and to which you have permissions).

And, to make things interesting, try doing anything else that touches 
the disks while doing this (even just an "ls ~").  If people think that 
the MySQL access while recording is an issue, imagine what this would do 
to recordings.