[mythtv-users] remaining disk space / delete slowly

[Simon Hobson]

James Linder <jam at tigger.ws> wrote:

>> No, the slow deletes are simply to allow MythTV to spread the I/O 
>> requirements for deleting a file (generally a very large, multi-gigabyte 
>> file) over a long period of time (IIRC, about 2min/GB) rather than 
>> telling the OS to delete the entire, huge file and letting it force all 
>> that I/O to occur "immediately". Therefore, if your system has other I/O 
>> requirements (such as, for example, writing out recording files for new 
>> recordings that are currently in progress, or writing information to 
>> your MySQL database (eg, about in-progress recordings), or reading 
>> information from your MySQL database (eg, to refresh the information 
>> about your existing/remaining recordings), that (much, much) 
>> higher-priority I/O can occur without having to wait on the unimportant, 
>> low-priority I/O associated with deleting a recording.  This can be very 
>> important when deleting a bunch of huge files because it's possible for 
>> the new recording information to grow too large for the system to keep 
>> in memory before it's written to disk at which point information  is 
>> lost (and files are corrupted--including recording files and even 
>> potentially your MySQL database data files).
> 
> Can someone who KNOWS offer some explanation, this sounds like gibberish
> 
> A file is an inode plus house heeping to say where in the hierarchical tree (AKA the file system) it appears
> A file consists of 
> 1 sector of pointers to data sectors
> 1 sector of pointers to a sector of pointers to data sectors
> 1 sector of pointers to a sector of pointers to a sector of pointers to data sectors
> 
> during delete nothing happens to the sea of data sectors

Ah, but it does (sort of) ...
When you delete the file, that sea of data sectors is returned to the free blocks list. I am but a novice in such matters, but AIUI in ext3 the free block list isn't just a bitmap of free blocks, it's a tree structure of free spaces sorted or indexed by size. So not only do the blocks have to be put back on the free blocks list, there will be amalgamation where possible to create a small list of large blocks as much as possible. Not only that, but files get "soft allocated" the whole of a free space even though they won't be using it.
Imagine a conceptually empty disk, no files on it, there's one free space which is the whole of the disk. Add a file, the free space gets cut in half, and one of those halves is used for the file. Add another file and it gets allocated the whole of the other half. Add a third file, the system takes half of the space from one of the existing files (obviously only taking unused space), and so on. As you add files, the system keeps splitting free space as needed.
In practice, the system doesn't start with one big free space, it has a relatively small list of fairly large chunks.
The reason for giving a file more free space than it needs is to minimise fragmentation as the file grows. In some older filesystems, a file gets created, then the next one gets put in the very next blocks on the disk. When the first file gets extended, it must now be split and get the next free blocks after that - file fragmentation, slower access, and a nice market for "defrag" utility vendors :-)

Now look at file deletions. Assuming we've had a lot of small files which we delete, there's loads of small allocations freed up. Some of these will be adjacent (either to other newly released chunks, or to already free chunks), and so can be merged into a smaller number of larger blocks. So it's not just a mater of shoving all the newly free blocks into the tree (which in itself may take some effort), we'll need to take some smaller free chunks out of the tree and put the merged larger chunks in. This latter bit is important as it allows defragmentation of free space as files are deleted.

So I think that's where the I/O storm when deleting either lots of large files or one very big file comes from - managing the tree of free space. I think the bit about running out of memory (really it means the write cache has used all currently free memory) and corrupting things is bogus - the system will effectively pause at the that point with processes blocked in "wait for I/O" (wio) state until the disk subsystem catches up with the writes. If you look at the output of top, you'll see a field for wio - depending on the workload, if that goes significantly above zero then you have a system which is constrained by it's disks.

Hence the slow delete option. Instead of throwing all this shuffling around of tree nodes at the system in one go - which may cause pauses in real-time data flow (recording and/or playback), it slowly truncates the files so that the I/O load is spread and issue is worked around.