[mythtv-users] Wiki: Balanced Disk I/O

[Michael T. Dean]

On 08/11/2015 11:48 AM, Peter Bennett (cats22) wrote:
> Hi MythTV experts
>
> I am updating the wiki and wondering about the Balanced Disk I/O setting.
>
> https://www.mythtv.org/wiki/Storage_Groups#Balanced_Disk_I.2FO
>
> The above page recommends if you are doing more than 2 recordings at 
> once to use this setting to get recordings put on separate file 
> systems to avoid buffer overruns. I am thinking of changing that 
> recommendation.

It's still one of the (2) best options, not because the drive can't keep 
up but because:

a) it creates files with the least amount of fragmentation (this is the 
primary benefit)***
b) when hard drives fill up, it doesn't favor a single hard drive for 
every single recording, leading to all new recordings going to the same 
drive and all expired recordings coming from that same drive (meaning 
that before long, the system is expiring new recordings when there are 
tons of older--possibly Deleted and/or Watched--recordings that could be 
expired instead if the system would just choose another disk).  If using 
Balanced Free Space or Balanced Percent Free Space, this is exactly what 
happens when disks fill up and autoexpire kicks in.  See 
http://www.gossamer-threads.com/lists/mythtv/users/556505#556505 (note, 
too, that neither Combination, nor Balanced Disk I/O will avert this 
issue if the user never records multiple shows at the same time--in 
which case, the user /must/ rebalance his/her recordings)****.
c) there's a bug in Balanced Free Space (and likely Balanced Percent 
Free Space), where it may choose to record to a directory that's not 
available on some remote backend, when users don't use network file 
systems and have multiple backends and have some directories available 
on the master backend that aren't available on remote backends--this 
isn't a common configuration.  (No need to mention the bug in the wiki, 
but this is yet another good reason to steer people away from Balanced 
*Free Space--because it may save that one user in a hundred (or possibly 
even less common) with that very unusual configuration from missing 
recordings due to non-existent directories.)

As Simon mentioned in another reply:

On 08/11/2015 12:27 PM, Simon Hobson wrote:
> Without looking, I believe I have the combination option set that balances disk I/O but favours the filesystem with the most free space. Ie one recording will go on the filesystem with the most free space, two will go across the two filesystems I have. IMO this makes the best use of the available disks I/O, while also reasonably favouring best use of available disk space.

This is true, but it's actually true for both Balanced Disk I/O as well 
as Combiantion.  The difference is that Combination also tries to prefer 
local file systems over network ones.  IMHO, either Balanced Disk I/O or 
Combination is the best choice.  Since Combination is identical to 
Balanced Disk I/O when there are no network file systems and since 
Combination is actually beneficial when there are network file systems, 
I'd recommend Combination for everyone.

FWIW, Combination used to be the default (and, there was good reason for 
it--as it's most appropriate for all systems). Then users complained 
that after they put in their new 2TB drive when their old 2TB drive 
filled up, MythTV was still recording to the old drive and expiring 
shows when they had plenty of space on the new drive.  So, the default 
was changed to Balanced Free Space so that users who put in empty drives 
wouldn't be surprised.  However, this setting (er, actually, the 
algorithm) is broken for anyone who doesn't have at least one drive with 
a significant portion of free space (therefore, it's a broken design, 
made available at popular request, but it is not a bug).  The proper 
solution for users with full drives who add new drives is to manually 
move some portion of recordings from full file systems to less-full file 
systems to balance recordings--ideally some "representative" sample of 
recordings, meaning about equivalent in age and/or priority and/or 
"LiveTVness"/"Watchedness"/"Deletedness".  That said, you can make the 
argument that setting the default Storage Group Disk Scheduler to 
Balanced Free Space /is/ a bug (and I won't argue that it's not).

Sorry for all the cryptic wording, which you will have to try to figure 
out how to write up in an understandable manner.  However, I sincerely 
appreciate your doing so--since I've been unable to figure out how to 
explain these issues for years, now. :)  Since I'm sure I haven't 
explained it well, feel free to ask more questions, and I'll be watching 
for replies.

Mike

*** There is no way***.5 for MythTV to pre-allocate any appropriate 
amount of space for the recording since each channel is likely to 
transmit at a different bitrate and, regardless, even if we had a guess 
of how much space the file will need, file systems all handle space 
allocation differently such that the only reliable way to pre-allocate 
space for a large file is to write 0's to the disk, then rewrite the 
file in that location.  However, as may be guessed, writing 6GB of 0's 
to disk (per recording) all at the top of the hour would suck up all I/O 
resources and right at the time when they're needed most (when setting 
up for writing the actual recording data, as well as recording metadata 
and other database data about the recording)--not to mention the I/O 
resource usage when truncating the excess space at the end of the 
recording, right as we're trying to pre-allocate for new recordings.
***.5 OK, so it would be possible for MythTV to pre-pre-allocate space 
by writing large zero files over long time periods to fill up the file 
system with "future" recording files, such that there are some available 
when it starts the recording, but as much as people complain about the 
lack of visibility of how much space is actually available with Deleted 
recordings, I'm sure having future recordings on there would be even 
more hated. :)

**** Recordings can be rebalanced by simply moving them around on disk 
using file system tools or Storage directories can be rebalanced by 
pre-deleting (from disk--by telling MythTV to delete the recording when 
it's already in the Deleted Recording Group) or by pre-expiring (by 
going to the System Status|Auto-Expire screen in mythfrontend and 
choosing to delete recordings from there) recordings from full file 
systems, and/or archiving shows to either an "Archive" (non-writable) 
Storage Group or to Video Library to make room on their full file 
systems.  See 
http://www.gossamer-threads.com/lists/mythtv/users/535888#535888 and 
http://www.gossamer-threads.com/lists/mythtv/users/516083#516083 . I 
highly recommend having some wiki pages (or sections of the Storage 
Group page) on Archive Storage Groups and balancing recordings across 
file systems.