[mythtv-users] Yet another silly action to drive the mythbox crazy

[Michael T. Dean]

On 08/20/2014 01:34 PM, Nicolas Krzywinski wrote:
> Hi,
>
> recently, I did just another epic silly mistake which had a huge 
> impact on the functionality of my myth machine.
>
>
> HERE comes the story (head on to the end if this is of no interest to 
> you):
>
> It all started with the program guide telling me the airdate of "Spy 
> Game" - one of my favorite movies which I can recommend to anyone, not 
> least because of the great soundtrack.
> The broadcasting channel was ZDF HD (the second german channel under 
> public law). Those channels are notorious for their huge delays when 
> airing. When setting up the recording rules, I always set a big 
> additional recording time at the end for recordings on those channels, 
> therefore.
> Now ... at that time I was in a hurry. I assume to having entered one 
> zero too much for the recording overtime, resulting to 300 minutes 
> instead of 30.
> But this I did not knew, until my myth machine started to stumble 
> suddenly, letting mythfrontend and mythwelcome die slowly, mute 
> logfiles, ssh login shell crying some crazy stuff and mysql turn to 
> some kind of disoriented state (what? I am supposed to have tables ...?).
> "Quickly" (after three reboots, including cold restart with no 
> recovery) I found out that the system disk was full.
> Apparently I did silly stuff like this before, as my fingers 
> automatically knew to execute the joker command "apt-get clean" to get 
> some megs instantly. This allowed everything to start up as usual.
> Then the investigation started, revealing a 46 gb recording named "Spy 
> Game" with something more than 7 hours recording time in high 
> definition. Roughly calculated back, this conforms to the mistake in 
> setting the recording overtime, assumed above.
> Looking on the solid state disk serving this partition being only ~128 
> gb, minus the system files, minus some recordings protected from 
> deletion, it becomes clear that there was some hard time for the 
> backend to manage the storage.
>
> Though this is a rare case with a great influence of misuse, there is 
> a simple solution to avoid such situations, described with the 
> following...
>
>
> HERE comes the proposal for improvement:
>
> Always preserve some free space on the system partition for the 
> background stuff being able to proceed their work.
> This condition for reserving some free space should even stop 
> recordings that would violate the free space otherwise.
>
>
> What do you think about that?

Our solution for that is two fold:

First, MythTV should never, ever be configured to record to the system's 
root file system specifically because filling it up can cause serious 
system-wide problems.  Partitions will allow even single-disk users to 
configure appropriately to avoid recording to the root file system.

Second, MythTV automatically expires recordings to make room for 
in-progress recordings.  If you're recording to one file system for 
performance or whatever and then moving to another, you may not have any 
other recordings on that file system to be expired (or if you disable 
auto-expire on all the recordings on a given file system)--in which case 
see solution 1, above, which will protect your system and note that any 
failed recordings are due to your writing to an "unsafe" file system 
(meaning one on which MythTV can't make additional room when necessary).

If someone wants to write all the code necessary to deal with "user 
misconfigured system and now we have no storage", feel free.  Make sure 
you deal with all the rest--like ensuring that if we're currently 
expiring recordings on the file system, we don't bail out of recordings 
just because of a temporary violation of the free space limit; or 2 
seconds after we detected that we busted the file system free space 
limit and shut down all recordings, some external process deleted files 
so we're no longer in violation of the free space limit, and then 3 
seconds after we try to resume recordings we're again in violation of 
the free space limit (and around and around); or we violated the limit 
and shut down current recordings and later we're supposed to start new 
recordings.

Oh, and be ready to change all your code when you submit the patch and 
someone says, "Wait, this shuts down my recording(s) just because one of 
my 12 file systems is full when it should just switch to recording the 
rest of the file to a different file system (possibly on another host, 
possibly without network file system access--meaning we'd have to stop 
the recording on that backend and restart it on a different one)." Or 
the million other ways a simple idea becomes a huge mess of difficult to 
understand conditions that try to always do the right thing--instead of 
just expecting a proper (or at least better) configuration.  ;)

In other words, I like the idea of failing gracefully if we run out of 
space, but the implementation is very difficult, which is why no one has 
ever done it--well, that and the fact that no 2 people will agree on the 
proper way to deal with it given all the myriad backend(s)/file 
system(s)/network/... configurations used with MythTV.

Mike