[mythtv-users] rebalance deleted recordings

[Simon Hobson]

"Brian J. Murrell" <brian at interlinx.bc.ca> wrote:

> On Thu, 2014-03-20 at 13:44 +0000, Simon Hobson wrote: 
>> 
>> Ah, IMO that is you problem.
> 
> Not really.  The same problem happens even if you expire recordings
> right away.  But let's discuss that below.  First I will respond to your
> other comments...
> 
>> If a recording is deleted, why keep it around ?
> 
> I would ask instead, why expire it before you need the space?  What
> benefit do you get from expiring it before you need the space.  I will
> counter with the benefit of delaying expiration until space is needed is
> that it extends the window in which you can pull a deleted recording
> back.

Just out of interest, is your garage stacked high with old newspapers going back years "just in case" ?

>> Keep it for a few days (allows for reversal of accidental deletions),
> 
> But why not extend that window as much as possible?

Because it causes this sort of problem ?

>> However, there is part of your issue which is correct - if a drive is almost full and has no deleted recordings on it then you'll have the problem.
> 
> Right.  This is where the opening comment picks up.  Deleting things
> right away is just a deferral of this same problem.  Ultimately you need
> to rebalance because you have a disk of recordings that you are not
> ready to delete/expire yet and you have disks that have free/available
> (where available means it might be full, but with some expirable -- i.e.
> deleted recordings on it) space and you want to balance that
> available/free space.
> 
> The only real difference is that with the "expire-when-you-need-to"
> method, you need to consult the Myth database to determine where
> "available" (i.e. vs. "free" which can be determined with a simple "df")
> space is.

What I find is that under *my* normal usage, Myth tends to fairly well balance use so I don't have a problem.

It is a known problem (in that it's come up here before) that Myth doesn't always "get it right" when disks are full/nearly full. I suppose the *correct* fix for that is to expand on Myths drive usage logic so that it will correctly only expire deleted recordings until there are none of those left, and then only expire non-deleted recordings in strict priority order (ie not taking account of which disk they are on). I suspect that catering for all corner cases might be "quite complicated" and cause a conflict between the need to balance I/O load vs the need to obey the discard priorities.
At present, which disk to put a recording on is determined before any space is freed up, and then space made for it - that is probably the root cause.

But there is another detail : You've told Myth that all those recordings you haven't watched can be expired if space is needed - if you changed that then it wouldn't expire them (but I suspect you'd get disk full problems instead).

>> I aim to keep more free space than the minimum I've told Myth to keep
> 
> But why?  All you are achieving by doing that is reducing your
> deleted-recording recovery window.  I can't see how that is useful given
> that Myth will manage "available" space.

I find it useful in that it allows Myth to work properly. To me, having a "as long as possible" wastebasket for recordings I've watched (almost all of my viewing is nothing more than time-shifting) isn't "useful".

>> I find it hard to believe that if you weren't keeping a load of deleted recordings around, that the recordings wouldn't balance themselves better over time - meaning you'd need to manually rebalance once after clearing some of the old deleted recordings.
> 
> Well, you better start believing it.  Whether I keep deleted recordings
> around and let Myth expire them as it needs space for new recordings has
> absolutely no bearing on this problem.

My mileage varies a lot from yours. If Myth doesn't have to expire stuff *at recording time* then it does correctly balance recordings across drives - tending towards using the emptiest one first. Thus, as I watch and delete stuff, space gets created across all drives*. Given that proviso - that when you watch something you delete it (eventually) - I find that things tend to sort themselves out.

* Statistically, unless you do something odd, then the recordings you delete are going to be spread across the drives. One "odd" thing would be to fill a drive up with old recordings that you don't get round to watching and deleting, but add an extra drive - so you only watch and delete "new" recordings that went onto the new drive.

To put some numbers on that, when I put a new drive in a few weeks ago, I had about 40-50G of free space on the (then) only 3TB drive. That drive still has 40-50G of space free on it, but the other drive (which I upgraded from a 750 to 3TB) has between 100 and 200G of extra recordings on it**. So Myth is automatically favouring the less full drive, and my "watch and delete" process is keeping free space up on the fuller drive even though I'm adding to the total quantity of recordings.

** Can't remember exactly how much free space it had, and I've put some other stuff on there as well.



TL;DR summary.
IMO, if you set Myth to delete deleted recordings after a certain period, and possibly do a one-off move of some recordings to rebalance free space, then you'll find it will "just work". True, that is working around a problem, but it only costs you some time window for recovering deleted recordings.