[mythtv] 'Special' storage groups and free space calculations

[Raymond Wagner]

On 12/8/2012 13:15, Alexander Fisher wrote:
> On 8 December 2012 17:30, Raymond Wagner <raymond at wagnerrp.com> wrote:
>> On 12/8/2012 08:45, Alexander Fisher wrote:
>>>
>>> That would still leave the livetv filesystem.
>>> Why is the livetv filesystem free space being included in the free
>>> space statistics?  It is already listed as a special group (and I've
>>> verified you can't schedule a recording to go to this group).  Does
>>> this make it a special special group?:)
>>
>>
>> The only groups that get included in freespace calculations are ones that
>> get recorded to. That means 'Default', 'LiveTV', and any other non-special
>> groups you define on your own. If no livetv paths are defined, livetv gets
>> stored in the default paths. If no default paths are defined, it drops back
>> to a compiled in '/mnt/store/'.
>
> I didn't define the 'Streaming' storage group myself.  It was part of
> the mythbuntu 12.04 install.
>
>  From http://code.mythtv.org/trac/changeset/7e1a77063e3f002c118149c91d2e5e86b3bb2c5a/mythtv
> 'Live Stream files are written to a 'Streaming' Storage Group if
> one is defined, otherwise they will go in ~/.mythtv/tmp/hls.'
>
> So I think that 'Streaming' should be added to the list of special
> groups as it's not a user-defined group that is created to hold
> recordings.

While HLS temporary files are related to recordings, they are currently 
outside the scope of the AutoExpirer, so how much room they take is 
irrelevant to storage available for recordings.

There are plans to internally restructure the database schema in a 
manner that will support multiple files tied to a single recording, 
potentially in time for 0.27. This would allow multiple transcoded 
copies of a recording to remain known to MythTV, and would also allow 
HLS transcodes to be tracked as well. At such time, what you suggest 
would need to be revisited.

> As for the Live TV group free space being included in the total, I'm
> still not sure I understand the rational.  I'm probably just
> misunderstanding something. :)
> but...
> What if default group filesystems are almost almost full, but the Live
> Tv group happens to have terabytes available on a separate filesystem?
> It will appear as if there is plenty of room for new recordings.  But
> this won't be true as scheduled recordings can't be to the Live Tv
> group.

Autoexpiration is a secondary task. The storage available to each 
configured storage path is calculated, and that is used by the Scheduler 
to determine where a specific recording is going. Once that is decided, 
the AutoExpirer determines whether content needs to be cleared off that 
path, or if there is sufficient free space to handle the recording.

Looking through the code, it seems any custom paths defined for the 
LiveTV group to _not_ get added to the free space calculations, so they 
are not matched up with other paths on the same partition. I'd like to 
get confirmation from someone who knows more about the process, but it 
appears the AutoExpirer assumes anyone using a specific LiveTV path will 
have put it on it's own partition.

> But when autoexpire kicks in, I better not also have any snapshots in
> existence!  I can envisage a situation where the auto expire code
> deletes all recordings :(

To be honest, I don't see any reason to snapshot recordings for that to 
be a problem. In any case, it's just going to be something that cannot 
be recommended, for many of the same reasons that mythlink.pl does not 
support hardlinks, and requires softlinks.

A bigger issue I could see would be a zpool with a number of zvols, and 
someone defining "copies" to be something greater than on, to make a 
pseudo-mirror. The actual recording throughput would be multiplied 
potentially several times what MythTV otherwise expects it to be. Since 
the AutoExpirer makes decisions on how frequently it needs to run, based 
off how fast the current recording load should consume disk space, it 
could potentially result in you running out of free disk space before 
the AutoExpirer runs again.

> I'd be interested to know how many other people are using zfs and
> whether they've run into this.

I run ZFS, but my recordings are still handled as if I weren't, with 
each drive on it's own zpool, and one path defined to Default for each 
drive. Now that I think about it, I could have defined them all to a 
shared zpool, and let ZFS load balance instead of MythTV, but the end 
result isn't much different.