[mythtv-users] SSD's for daily myth recordings storage and part time RAID for capacity.

[Michael T. Dean]

On 04/22/2016 12:22 PM, Simon Hobson wrote:
> David Scammell wrote:
>
>> I'm considering using a moderate sized ssd for daily myth recordings so that the larger RAID 5 array drives can be spun down for most of the day, perhaps only brought up during main telly watching hours or when anyone is about. I'm expecting the machine to record all the daily kid's programs with the main RAID array shut off to save on power and heat.
>>
>> I'd like to copy (rsync/whatever) those recent recordings from SSD to RAID only when the RAID is spun up. I will be able to clear down sufficient free space on the SSD for the next days recordings so as to not need to expire anything straight way, at least at first but I'm struggling to work out how myth will be able to expire recordings from the RAID array once that gets full, as it will be available only part time.
> This one has been stewing away at the back of my mind ...
>
> One approach I can think of is simply to harness Myth's abilities directly. Set an appropriate value for how much space you want to leave on the raid array, and let Myth expire stuff as required. But I see a few wrinkles in the plan ...
>
> 1) If Myth isn't recording to the array (or more precisely, to a storage group located on the array) then it'll not autoexpire/delete to make space on it (AIUI). Workarounds I can think of are : does Myth have a call to trigger auto-expire/freeing of space;

No.

>   or setup a dummy recording onto the raid array to trigger it.
>
> 2) It's so long since I set it, I can't remember if the required space variable is global or per-filesystem, or something else.

Global.

>   I think it's global

Right.

>   so if you (for example) wanted to keep 20G free on the raid array, you'd also be forced to "waste" 20G of space on the SSD*.
>
> 3) There could potentially be situations where you record more in a day than you have free space for on the raid array (this kind of goes hand in hand with 1 & 2). So you might need to copy/move files one at a time, triggering auto-expire/deletion as you go.
> I wonder if it's possible to set up a dummy recording which has very low bandwidth, start it before you start moving files, then move files one at a time, pausing if the free space becomes less than the size of the next file. Myth will remove something creating free space, and then the script can carry on. Finally, when the script has finished, stop the recording.
>
> As long as the "free space to keep" value is larger than the largest recording file you might ever have, then this should work. IMO, the best way to avoid complications is to avoid duplicating what Myth does, and let it do it itself - that way you are not subject to database schema changes, and you don't need to worry about accidentally "corrupting" the database.
>
> For bonus points, I suppose you could periodically check the free space on the SSD*, and it it's getting low then start another run to move files.


It may be easier just to "manually" expire the programs off the RAID to 
make room.  As you mentioned, unless mythbackend is actively recording a 
show to the file system in question, it will never perform any 
expiration.  Therefore, if MythTV is never actually recording to the 
RAID, recordings will have to be manually deleted (to the Deleted 
recording group and re-deleted to remove from disk) to actually be 
removed from the MythTV system and from the disk.

As far as choosing what to remove, you can start with the Live TV 
recordings (anything in the LiveTV--pretty sure the DB name is LiveTV, 
but the display will show 'Live TV'--recording group) followed by the 
Deleted recordings (anything in the Deleted recording group).  An easy 
way to remove them is with 
https://www.mythtv.org/wiki/Delete_recordings.py using the argument 
--recgroup=Deleted or --recgroup=LiveTV.

As far as undeleted, but auto-expirable shows go, you can ask 
mythbackend for the list of shows (in expire priority order) with 
mythbackend --printexpire .  However, to automate it, you'd want to use 
something like the Python bindings to write a little script. 
Unfortunately, the Python bindings provide a convenience method for 
retrieving expirable recordings in MythBE.getExpiring() ( 
https://github.com/MythTV/mythtv/blob/master/mythtv/bindings/python/MythTV/methodheap.py#L228 
), but since it uses sorted=True, it seems like it will re-order the 
programs into time order (rather than return them in auto-expire 
priority order). If so, you could manually submit the ProgramQuery 
without specifying sorted=True (or specifying sorted=False, the 
default)--and if you confirm this is the case, please submit a bug 
report, ideally with a patch that changes the sorted=True on line 228 to 
False.

Better may be to use MythXML's getExpiring() method (as shown in the 
attached example script--attached so that mail/archives don't mess with 
spacing/break script), which does seem to return the recordings in 
auto-expire priority order.  But, if you do take this approach, please 
also do a quick test of the commented-out MythBE.getExpiring() line 
(uncomment it and comment the MythXML.getExpiring() line) to see if I'm 
correct that the mythBE.getExpiring() is buggy and breaks ordering (and 
then submit that bug report I mentioned :).

Once you have the list of recordings, you can go through them and delete 
them (once or twice).  This is left as an exercise for the reader (any 
reader who feels like helping to make a manual_expire.py script :).  If 
you do come up with a script, please post it at 
https://www.mythtv.org/wiki/Category:Python_Scripts .

Note, too, that the Python-bindings approach (asking the backend for a 
list of expirable recordings) would actually list the short Live TV 
programs, normal Live TV programs, and Deleted programs, so you could 
just make a "manual expire" script that handles everything (and you 
wouldn't need to use Delete_recordings.py, as mentioned above).  You 
could even write it to accept some arguments, such as an amount of 
storage to free, and let it start deleting (to Deleted group, if not 
already, then again to remove from the Deleted group), until it's 
expired programs whose total size is greater than the specified size.

Thanks to anyone who actually does test the MythBE.getExpiring() for me 
and, if appropriate, submits a bug report and/or patch.

Mike
-------------- next part --------------
#!/usr/bin/env python
import MythTV
# Does MythBE().getExpiring() return shows in time order for you?
# (different from MythXML.getExpiring())
#for prog in MythTV.MythBE().getExpiring():
for prog in MythTV.MythXML().getExpiring():
     print "{0.title:<40} {0.subtitle:<30} {0.starttime}".format(prog)