[mythtv-users] Storage directory management across SSD/HD

[Stephen Worthington]

On Fri, 10 Mar 2023 11:39:20 -0600, you wrote:

>All,
>
>   I have about 300G in SSD space and several TBs in spinning disk 
>RAID1. Currently, I do not use SSD for recording at all, as it is very 
>small compared to HDs. Is it possible to add the SSD to storage 
>directory and somehow make mythtv use it like a cache? I can write small 
>cron job  to balance the contents overnight so that only recent 
>recordings are present in SSD.  The only question is how do I tell 
>mythtv to use only SSD for new recordings as much as possible?
>
>It appears that most of the storage management methods in mythtv are 
>focused on distributing recordings based available space rather than the 
>type of storage or user preferred targets. Can I do something better? I 
>put all spinning disks in one storage group and SSD in default storage 
>group. I manage moving files overnight (or more frequently) to keep the 
>balance. This seems like a possibility.
>
> 1. What happens if SSD fills up? Does mythtv automatically go to
>    another storage group to record the new show even if the recording
>    rule says "default" is the storage group?

No, it will delete from the SSD to create space.  The recording rule
for each recording tells which storage group to use.  But I believe
MythTV is not selective when it comes to keeping the minimum storage
space available on all recording storage groups - if any storage group
falls below the minimum setting (around 20 Gibtyes), recordings will
be deleted to get it back over the minimum, even if that storage group
is not actually being used to record to.  So you have to be careful
when a destination drive is getting full that you do not copy a file
that would take the free space below that limit.  If you do, shortly
after that MythTV will delete recordings from that drive until its
free space is above the limit again.  Even if that drive is never used
directly for recording to, as long as it is in a storagegroup other
than one of the special ones (eg music, pictures), it will always have
recordings deleted to keep its free space above the limit.

> 2. Also, what happens when my script tries to move a file from SSD to
>    HD and in the process there are duplicate copies of the same file in
>    multiple storage directories when I trigger a lookup/watch from a
>    frontend?

You need to make sure your script does not create duplicates.  You do
that by copying to temporary file names (eg add .tmp on the end). Then
delete the original recordings and rename the .tmp files to the
correct name.

> 3. Even worse, what happens when a file moves from SSD to HD while I am
>    watching a show?

Do not do that, if possible.  It may sort of work, as I believe that
when Linux attempts to delete a file that is in use, it will not
delete it immediately but will wait until it is no longer in use.  But
the ancillary files (.png screenshot files, for example), will not be
in use and then will be on a different drive from the corresponding
.ts file.  And you may wind up with two copies of the .ts file until
the original gets deleted, if it does actually get deleted.

In my mythsgu Python program where I move recordings from my 7
recording drives to my (currently 6) offline archive drives, I have
put a lot of work into deciding whether MythTV is busy and only moving
files when it is not.  If it suddenly becomes busy, the file being
moved gets its copying paused (or cancelled) until MythTV is idle
again.  It does that using several different techniques.  Firstly, it
looks at the inuseprograms table in the database.  If that table is
not empty, MythTV is busy.  Second, it looks at the upcoming
recordings list, and stops doing things well before the next scheduled
recording time.  Thirdly, it traps all event messages and whenever it
sees one, re-evaluates the busy status.

http://www.jsw.gen.nz/mythtv/mythsgu

See the "mythsgu fill" and "mythsgu pack" commands for the relevant
code.

Note that mythsgu was originally written a long time ago in Python 2
and has been converted to Python 3.  It uses the now obsolete asyncore
package to do asynchronous operations, and asyncore is now deprecated
and will be removed shortly, so I need to rewrite it to use asyncio.

I am not sure why you feel the need to have recordings go to your SSD.
As long as hard drives are spinning at the time, starting a recording
or playback is very fast and there is not much gained by using an SSD.
And recording such huge amounts of data to any SSD will drastically
shorten its lifetime and you will have to be replacing it often.  And
making sure you have proper monitoring and notification as it reaches
the end of its lifetime.  SSDs these days usually have a specification
for their lifetime in "terrabytes written" (TBW).  That should make it
fairly easy to estimate the lifetime based on how many Gbytes you
write to recordings every day - all other use will be minimal compared
to that traffic.

If you are finding that you are recording too many things at once and
your hard drives can not cope, then you should probably just add
another recording hard drive.  Each hard drive will be able to record
at least two recordings at once and probably three.  Using RAID1
means, of course, that you would need to add two drives.  Most of us
do not bother with RAID for recordings.