[mythtv-users] Managing storage groups intelligently

[Stephen Worthington]

On Tue, 1 Sep 2020 11:34:44 -0500, you wrote:

>I come from mythtv 0.27/0.28. So, I apologize if this is already handled 
>in later versions.
>
>Is it possible to prioritize storage groups when recording? I plan to 
>add a 1TB SSD to the storage groups and would like recordings to choose 
>that over spinning disk whenever possible.
>
>If possible, I like to try this after the above is resolved. Manually 
>move files between SSD and spinning disk (both are storage groups) to 
>keep only recently created/accessed file in SSD and also make sure at 
>lease 100GB space is left for any new recording. I will do this using 
>overnight cron job.?? Since mythtv finds recorded programs as long as 
>they are present in one of the storage groups, file moves should not 
>affect look up for media files.?? Do you see issue with my approach?
>
>Yes, I already investigated bcache/dmcache/lvmcache etc., and was 
>advised that they are either overkill or unreliable or ineffective in 
>mythtv environment.
>
>Regards
>Ramesh.

You can do what I do, which is to have my recordings go to the
partitions in my "Default" storage group, but also have another
storage group I call "Archives".  In my case, the "Default" drives CMR
drives and most of the "Archives" drives are shingled (SMR) drives,
which are unsuitable for recording to.  I currently have 7 drives on
"Default" so I am safe to do up to 14 recordings at once before I have
to worry about getting damaged recordings.  There are 5 drives on
"Archives".  When playing back recordings, MythTV will find the files
in any storage group, but will only record to the storage group
specified in the recording rule.  The default storage group used when
creating new recording rules is set in the "Default (Template)"
template rule.  So when my recording drives are getting full, I move
recordings from them to my "Archives" drives until I have enough free
space again.

It is safe to move files about in the storage groups as long as they
are not in use by MythTV.  Since it takes a while to move recording
files as they are relatively large, you can have a problem if you
start moving a file and then MythTV needs to use it during the move.
So the safe way to do a move is to copy the file to its new location
using a different name that MythTV will not recognise as a recording
file, and then when it has been copied, delete the source file and
rename the destination file to the correct name.  If you want to
prevent MythTV from using the file while it is being copied, you can
rename the source file before moving it, so MythTV will think it is
just a missing recording file.

You also need to worry about performance problems.  When MythTV is
recording or playing back, it is using much less than the maximum
sequential speed of a drive.  When you are copying or moving a file,
that operation normally takes the full sequential bandwidth available
on the slower of the two drives.  So if MythTV is using that drive at
the same time, it may not get access to the file(s) it is using fast
enough.  On playback, that results in stuttering.  For recordings, it
causes damaged recordings due to buffers failing to be written to disk
in time.

As a result of these problems and probably some others I have not
thought about, it is better to not do non-MythTV use of its recording
drives at the same time as MythTV is using them, if you want to do
anything using high bandwidth.

In my case, I have written myself a Python program (mythsgu) that
moves recordings from "Default" to "Archives".  I just run it manually
when I need to, but you could run it automatically if you want.  It
moves the oldest recordings first.  It is set up so that it gets
events from MythTV and checks the status of MythTV to ensure that it
will not move files when MythTV is busy.  If it is moving a file and
it gets an event and subsequently discovers (by checking the
inuseprograms table) that MythTV has become busy (perhaps someone
started to play a recording), it will pause its move operation until
MythTV is no longer busy.  It keeps track of when the next scheduled
recording will happen and pauses operations when it gets near to that
time.  It is only capable of discovering MythTV activity for
recordings, as MythTV does not report when it is playing video files
as it does for recordings.  That does not matter for me as all my
video files (and pictures and music files) are on a different set of
drives from my recording files.

Currently mythsgu is Python 2 - I have not upgraded to MythTV v31 yet.
I am just in the process of converting it to Python 3 after which I
will be doing the upgrade.  The latest Python 2 version (0.7) is here:

http://www.jsw.gen.nz/mythtv/mythsgu

It is not very well documented as to how to set it up, so if you would
like to try it, please let me know and I can help with that.

I believe I also saw in a recent post here that it is possible to
assign priorities to the various drives in storage groups.  This does
not have a proper user interface and does not seem to be very well
documented, but there was detail about how to set it up in that post.
I only skimmed over the details, but I think you could tell the
scheduler to record only up to a certain number of recordings at once
on a drive and after that it would use a lower priority drive.

One other consideration which probably does not apply to you is that
when mythsgu finishes, my recording drives normally have very
unbalanced amounts of free space.  That means that the drives that
have the most free space will be preferentially chosen to be used for
new recordings.  I prefer that all my recording drives have roughly
equal free space to prevent this, so after I run mythsgu, I run an
older script I wrote that balances the free space by moving randomly
chosen recordings from the recording drive with the least free space
to the one with the most free space:

http://www.jsw.gen.nz/mythtv/balance_storage.sh

This script stops itself ahead of the next scheduled recording, but is
not as sophisticated as mythsgu in being able to detect most MythTV
activity and pause when it happens.  I am intending eventually to move
the balancing function to become part of mythsgu so that it will have
the same level of sophistication.