[mythtv-users] Storage directory management across SSD/HD

[Ram Ramesh]

On 3/10/23 21:32, Stephen Worthington wrote:
> On Fri, 10 Mar 2023 11:39:20 -0600, you wrote:
>
>> All,
>>
>>    I have about 300G in SSD space and several TBs in spinning disk
>> RAID1. Currently, I do not use SSD for recording at all, as it is very
>> small compared to HDs. Is it possible to add the SSD to storage
>> directory and somehow make mythtv use it like a cache? I can write small
>> cron job  to balance the contents overnight so that only recent
>> recordings are present in SSD.  The only question is how do I tell
>> mythtv to use only SSD for new recordings as much as possible?
>>
>> It appears that most of the storage management methods in mythtv are
>> focused on distributing recordings based available space rather than the
>> type of storage or user preferred targets. Can I do something better? I
>> put all spinning disks in one storage group and SSD in default storage
>> group. I manage moving files overnight (or more frequently) to keep the
>> balance. This seems like a possibility.
>>
>> 1. What happens if SSD fills up? Does mythtv automatically go to
>>     another storage group to record the new show even if the recording
>>     rule says "default" is the storage group?
> No, it will delete from the SSD to create space.  The recording rule
> for each recording tells which storage group to use.  But I believe
> MythTV is not selective when it comes to keeping the minimum storage
> space available on all recording storage groups - if any storage group
> falls below the minimum setting (around 20 Gibtyes), recordings will
> be deleted to get it back over the minimum, even if that storage group
> is not actually being used to record to.  So you have to be careful
> when a destination drive is getting full that you do not copy a file
> that would take the free space below that limit.  If you do, shortly
> after that MythTV will delete recordings from that drive until its
> free space is above the limit again.  Even if that drive is never used
> directly for recording to, as long as it is in a storagegroup other
> than one of the special ones (eg music, pictures), it will always have
Do you mean mythtv will delete files even if they are not marked to be 
deleted? I expect that it will not record any new shows when disk is 
full. Deleting a recording that I want to keep is not very user 
friendly.  I never had this happen as I always had enough space. May be 
I never had to deal with it and therefore assumed this never happens.
> recordings deleted to keep its free space above the limit.
>
>> 2. Also, what happens when my script tries to move a file from SSD to
>>     HD and in the process there are duplicate copies of the same file in
>>     multiple storage directories when I trigger a lookup/watch from a
>>     frontend?
> You need to make sure your script does not create duplicates.  You do
> that by copying to temporary file names (eg add .tmp on the end). Then
> delete the original recordings and rename the .tmp files to the
> correct name.
>
>> 3. Even worse, what happens when a file moves from SSD to HD while I am
>>     watching a show?
> Do not do that, if possible.  It may sort of work, as I believe that
> when Linux attempts to delete a file that is in use, it will not
> delete it immediately but will wait until it is no longer in use.  But
> the ancillary files (.png screenshot files, for example), will not be
> in use and then will be on a different drive from the corresponding
> .ts file.  And you may wind up with two copies of the .ts file until
> the original gets deleted, if it does actually get deleted.
>
> In my mythsgu Python program where I move recordings from my 7
> recording drives to my (currently 6) offline archive drives, I have
> put a lot of work into deciding whether MythTV is busy and only moving
> files when it is not.  If it suddenly becomes busy, the file being
> moved gets its copying paused (or cancelled) until MythTV is idle
> again.  It does that using several different techniques.  Firstly, it
> looks at the inuseprograms table in the database.  If that table is
> not empty, MythTV is busy.  Second, it looks at the upcoming
> recordings list, and stops doing things well before the next scheduled
> recording time.  Thirdly, it traps all event messages and whenever it
> sees one, re-evaluates the busy status.
>
> http://www.jsw.gen.nz/mythtv/mythsgu
>
> See the "mythsgu fill" and "mythsgu pack" commands for the relevant
> code.
>
> Note that mythsgu was originally written a long time ago in Python 2
> and has been converted to Python 3.  It uses the now obsolete asyncore
> package to do asynchronous operations, and asyncore is now deprecated
> and will be removed shortly, so I need to rewrite it to use asyncio.
>
> I am not sure why you feel the need to have recordings go to your SSD.
> As long as hard drives are spinning at the time, starting a recording
> or playback is very fast and there is not much gained by using an SSD.
> And recording such huge amounts of data to any SSD will drastically
> shorten its lifetime and you will have to be replacing it often.  And
> making sure you have proper monitoring and notification as it reaches
> the end of its lifetime.  SSDs these days usually have a specification
> for their lifetime in "terrabytes written" (TBW).  That should make it
> fairly easy to estimate the lifetime based on how many Gbytes you
> write to recordings every day - all other use will be minimal compared
> to that traffic.
>
> If you are finding that you are recording too many things at once and
> your hard drives can not cope, then you should probably just add
> another recording hard drive.  Each hard drive will be able to record
> at least two recordings at once and probably three.  Using RAID1
> means, of course, that you would need to add two drives.  Most of us
> do not bother with RAID for recordings.
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums:https://forum.mythtv.org
Ok, it looks like mythtv is not really setup for this type of dynamic 
movement.  I had no intention of doing any dyncamic movement, but was 
asking just to see what is possible. I only had plans to rebalance at 
2am when I know nothing is being recorded or watched.

After reading through your response, I do not think I will attempt to do 
anything to add the SSD. Typically, my SSDs get so under utilized that 
they never wear out. I thought this will balance out when I get the urge 
to upgrade due to size.

I have one machine that is always on. So, I run everything on it. It is 
my backup machine as well as a front & backend. Thus it has a lot of 
storage and RAID to be failure tolerant.

Regards
Ramesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-users/attachments/20230310/4a8cadb6/attachment.htm>