[mythtv-users] advice on recording direct to NAS

Stephen Worthington stephen_agent at jsw.gen.nz
Tue May 26 00:39:09 UTC 2015


On Mon, 25 May 2015 23:07:56 +0930, you wrote:

>
>
>> -----Original Message-----
>> From: mythtv-users [mailto:mythtv-users-bounces at mythtv.org] On Behalf
>> Of Stephen Worthington
>> Sent: Saturday, 23 May 2015 10:56 PM
>> To: Discussion about MythTV
>> Subject: Re: [mythtv-users] advice on recording direct to NAS
>> 
>> On Sat, 23 May 2015 04:56:58 +0000, you wrote:
>> 
>> >On Fri, May 22, 2015 at 9:51 PM Mark Perkins <perkins1724 at hotmail.com>
>> >wrote:
>> >
>> >>  Hi all, looking for advice:
>> >>
>> >> Background: I've just lost a second HDD in 6months and its getting
>> >> very, very, very annoying.
>> >>
>> >> I have a NAS that would have capacity to record to, but have not done
>> >> so because of concern for failed recordings due to potential
>> >> inconsistencies in network and / or NAS performance. I run 4 HDD in
>> >> my mythbackend supporting 30tuners (nominal worst case 8 recordings
>> >> at a time due to overlap of pre/post roll).
>> >>
>> >> Does anyone record directly to a NAS folder? How did you share the
>> >> folder with your server? Does it work well? Do temporary spikes in
>> >> network load (for example someone starting a large download or
>> >> streaming a HD movie
>> >> elsewhere) cause problems?
>> >>
>> >> Or am I better recording to local drives and using some sort of cron
>> >> job to move stuff to the NAS periodically?
>> >>
>> >> Or (thinking on the fly) should I abandon the 4 stand alone drives
>> >> and set them up in a RAID array locally?
>> >>
>> >> Any advice gratefully received.
>> >>  _______________________________________________
>> >> mythtv-users mailing list
>> >> mythtv-users at mythtv.org
>> >> http://lists.mythtv.org/mailman/listinfo/mythtv-users
>> >> http://wiki.mythtv.org/Mailing_List_etiquette
>> >> MythTV Forums: https://forum.mythtv.org
>> >
>> >
>> >I've previously recorded to my syncology NAS, and just recently last
>> >Friday moved back to using it (I moved away from it not for performance
>> >reasons, but space reasons. I moved back after I upgraded my NAS drives
>> >and moved my backend to a Raspberry Pi 2). I share the folder off my
>> >NAS via NFS and have set MythTV to keep 100GB free space (so I can move
>> >files to the NAS without worrying about free space).
>> >
>> >I've not seen any performance issues recording to the NAS. I've got a
>> >HD Homerun Prime.
>> 
>> I would recommend doing some performance testing on your NAS, then
>> calculating the Ethernet traffic also before using it for recording.
>> There are plenty of NAS boxes out there that perform quite badly, and I
>have
>> seen performance issues with recording directly to hard drives when I only
>> had two, so I would be very careful of using something slower than hard
>> drives.  Generally, a home made NAS box using an old PC will perform
>better
>> than a commercial one intended for the home market - NAS boxes often
>> have quite cheap small CPUs and have lots of performance bottlenecks, and
>> sometimes pretty bad software.  An old PC tends to have a CPU and other
>> hardware that matches its drives performance capabilities rather better
>than
>> a cheap NAS does.  The downsides of using an old PC as a NAS are higher
>> 24/7 power use (older PCS use more power than newer ones), and maybe
>> fan noise.
>> 
>> A good way to do a real life test would be to set up a special recording
>group
>> pointing to the NAS drive(s) and then just create some one off recording
>> rules set to record to that storage group.  See how many recordings at
>once
>> can be done safely, and try some other use of the NAS at the same time.
>> Make sure to try starting a number of recordings at exactly the same time,
>as
>> the workload of a filesystem can be substantially higher on file creation
>(or
>> deletion), with the heads have to move to the various parts of the drive
>> where it keeps directory and file information, as well as to the actual
>> recording files themselves.  And remember that HD recordings use lots more
>> data than SD ones, often four times as much, so they generate more disk
>> activity and more Ethernet traffic.
>> 
>> A NAS should be fine for moving recordings to after they finish recording.
>> Just set up a separate storage group for the NAS drive(s).
>> MythTV will be able to play back files moved to that storage group, but
>will
>> only record to the Default storage group unless you tell it to use another
>> storage group in the recording rule.  Then a job run automatically from
>cron
>> or maybe at the end of a recording can move files off the recording
>drive(s)
>> to the NAS.  The only gotcha I can see is if someone tries to play a file
>that is
>> currently being moved or the script tries to move a file that is being
>played or
>> recorded, so a script that ran in the middle of the night might be a good
>idea,
>> if your household is free of night owls.  Once the script has checked that
>the
>> file is not a current recording or playback file, a good safeguard would
>be to
>> have it rename the file it is currently moving, so MythTV can not find it.
>Then
>> once it has completed the move to the NAS drive, rename it back again.
>> Then there would only be a very short window of opportunity for a clash
>for
>> use of the file.  Once it is renamed, it will show up in mythfrontend
>greyed
>> out with an X icon (or whatever your theme does for missing files).  If a
>file is
>> shown like that, you normally need to page up and page down or something
>> like that to get mythfrontend to refresh its status for the file so it can
>be
>> played.
>> _______________________________________________
>
>
>Thanks for the pointers Stephen. I'm going to try moving recordings over by
>bash script and cron job in the first instance. I'm not much of a programmer
>but I've put together something that I think will do the job, including
>checking whether a job is in progress first before moving the file. I'm
>still yet to test it but if anyone spots any obvious errors don't hesitate
>to let me know. I have left 'echo' in front of the 'mv' commands for posting
>to the list, just to be on the safe side, and the TEST_COUNTER is just used
>to limit executions through the loop while testing. I'm on Ubuntu 14.04 and
>MythTV 0.27.
>
>#!/bin/bash
>
>MYTHDB="mythconverg"
>MYTHUSER="mythtv"
>MYTHPWD="mythtv"
>BASE_DIR1="/media/fs3/recordings/"
>BASE_DIR2="/media/fs4/recordings/"
>NAS_DIR="/media/servertwo/Myth_Recordings/"
>COUNTER=0
>TEST_COUNTER=0
>
>for FILE in $(find ${BASE_DIR1} ${BASE_DIR2} -type f | grep "\.mpg$"); do
>
>    let TEST_COUNTER=TEST_COUNTER+1
>    COUNTER=0
>
>    read -ra vars <<< $(mysql -D$MYTHDB -u$MYTHUSER -p$MYTHPWD -se "SELECT
>CONCAT(chanid, '_', DATE_FORMAT(starttime,'%Y%m%d%H%i%s'), '.mpg') As
>BASENAME FROM inuseprograms")
>
>    for JOB in "${vars[@]}"; do
>        if [[ "$FILE" == *"$JOB" ]]
>        then
>            let COUNTER=COUNTER+1
>        fi
>    done
>
>    if [ "$COUNTER" -eq "0" ]
>    then
>
>        #first move the file to a new name on the same directory so that it
>is no longer available to mythtv
>        echo "mv $FILE ${FILE}_block"
>
>        #need to extract just the original file name
>        FILENAME=$(echo "$FILE" | sed 's/.*\///')
>
>        #need to move the file to the NAS keeping it blocked from mythtv
>        echo "mv ${FILE}_block ${NAS_DIR}${FILENAME}_block"
>
>        #now to return the file in the new location back to its original
>name
>        echo "mv ${NAS_DIR}${FILENAME}_block ${NAS_DIR}${FILENAME}"
>
>        #reset the counter each time
>        COUNTER=0
>    else
>        echo "$FILE is in use, skipping"
>        COUNTER=0
>    fi
>
>    if [ "$TEST_COUNTER" -gt "0" ]
>    then
>        break
>    fi
>
>done

That looks pretty good.  There is still a potential for a timing
problem when MythTV is adding something to the inuseprograms table, as
your script could be renaming the file at the same time as MythTV is
adding the entry to the table.  That would not be a problem with
recording files - I would presume that MythTV would create the
inuseprograms row before actually starting to create a recording file,
so your script would not be able to see the file before the row was
created.  But for files that someone is starting to play back, there
is a short window of time between when your script has checked the
inuseprograms table and when it has actually renamed the file.  The
way to fix that might be to create an inuseprograms row in the script
before touching the file.  I have never looked at simultaneous use of
mysql rows, but there is probably a way to make something like that
work, so that if your script tried to create a row at the same time as
MythTV, it would get back an appropriate error, or MythTV would,
depending on which row creation succeeded.  But it would likely need
input from one of the devs to tell us exactly how that might work.  I
do not think the problem would cause any fatal errors - in the worst
case you would likely just wind up with two copies of the file, one on
the NAS and one where it was before.  In the mean time, running your
script at a time of day no-one is likely to be starting playback is
likely to be sufficient in real life.  You should add an error check
on the rename mv command, so that if it fails due to the file being in
use, you can abort the script before attempting to move the file.

I believe that mythutil has an option that can be used to display
messages on the frontends, so it might be useful to display a message
at the start of the script saying that no-one should start to play
anything, and one at the end saying it was all clear.


More information about the mythtv-users mailing list