[mythtv-users] backend storage performance needs?

Tue May 19 15:03:36 UTC 2020

On Tue, May 19, 2020 at 10:04 AM Stephen Worthington <
stephen_agent at jsw.gen.nz> wrote:

> On Tue, 19 May 2020 06:44:41 -0400, you wrote:
>
> >Just an opinion request here.  When I built my current backend it was an
> >old motherboard with lots of SATA II (3Gb/s) ports. I was worried about
> >the need to have 2 recording drives defined so 2 simultaneous recordings
> >would go to different drives.
> >
> >I also wanted RAID mirrors so I used 4-2TB drives with mdadm RAID 1. A
> >lot to maintain as drives do fail and RAID volumes have to be replaced
> >and rebuilt. Replacing and rebuilding has occurred on average of once a
> >year.
> >
> >So the question is, if I bought a new motherboard with SATA III (6Gb.s)
> >ports and new SATA III high performance hard drives would I really need
> >to define 2 recording drives? i.e. use 2-4TB drives in RAID 1 mirror
> >configuration.
> >
> >My tuner is a PCIe Hauppauge WinTV Quat HD. I have 4 tuners defined with
> >each tuner having a maximum recordings of 2. But I've never recorded
> >more than 4 channels at once with some back-to-back programs on the same
> >channel.
> >
> >With the current speed of processors, serial connections, and hard
> >drives do I still need 2 recording drives defined?
> >
> >Since SSDs and RAID is not recommended, I avoided that in the discussion.
> >
> >Jim A
>
> There is no exact answer to just how many recordings at once you can
> do with one hard drive.  There are quite a number of factors to
> consider.  For example, if you are recording from all four tuners, and
> you have back-to-back programmes scheduled for all four tuners, you
> can suddenly be recording eight programmes at once for the overlap
> period.  On top of that, you may also be playing back one recording
> per frontend.  And as you get more locations on the disk where there
> is active writing (or reading), that means the heads will be spending
> little time on station before they have to move again to the next
> active location.  It is not a linear problem - going from four to five
> active locations uses far more disk resource (available head movement)
> than going from one to two active locations.
>
> Then you get the problem of the filesystem.  Each recording will get
> to a point where it needs more space to be allocated.  So the heads
> need to move from where the recording is to the system areas to find
> and allocate that space.  That may be fine when it happens for one
> recording, but if it happens for all four or all eight at once,
> suddenly the heads are moving great distances back and forth and may
> be too late getting back to where the recording is being written.
> Filesystems vary as to how they handle that - some have the system
> areas spread out across the disk to allow shorter head movement.  Some
> have all the system areas at one end of the disk.
>
> Then there is the organisation of the disk into tracks and cylinders.
> A disk with more platters and fewer tracks per platter has less
> distance to move the heads between things.
>
> Then you need to consider what may happen when the drive is getting
> full.  If the filesystem is not good at keeping the free space
> contiguous, then the heads will have to move much longer distances
> between the files.
>
> Modern fast hard drives can have tremendous sequential write speeds -
> those have been increasing all the time as the rotational bit
> densities increase.  But the speeds of head movement have not
> increased in proportion.  So they tend to be the limiting factor, not
> the write speed.  And the drive manufacturers are much less
> forthcoming about stepping rates and the like - they like to advertise
> the super sequential write speeds.
>
> And then there is the cache - a large RAM cache and a good caching
> algorithm (where the head movement is minimised) can make a big
> difference when the heads have to move away from a recording to update
> the system areas.  The updates for all the open files can be delayed
> and then all done at once, rather than the heads moving back and
> forth.
>
> Personally, I am quite conservative with the number of drives versus
> number of simultaneous recordings.  Part of that is because I have
> several older drives (3 x WD Green/Red, 6-7 years old), but also
> because when I started out doing lots of recordings at once, I did get
> occasions where I had all the recordings I was doing simultaneously
> fail or be corrupted due to exceeding the ability of my (then two)
> drives to handle the load.  So now I have seven recording drives, and
> am upgrading them slowly to much more modern and higher capacity.  I
> plan on no more than two recordings at once per drive.  I do regularly
> (at least once a week) do 10-12 recordings at once, and with that
> setup I have never had a problem, even when the load increases sharply
> (potentially doubles) during back-to-back overlaps.  I only use normal
> drives, not RAID.  I can not afford the extra costs of doing RAID.
>
> Except for their noise, my favourite drive is now the Seagate Exos
> series enterprise helium drives.  I have an ST12000NM0007 (12 Tbytes)
> and an ST14000NM0018 (14 Tbytes) in my main MythTV box now.  They are
> amazingly fast, and are good at handling transaction processing where
> the data can be spread across the entire disk.  And they are much
> cheaper than anything else with the same performance - they are
> actually cheaper here than the high grade NAS rated drives.  As
> enterprise drives, I expect them to have a long lifetime, but I have
> yet to find out just how long they will last - only time will tell.  I
> have been very happy with all the WD drives just keeping on going for
> 8+ years.  And Seagate did make some horribly bad drives with very
> high failure rates for a while - I had 5 of those drives all die early
> (< 2 years) and only one of them that lasted, and that only for about
> 5 years.  Fortunately, they were mostly so bad that they died while
> still under warranty, and under New Zealand law I was able to refuse
> to accept a replacement with the same type of drive as they clearly
> were not of an acceptable standard of durability.  I got very good at
> recovering the data onto the replacement drive, so I only lost maybe
> 10-15 recordings from all of that, as the mode of death was to get
> more and more difficult to read sectors, rather than the entire drive
> just collapsing and all the data being gone.
>
> If you can afford them, even better than the Exos drives are the
> Hitachi (now WD) helium drives.  Unfortunately, their price is
> significantly higher for the same capacity - maybe 40% more.  But the
> Hitachi engineers have always made very reliable drives.  The two
> three Tbyte ones I retired when I upgraded to my Exos drives had been
> in service 24/7 for eight and nine years respectively.  And one is
> back in service on my mother's MythTV box as she ran out of space.
>
> Do not buy the cheap "desktop" drives - they are unsuitable for 24/7
> operation and will die early if used like that.  You want at least NAS
> rated drives.
>

Thanks for your thoughts.

I'm wondering if instead of RAID 1, I should just use 1-2TB SSD as a
recording drive and then once a night do a cronjob that rsync's the
recordings to an external drive for backup. I really only have about 800GB
of actively use space for recordings because once my wife and I have seen
them we delete them. Not building a library.

I was doing the rsync part in preparation to upgrading to mythtv V31 in
case I could not just move the current RAID.

Also I know that I can record 4 HD programs while watching a recording all
at once because I did this as a test on my RPi4, HDHR Quatro, and
USB3-to-SATAIII 1TB SSD.

The RAID needs are because I also use my backend as a NAS for other data,
mostly backups.
Then my drive list would be:

   - 1-2TB SSD as boot drive and Mythtv Recordings
   - 2-2TB HDD as RAID 1 mirror for general NAS use for home computers.
   - 1 external 2TB HDD drive to backup recordings daily.

That would take head movement out of the equation and at most I'd lose one
day's recordings.

Jim  A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-users/attachments/20200519/27daa2dd/attachment.htm>