[mythtv-users] Two-backends with two-frontends on one machine?

[Michael T. Dean]

On 07/11/2007 02:57 PM, Billy Macdonald wrote:
> On 7/11/07, Michael T. Dean <mtdean at thirdcontact.com> wrote:
>   
>> On 07/11/2007 11:27 AM, Steven Adeff wrote:
>>     
>>> 1) RAID 1 seems like overkill, why not go with RAID5? I've got a 6
>>> drive RAID 5 device that keeps up with my 4 HD tuner backend with two
>>> HD capable frontends no problem.
>>>       
>> And, in the future (i.e. 0.21 and up), RAID may in fact work against
>> you.  With Storage Groups, you can set up your system to write each
>> recording to a separate disk (provided enough disks), thereby
>> significantly reducing fragmentation and seek issues.  I have 4 HDTV
>> capture cards and more drives than capture cards and have configured
>> Myth so that it will only write two shows to the same disk if that's the
>> only way to record the show (i.e. if all the other disks are full).
>> After using it this way for some time, I can say I'll never use a
>> multiple-disk (RAID or LVM with multiple physical volumes) configuration
>> with Myth again.
>>     
> RAID 1&5 provide redundancy that I don't think your solutions
> addresses.

Right--because, IMHO, it's just TV.  I'll settle for "lose one disk
means lose one disk of data."

>   Your solution seems to be solving more of a problem that
> isn't there IMO.

Says someone who hasn't yet had the fun of defrag'ing a severely
fragmented Myth recordings drive.  I put all my SDTV on one disk when I
moved to an all HDTV solution, and by the time I had watched/deleted 75%
of the SDTV, the new HDTV recordings on that drive were so fragmented
that Myth couldn't play back shows without bad prebuffering pauses
because the disk couldn't read all parts of the shows in real-time. 
When I decided to defrag (by copying to another disk,
deleting/formatting the old disk, and moving back) it took 10 hours to
copy the 200GB from that drive (and less than 2 hours to put it back). 
The reason it got so fragmented was because my 1GB/hr SDTV recordings
(most of which were half-hour recordings) were so small compared to my
4-8.5GB/hr HDTV recordings, creating issues for the filesystem's
allocation strategy.

To help prevent similar problems, I moved the HDTV back onto the disk
first, then wrote a huge file of zeros, leaving just enough space for
all my remaining SDTV at the end of the disk, then deleted the file of
zeros.  Since all my new recordings are HDTV (and, therefore, of the
same order of magnitude), I'm unlikely to have similar issues in the
future, but for those recording HDTV and SDTV, fragmentation can be an
issue.

Also for those who use multiple capture cards but only one disk (or
volume or RAID), recordings will be fragmented when 2 or more occur at
the same time (because there's no way for Myth to pre-allocate the space
for the first recording, so chunks of each recording get placed in an
alternating fashion across the disk).  Yeah, I know about how modern
filesystems try to prevent fragmentation by allocating blocks around
those required for a write in case the file grows, but I don't know of
any filesystem that will allocate blocks allowing a file to grow to 8GiB...

But, don't believe me.  I wouldn't.  See this paper detailing the
results of a study by Philips Research (in which they even mention
MythTV :).
https://ols2006.108.redhat.com/reprints/denijs-reprint.pdf

While the paper suggests the effects of fragmentation are not an issue
on a PVR system, remember the filesystem tests were run /after/ choosing
the most appropriate test scenario (i.e. after they determined what
factors would lead to unreasonable fragmentation, etc.).  And, the paper
arrives at this conclusion using /averages/ for post-test filesystem
performance--not instantaneous performance figures.  On my bad
filesystem described above, the average performance was significantly
better than required (i.e. took 10 hours to read (while rewriting to
another drive on another bus) about 70 hours of TV; however, portions of
the HDTV shows could not be read in real time).  (Note, my test harness
was a Myth box in use for 3 years--rather than pvrsim--and I only have
one test result, so I can't claim my results are conclusive. :)

The paper also explains that you should also specify a per-filesystem
minimum free space (equal to about 5% of total space) after which
auto-expire should begin deleting recordings to prevent excessive
fragmentation.  If you run the disk full and let autoexpire delete only
enough recordings to write the current recording, that recording will
tend to have the same number of fragments as the number of files on the
disk.  When you write to a large LVM or RAID array with all your
(hundreds of) recordings, that can be a significant performance
penalty.  Guess I may have to make a modification to "Extra Disk Space
(in Gigabytes)" to allow it to be specified per filesystem now that
Chris Pinkham has added storage groups allowing the use of multiple
filesystems.

>   I have 4 HDtuners, and 3 pvr250's.  All 7 tuners can
> record to my LVM volume (not striped, 3 disks) and I can play back an
> HD recording at the same time.  And these are cheap IDE drives that I
> got on sale at Best Buy.  Now being able to store recordings in more
> than one location does provide flexibility which I believe is a good
> thing.  But I'm not sure it's really needed for performance.
>   

I agree.  It's not needed--until it's needed.  :)  Eventually, you may
encounter issues.

Although my friends' TiVo's have small (almost unnoticeable) pauses
occasionally, I don't want any of them on my Myth box, so I choose to
minimize fragmentation and seek issues (among other things) on my
system.  You're welcome to choose a different approach.

> And this past weekend I learned to really like LVM.  I mistakenly made
> my storage partition reiserFS, which I see in the docs is not good for
> myth.  I was able to shrink my filesystem, shrink the lvm volume,
> create a new volume, move some files, shrink, grow, copy, shrink,
> grow, copy until all the files were moved to the new filesystem where
> the old filesystem once lived.  All in all, pretty cool I thought.

LVM is good.  I just won't ever put more than one PV in an LV for Myth
ever again.  Have you ever had one PV fail in an LV?  :)

Mike