[mythtv-users] Two-backends with two-frontends on one machine?

[Billy Macdonald]

On 7/11/07, Michael T. Dean <mtdean at thirdcontact.com> wrote:
> On 07/11/2007 02:57 PM, Billy Macdonald wrote:
> > On 7/11/07, Michael T. Dean <mtdean at thirdcontact.com> wrote:
> >
> >> On 07/11/2007 11:27 AM, Steven Adeff wrote:
> >>
> >>> 1) RAID 1 seems like overkill, why not go with RAID5? I've got a 6
> >>> drive RAID 5 device that keeps up with my 4 HD tuner backend with two
> >>> HD capable frontends no problem.
> >>>
> >> And, in the future (i.e. 0.21 and up), RAID may in fact work against
> >> you.  With Storage Groups, you can set up your system to write each
> >> recording to a separate disk (provided enough disks), thereby
> >> significantly reducing fragmentation and seek issues.  I have 4 HDTV
> >> capture cards and more drives than capture cards and have configured
> >> Myth so that it will only write two shows to the same disk if that's the
> >> only way to record the show (i.e. if all the other disks are full).
> >> After using it this way for some time, I can say I'll never use a
> >> multiple-disk (RAID or LVM with multiple physical volumes) configuration
> >> with Myth again.
> >>
> > RAID 1&5 provide redundancy that I don't think your solutions
> > addresses.
>
> Right--because, IMHO, it's just TV.  I'll settle for "lose one disk
> means lose one disk of data."
>
> >   Your solution seems to be solving more of a problem that
> > isn't there IMO.
>
> Says someone who hasn't yet had the fun of defrag'ing a severely
> fragmented Myth recordings drive.  I put all my SDTV on one disk when I
> moved to an all HDTV solution, and by the time I had watched/deleted 75%
> of the SDTV, the new HDTV recordings on that drive were so fragmented
> that Myth couldn't play back shows without bad prebuffering pauses
> because the disk couldn't read all parts of the shows in real-time.
> When I decided to defrag (by copying to another disk,
> deleting/formatting the old disk, and moving back) it took 10 hours to
> copy the 200GB from that drive (and less than 2 hours to put it back).
> The reason it got so fragmented was because my 1GB/hr SDTV recordings
> (most of which were half-hour recordings) were so small compared to my
> 4-8.5GB/hr HDTV recordings, creating issues for the filesystem's
> allocation strategy.
>
> To help prevent similar problems, I moved the HDTV back onto the disk
> first, then wrote a huge file of zeros, leaving just enough space for
> all my remaining SDTV at the end of the disk, then deleted the file of
> zeros.  Since all my new recordings are HDTV (and, therefore, of the
> same order of magnitude), I'm unlikely to have similar issues in the
> future, but for those recording HDTV and SDTV, fragmentation can be an
> issue.
>
> Also for those who use multiple capture cards but only one disk (or
> volume or RAID), recordings will be fragmented when 2 or more occur at
> the same time (because there's no way for Myth to pre-allocate the space
> for the first recording, so chunks of each recording get placed in an
> alternating fashion across the disk).  Yeah, I know about how modern
> filesystems try to prevent fragmentation by allocating blocks around
> those required for a write in case the file grows, but I don't know of
> any filesystem that will allocate blocks allowing a file to grow to 8GiB...
>
> But, don't believe me.  I wouldn't.  See this paper detailing the
> results of a study by Philips Research (in which they even mention
> MythTV :).
> https://ols2006.108.redhat.com/reprints/denijs-reprint.pdf
>
> While the paper suggests the effects of fragmentation are not an issue
> on a PVR system, remember the filesystem tests were run /after/ choosing
> the most appropriate test scenario (i.e. after they determined what
> factors would lead to unreasonable fragmentation, etc.).  And, the paper
> arrives at this conclusion using /averages/ for post-test filesystem
> performance--not instantaneous performance figures.  On my bad
> filesystem described above, the average performance was significantly
> better than required (i.e. took 10 hours to read (while rewriting to
> another drive on another bus) about 70 hours of TV; however, portions of
> the HDTV shows could not be read in real time).  (Note, my test harness
> was a Myth box in use for 3 years--rather than pvrsim--and I only have
> one test result, so I can't claim my results are conclusive. :)
>
> The paper also explains that you should also specify a per-filesystem
> minimum free space (equal to about 5% of total space) after which
> auto-expire should begin deleting recordings to prevent excessive
> fragmentation.  If you run the disk full and let autoexpire delete only
> enough recordings to write the current recording, that recording will
> tend to have the same number of fragments as the number of files on the
> disk.  When you write to a large LVM or RAID array with all your
> (hundreds of) recordings, that can be a significant performance
> penalty.  Guess I may have to make a modification to "Extra Disk Space
> (in Gigabytes)" to allow it to be specified per filesystem now that
> Chris Pinkham has added storage groups allowing the use of multiple
> filesystems.
>
> >   I have 4 HDtuners, and 3 pvr250's.  All 7 tuners can
> > record to my LVM volume (not striped, 3 disks) and I can play back an
> > HD recording at the same time.  And these are cheap IDE drives that I
> > got on sale at Best Buy.  Now being able to store recordings in more
> > than one location does provide flexibility which I believe is a good
> > thing.  But I'm not sure it's really needed for performance.
> >
>
> I agree.  It's not needed--until it's needed.  :)  Eventually, you may
> encounter issues.
>
> Although my friends' TiVo's have small (almost unnoticeable) pauses
> occasionally, I don't want any of them on my Myth box, so I choose to
> minimize fragmentation and seek issues (among other things) on my
> system.  You're welcome to choose a different approach.
>
> > And this past weekend I learned to really like LVM.  I mistakenly made
> > my storage partition reiserFS, which I see in the docs is not good for
> > myth.  I was able to shrink my filesystem, shrink the lvm volume,
> > create a new volume, move some files, shrink, grow, copy, shrink,
> > grow, copy until all the files were moved to the new filesystem where
> > the old filesystem once lived.  All in all, pretty cool I thought.
>
> LVM is good.  I just won't ever put more than one PV in an LV for Myth
> ever again.  Have you ever had one PV fail in an LV?  :)
>
> Mike

Not yet, though I thought I did with all the reiserfs errors I was
getting.  With my filesystem swap, I'm sure it's the opposite of
optimal.  I have no idea how LVM shrunk and grew the partitions.

Our drives are usually around 50-75% full.  The wife gets paranoid if
it gets hire and deletes stuff.  I've really never been concerned
about fragmentation, but your arguements are quite valid.  At this
point it is "good enough" so I've never looked at addressing
performance.  I'm sure I get a prebuffering pause here or there as it
will stutter sometimes, but rarely.  I'd look, but then it would be a
mission to fix them, which I don't feel like doing.