[mythtv-users] Building a library.....chews up space!!

[Raymond Wagner]

On 3/28/2011 09:39, Another Sillyname wrote:
> I take broadcast HD source material and transcode it using my own
> scripts to about 25-35% of it's original size, if you include the
> advert cuts the savings are huge.

H264 was only ever designed to handle about double the compression 
efficiency of MPEG2, and if using comparable quality compressors from 
both, that estimate tends to hold true.  If you are clipping commercials 
as well as transcoding, then final output of 25-35% of the original is 
reasonable.  If you are transcoding the whole source video, either your 
source is using an inefficient compressor, or you are going to have 
significant loss.

> Transcode time for Dog Soldiers on a different backend mapped to the
> same drive mappings was 2 hours 33 minutes for a two pass transcode

There is no reason to use a two-pass transcode.  In a two-pass encode, 
the first pass is an analysis, measuring the relative compressibility 
throughout the video.  These measurements allow a quantizer to be chosen 
that will accurately achieve the desired file size.  You only care about 
file size when encoding to a limited medium, one movie on a CD, or two 
movies on a DVD, or four on your iPod.  When this is going on a 7TB disk 
array, who cares how large it is.  Just choose a decent quantizer (18-20 
for x264), and let the encoder do whatever it wants.

Regardless, running a two-pass HD encode even at 70% realtime is pretty 
absurd, unless you're on some beastly multiprocessor machine.  Likely, 
you're running a motion estimation algorithm that is a lot lighter 
weight than the current default in x264.

> I often run multiple transcodes simultaneously on the backend and the
> time performance hit is only about 15% per extra transcode (limit is 3
> for HD transcodes though, 6 for SD).

That makes no sense.  H264 is multithreaded, and x264 will default to 
using enough slices/threads to use all the computational resources of 
the machine.  Running two transcodes simultaneously will result in each 
running at slight less than half speed.  Three would result in less than 
one third speed for each.  Are you actually using h264?  Are you 
intentionally limiting the slice count for unknown reasons?

> While I appreciate drives are relatively cheap putting in RAID5 to
> ensure redundancy is a little bit more complicated and growing the
> library therefore gets more complicated still.

All it takes is a bit of planning.  That 7TB array in software RAID5 
would cost you maybe $400 to put together with 2TB drives.  A few years 
down the line, just replace the whole thing with 5-6TB drives.  Growing 
is complicated, yes, but ultimately unnecessary.

> Anyway has anyone else started to build a decent library?

At least on the IRC channel, there are several dozen of us with 
comparable arrays, and some several times larger.  Personally, I'm 
around 10TB, and as mentioned, throwing everything in the base directory 
with gallery view and pgup/pgdn works well enough for my needs.  If you 
have a keyboard, you may also want to try using the incremental search 
capability in MythVideo.

http://www.mythtv.org/wiki/MythVideo#Incremental_Search