[mythtv-users] Little OT: RealTime Parallel Multi-PC Transcoding

[Raymond Wagner]

On 6/15/2010 17:05, belcampo wrote:
>> Do you actually need to have realtime h264, or do you just want to 
>> speed up transcodes?  If the latter, is there any reason why you 
>> cannot just run two transcodes in parallel?
> If I want a speedup as you say I should run 2 transcodes in parallel, 
> how would I do that, independantly, on the same source.
> As mentioned in the 1st post multithreading is very inefficient. Also 
> see http://saintdevelopment.com/codecs/bfs-vs-cfs.txt which I found 
> here http://x264dev.multimedia.cx/
> After 2 Cores the efficiency drops dramatically. Where my approach 
> would /should be more efficient, at least I think and hope so.

I would have to see more information surrounding that test to draw any 
conclusions from it.  Certainly on a quad core processor, running more 
than four threads is going to provide negligable, if any, gains.

The Core2Quad running DDR2 memory is starved for bandwidth on a number 
of applications.  A couple years back, we did testing with a Q6600, 
using a turbomachinery solver that we knew was good for efficient 
parallelization across hundreds of machines.  Two processes ran with 75% 
speedup over one, while four processes only ran with 20% speedup over 
two.  DDR3, and especially triple channel i7s should significantly 
improve these numbers.

Video encoding using 'slices' is what is called 'ridiculously 
parallel'.  Aside from some preprocessing of the raw video, the encoding 
process is batch.  The encoding threads are completely independent, 
working on their own data set, and there is no need for communication 
between them.  Ignoring bottlenecks on shared resources like memory or 
communication buses, the encoder should scale linearly with the number 
of cores.

>> You would want the video cuts to be tens of seconds long to reduce 
>> data and startup overhead.
> Could you be more specific on this ?

Starting up processes is expensive.  The startup and teardown of 
something complex like x264 is probably going to be a couple seconds.  
If you're only going to be encoding chunks of a couple seconds long, 
that is a huge amount of overhead.