[mythtv-users] Little OT: RealTime Parallel Multi-PC Transcoding

[Nick Rout]

On Wed, Jun 16, 2010 at 12:14 PM, Raymond Wagner <raymond at wagnerrp.com> wrote:
> On 6/15/2010 17:05, belcampo wrote:
>>>
>>> Do you actually need to have realtime h264, or do you just want to speed
>>> up transcodes?  If the latter, is there any reason why you cannot just run
>>> two transcodes in parallel?
>>
>> If I want a speedup as you say I should run 2 transcodes in parallel, how
>> would I do that, independantly, on the same source.
>> As mentioned in the 1st post multithreading is very inefficient. Also see
>> http://saintdevelopment.com/codecs/bfs-vs-cfs.txt which I found here
>> http://x264dev.multimedia.cx/
>> After 2 Cores the efficiency drops dramatically. Where my approach would
>> /should be more efficient, at least I think and hope so.
>
> I would have to see more information surrounding that test to draw any
> conclusions from it.  Certainly on a quad core processor, running more than
> four threads is going to provide negligable, if any, gains.
>
> The Core2Quad running DDR2 memory is starved for bandwidth on a number of
> applications.  A couple years back, we did testing with a Q6600, using a
> turbomachinery solver that we knew was good for efficient parallelization
> across hundreds of machines.  Two processes ran with 75% speedup over one,
> while four processes only ran with 20% speedup over two.  DDR3, and
> especially triple channel i7s should significantly improve these numbers.
>
> Video encoding using 'slices' is what is called 'ridiculously parallel'.
>  Aside from some preprocessing of the raw video, the encoding process is
> batch.  The encoding threads are completely independent, working on their
> own data set, and there is no need for communication between them.  Ignoring
> bottlenecks on shared resources like memory or communication buses, the
> encoder should scale linearly with the number of cores.
>
>>> You would want the video cuts to be tens of seconds long to reduce data
>>> and startup overhead.
>>
>> Could you be more specific on this ?
>
> Starting up processes is expensive.  The startup and teardown of something
> complex like x264 is probably going to be a couple seconds.  If you're only
> going to be encoding chunks of a couple seconds long, that is a huge amount
> of overhead.

In terms of looking at an existing implementation, you might want to
take a look at the code in dvd:rip http://www.exit1.org/dvdrip/

dvd:rip is a perl dvdripper that uses transcode for transcoding. The
point is that it does have a cluster mode where you can distribute
your encoding round a lan. You may get some ideas from their code, or
be able to hack it to do your bidding.