[mythtv-users] Little OT: RealTime Parallel Multi-PC Transcoding

[belcampo]

Nick Rout wrote:
> On Wed, Jun 16, 2010 at 12:14 PM, Raymond Wagner <raymond at wagnerrp.com> wrote:
>> On 6/15/2010 17:05, belcampo wrote:
>>>> Do you actually need to have realtime h264, or do you just want to speed
>>>> up transcodes?  If the latter, is there any reason why you cannot just run
>>>> two transcodes in parallel?
>>> If I want a speedup as you say I should run 2 transcodes in parallel, how
>>> would I do that, independantly, on the same source.
>>> As mentioned in the 1st post multithreading is very inefficient. Also see
>>> http://saintdevelopment.com/codecs/bfs-vs-cfs.txt which I found here
>>> http://x264dev.multimedia.cx/
>>> After 2 Cores the efficiency drops dramatically. Where my approach would
>>> /should be more efficient, at least I think and hope so.
>> I would have to see more information surrounding that test to draw any
>> conclusions from it.  Certainly on a quad core processor, running more than
>> four threads is going to provide negligable, if any, gains.
>>
>> The Core2Quad running DDR2 memory is starved for bandwidth on a number of
>> applications.  A couple years back, we did testing with a Q6600, using a
>> turbomachinery solver that we knew was good for efficient parallelization
>> across hundreds of machines.  Two processes ran with 75% speedup over one,
>> while four processes only ran with 20% speedup over two.  DDR3, and
>> especially triple channel i7s should significantly improve these numbers.
>>
>> Video encoding using 'slices' is what is called 'ridiculously parallel'.
>>  Aside from some preprocessing of the raw video, the encoding process is
>> batch.  The encoding threads are completely independent, working on their
>> own data set, and there is no need for communication between them.  Ignoring
>> bottlenecks on shared resources like memory or communication buses, the
>> encoder should scale linearly with the number of cores.
>>
>>>> You would want the video cuts to be tens of seconds long to reduce data
>>>> and startup overhead.
>>> Could you be more specific on this ?
>> Starting up processes is expensive.  The startup and teardown of something
>> complex like x264 is probably going to be a couple seconds.  If you're only
>> going to be encoding chunks of a couple seconds long, that is a huge amount
>> of overhead.
> 
> In terms of looking at an existing implementation, you might want to
> take a look at the code in dvd:rip http://www.exit1.org/dvdrip/
> 
> dvd:rip is a perl dvdripper that uses transcode for transcoding. The
> point is that it does have a cluster mode where you can distribute
> your encoding round a lan. You may get some ideas from their code, or
> be able to hack it to do your bidding.
I know/knew that solution, I already use that approach when I transcode 
already recorded stuff with 3 PC's with a total of 6Cores. They divide 
the source in 6-pieces and let each core transcode one of the parts.
My current step involves live-encoding, on a recording in progress. But 
thanks for your suggestion.

Henk
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users