[mythtv] Thoughts on encoder farms / distributed capturing and viewing

[Craig McLaughlin]

On Fri, 2002-11-01 at 13:52, Robert Middleswarth wrote:
> >I regularly toss very large datasets around my network here, and often 
> >saturate my 100Mbps switch without too much trouble (note: my 100Mbps
> >switch is actually a cheapo one that only has a 200Mbps backplane).
> >  
> >
> I just tried this at home I xfer 3 7??M files from a Redhat 7.3 Box to a 
> Redhat 8.0 Box and my Xfer rates were 5mb accroding to the FTP Client.  
> NFS might be faster then FTP but still the max I am getting is 5mb over 
> my cheapo switch and intel / reltek 8139 cards.

I strongly suspect that you're seeing 5M BYTES/s, not 5M BITS.  As James
da Silva pointed out, many FTP clients these days report bytes, not
bits.  So 5MB ~~ 40Mbits, not even counting things like TCP overhead. 
One way to check would be to time the transfers and do the math by hand.

> If you use the rtjpeg codex which produces files optmized for Speed not 
> size you aren't going to be able to do more then 2 streams.  If I am 
> rembering correctly I have seen number for 2.5 to 3.33 megs for MPEG4 so 
> if you say only 3 Megs for rtjpeg and say get around 6 Megs can be 
> xfered then 2 Stream would work fine but more then that would require 
> some form of load balancing multi-nic card.  If I screwed up the number 
> some place let me known as conversion between Bytes and Bits get 
> confusing and could make my number off by a factor of 8.

Check your math, as I suggest above.  And your systems, as suggested
below.  I mean, if you want to get a multi-nic network in place, and
load-balance (don't do it in hardware, you can set it up with something
like firehose, as others have suggested), that's fine... but why waste
the money and complicate things that aren't that complicated?

> After running that test I was think there might be some limits based on 
> HD Speed and I think both machines have DMA off so that could be 
> improved.  However I doubt it will go from 5 to 30.  

Well, perhaps you should try it before you decide:

DMA off:
/dev/hdb1:
 Timing buffered disk reads:  64 MB in  7.55 seconds =  8.48 MB/sec

DMA on:
/dev/hdb1:
 Timing buffered disk reads:  64 MB in  1.98 seconds = 32.32 MB/sec

(This is just from one system, only tweaking the DMA flag in hdparm, and
only doing read tests... I trust it's sufficient... also, note that this
is Bytes, not bits.)

Also, with regards to codec selection; I'm regularly playing DVDs from
the drive in my laptop which is NFS mounted to one video display box
(which plays the disc using mplayer).  Network utilization shows about
6-10Mbits/s.  It's running over an 802.11a link to the AP, which then is
on the 100Mb switch.

I really don't know what else to do / say.  Your issue appears to be in
your particular setup, which isn't usual.  The math and my and others'
experiences show that you *can* run multiple streams around a LAN
without problems or exotic hardware.

--Craig