[mythtv-users] Storing recordings on network share

[Simon Hobson]

Travis Tabbal wrote:

>>  >The threadedfilewriter has a one second sync loop for each file.  This
>>  >sync loop runs independently for each file, and only syncs that single
>>  >file.  If you are writing a bunch of other data to disk independently,
>>  >those writes should be unaffected by MythTV's loop.

>>But what about the other way around ? If other stuff blocks that 1
>>second loop - does that mean the writing process blocks and loses
>>incoming data that could otherwise be buffered ? Several gigs of RAM
>>will buffer quite a lot of video when it's a 2mbps stream !

>But who uses 2mbps these days?

Lots of people ! Over here in the UK, DVB-T 
(which we generally call Freeview) varies but 
isn't a particularly high bit rate. Some of the 
commercial channels can be less than 1GByte/hr or 
about 2Mbits/s, while the BBC channels can be 
more than 3 times that. That's all for SD though 
- I'll need a DVB-T2 tuner to get HD.

>My recordings are ATSC from a HDHR, so average 
>15mbps or so.  And Myth won't just keep 
>buffering till it runs out of RAM. It has a 
>fixed size ringbuffer and it can and does 
>overflow.

That is useful to know.
How is the size of that buffer determined ? Can 
it be controlled by the admin ? Other than taking 
memory away from other areas, is there any 
penalty for using a larger buffer ?

I'm thinking that allocating a significantly 
larger buffer would help enormously with these 
problems - as someone else mentioned, RAM is 
cheap these days. I've a brand new HP Microserver 
dedicated to Myth that effectively cost only £140 
after cashback, and for another £70 I can stuff 
it with 8G or ECC RAM (it's got 2G at the moment) 
- so memory isn't a constraint and I'd not have 
any concern about allocating a buffer into tens 
of seconds (or even minutes) long.

>I had issues when recording to a RAID6 (ZFS 
>RAIDZ2) over an NFS/SMB link to a VM on the same 
>server. Granted, there are a lot of layers 
>there, but normal file copies are very fast. I 
>solved it by putting recordings on a small LVM 
>exported from the dom0 on a pair of RAID1 drives 
>then using scripts to copy them to the network 
>share.

Hmm, messy ! I'd rather sort out better buffering.

>I saw some posts here talking about the speed of 
>the SATA interface. Please keep in mind that for 
>our uses, that is completely irrelevant. It's 
>the seek time that's killing you here, the 
>random I/O, NOT the interface speed. Look at 
>disk benchmarks, even for SSDs, and you will see 
>an interesting pattern. Streaming I/O will be 
>high, while small random I/O will be REALLY slow 
>in comparison, a small fraction. Once you get a 
>few streams of random I/O going, disks can't 
>keep up. Myth calling sync on the disks isn't 
>helping as now the OS cache and such can't help 
>hide the pattern, though there are reasons they 
>do this. That is why people recommend putting 
>recordings on their own drives. It limits all 
>the seeks from other system activity from 
>interfering with recordings. Using a pair of 
>smaller drives in a storage group is another win 
>as myth will balance recordings across the disks 
>for you, like a RAID0, without the data 
>integrity issues.

Spot on.
Not only is a SATA connection fast, it is also 
point to point and dedicated to the single disk 
it connects - unless you use a port expander in 
which case it's shared between all devices on the 
expander. Most usually though, there is a 
controller channel per disk until you get into 
quite sizable arrays.

However, it isn't all that long since bus 
throughput was an issue - especially on (for 
example) a SCSI bus with multiple high 
performance drives. Yes, in the past I'm been 
down the road of carefully arranging my server 
drives equally across all the available busses - 
but I think then the PCI-something slot was 
probably the streaming throughput bottleneck.
-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.