[mythtv-users] Some master/slave backend advise required

Johan Van der Kolk johan.vanderkolk at gmail.com
Mon Jun 2 22:56:44 UTC 2014


Thanks for all your comments.

I appreciate the advice on ZFS, however I’m not running ARC, since I was aware of the memory requirements. I do not use ARC, compression or deduplication, not enough memory..
And from reading up (no practical experience) ARC has benefits in improving reading data, not so much writing data (which may be my bottleneck). I’m just running it in a RAIDZ2-0 configuration, hoping for some fault tolerance. Performance was never the goal. That now has changed, and I need to review the why and how, guess that is back to the drawing board. 
But memory can be upgraded if that helps the write performance..

Now it seems this thread has focused on the pro’s and cons of ZFS, and I’m a total novice on that subject, hence my very basic installation. You have to start learning somehow, somewhere..
I have learned a lot already here.

Just to get back to the basic question…
Processes are killed, I said…Maybe causing a misunderstanding. Some processes expected to do real-time data stream processing cannot cope with the lack of CPU idle time, mess up and disappear. So maybe not to well written for these circumstances. But I need them, so I want to split some load over two machines. A master backend (the current backend) and a slave backend (a new backend with the receivers).

My aim was to keep all the data on the master backend, which also has the 7TB ZFS Zraid2-0, (raw 12TB in 6 disks) and use that processor for the SQL database, master backend comflagging and some other services. Using the slave backend only to process and push the data up to master backend with the ZFS storage pool. That does not seem to be a recommended option.

Infiniband looks attractive from performance perspective, but I have no idea about cable pricing. I need to cover a distance of about 50 meters with lots of bends. 

From the comments I received so far, it looks like if i have to throw $$$ at the problem, it is best spent on a powerful slave backend with plenty processing power and local storage. I just miss the fault tolerance, unless I just use plain raid mirror.
Just afraid to drop that system in the garden shed…although it may be the safest location for a server, who would steal a server if he can take the lawnmower :-)

At least I’m encouraged to look again at my ZFS storage, and improve that area as well. Time to use google and the ZoL mailing list. (If it only for the learning experience, it is still a good exercise to get more familiar)

thanks again for all your constructive comments!

Johan






On 02 Jun 2014, at 23:08, Raymond Wagner <raymond at wagnerrp.com> wrote:

> 
> On Jun 2, 2014, at 16:55, James Bailey <paradoxbound at gmail.com> wrote:
> 
>> Hello 4GB is the minimum spec for ZFS then you should add more for MythTV, I ran 16GB on my backend with ZFS on Linux and had no problems. You want to limit your ARC to no more than 50% of RAM. If you are mainly writing SSDs are not going to help unless you use a couple of smaller ones for the ZIL because your ZIL is killing your IOPs. you need to mirror these because if you lose a separate ZIL Drive your ZFS array is fscked. As always you should pair ZFS with ECC RAM. don't bother with depude unless your workload really suits it as it will kill disk performance. 
>> 
>> If you haven't been there already talk to the people on the ZoL mailing list they are very knowledgeable and quite friendly if you ask your question right. You will get lots of tuning advise from them for your work load.
>> 
>> If you end up splitting your storage from your backend, consider a couple of cheap Infiband cards from Ebay crazy bandwidth 10Gbit ethernet emulation or 20Gbit native.
>> 
>> Some of the new Intel Avonton storage boards are a reasonable price and you can stick up to 32GB of ECC ram on them.
>> 
>> Jim
>> 
>> 
>> On 2 June 2014 19:59, Johan Van der Kolk <johan.vanderkolk at gmail.com> wrote:
>> Hi,
>> 
>> I’m running myth 0.27 (v0.27.1-7-g41d04b6) on Ubuntu server 14.04 with an i5(3GHz) and 4GB memory. I’m using 4 DVB-S2 tuners as source.
>> Due to relocation of the dish I have to install a slave backend. I can’t (don’t want to) move my server to the garden shed.
>> 
>> I have found that recording about 8-10 HD channels simultaneously will start killing processes on the backend (not the myth processes though), but the CPU load of the DVB-s2 stuff by itself requires 100% cpu. Mythcommflagging did not help either, although I could limit the simultaneous jobs to get more breathing space.
>> So my thought was to solve both problems at the same time with a new slave backend, where the dvb-s2 stuff is running, by itself. Storage and commflagging still to be done on the master backend (which has 8TB of ZFS storage)
>> 
>> I can see two issues now:
>> Bandwidth: When recording 10 channels (what I want to achieve), I estimate worst case (based on what i recorded so far) 3GB per channel per hour, or 833Mb/s one way traffic, without other overhead. And not watching anything...
>> How to solve this, and does myth traffic between master and slave benefit from Jumbo Frames. I could use two network cards in each machine, and create a 2 x1 Gb trunk between the switches.
>> Second part of the problem might be that the ZFS file storage is not fast enough. (upgrade to SSD maybe)
>> 
>> Configuration:
>> Is it possible to configure mythtv in such a way that it does what I want, or are there better ways to do this?
>> 
>> Any help appreciated!
>> 
>> Johan
>> _______________________________________________
>> mythtv-users mailing list
>> mythtv-users at mythtv.org
>> http://www.mythtv.org/mailman/listinfo/mythtv-users
>> http://wiki.mythtv.org/Mailing_List_etiquette
>> MythTV Forums: https://forum.mythtv.org
>> 
>> _______________________________________________
>> mythtv-users mailing list
>> mythtv-users at mythtv.org
>> http://www.mythtv.org/mailman/listinfo/mythtv-users
>> http://wiki.mythtv.org/Mailing_List_etiquette
>> MythTV Forums: https://forum.mythtv.org
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://www.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mythtv.org/pipermail/mythtv-users/attachments/20140603/60c46b1f/attachment.html>


More information about the mythtv-users mailing list