[mythtv-users] RAID 6 benchmarks

Yeechang Lee ylee at pobox.com
Wed Jun 16 06:44:07 UTC 2010


I recently rebuilt the mdadm software RAID 6 array that forms the bulk
of my MythTV recording space, and would like to report on my
experiences benchmarking it.

COMPONENTS

16 Western Digital RE2 500GB SATA disks
Highpoint 2240 PCI-X 16-port controller, in JBOD mode
1.6GHz quad-core Xeon processor
Supermicro X7DVL-E server-class motherboard with 2GB RAM
mdadm Linux software RAID 6
CentOS 5 with the centos.plus kernels for JFS support

BACKGROUND

When I brought the server and the array on line in January 2007 each
RAID 6 array creation took about seven hours; the process is
diskbound, not CPU bound, and adjusting the settings in
/proc/sys/dev/raid don't help much. I didn't have the patience to
experiment as much as I'd like to have regarding the optimal chunk
size and ended up picking 128K, double the default.

mdadm still takes about seven hours to create an array but now has the
--assume-clean option, permitting very fast array creation. The option
is not safe for all RAID levels but RAID 6 is OK. I intended to create
and destroy arrays using chunk sizes from 32K to 1024K to find the
right balance of read and write speeds, then recreate the array with
the optimal chunk size without --assume-clean for permanent use.

METHODOLOGY

My testing methodology was not highly sophsticated. For each chunk
size I did the following:

$ sudo mdadm --create /dev/md0 \
  -e 1.2 \ # The old 0.9 superblock is no longer recommended for new arrays
  --level=6 \ # RAID 6
  --raid-devices=16 \ # 16 disks in array
  /dev/sd[b-q] \ # The disks' device nodes
  --chunk=___ # 32 to 1024

$ sudo mkfs.jfs /dev/md0 # Create array
$ mount /dev/md0 /mnt/arraymountpoint
$ cd /mnt/arraymountpoint
$ mkdir tmp; chmod 1777 tmp; cd tmp
$ su
# echo 3 > /proc/sys/vm/drop_caches # Clears RAM caches
# exit

The first test writes a 10GB file as fast as possible. 10GB is a good
proxy size for my MythTV recordings, which are up to 8GB per hour per
tuner in MPEG-2 form from over-the-air or FireWire. During this and
all other tests I had top running in another window. I was not
concerned about CPU usage, per se; rather, I watched "wa" or iowait,
the percent of time the CPU is idle but waits for the disk. All else
being equal, lower iowait is better.

$ rm -rf 10gb; time (dd if=/dev/zero \
  of=10gb bs=1024k count=10000; sync)

I then again cleared the cache with

# echo 3 > /proc/sys/vm/drop_caches

then ran the read test:

$ dd if=10gb of=/dev/null bs=1024k count=10000

then

$ cd ~
$ sudo umount /mnt/arraymountpoint
$ sudo mdadm --stop /dev/md0

then repeated the above with the next chunk size.

RESULTS

32K	MB/s		iowait
read	200-220M	14%
write	250M		7-14%

64K
read	250M		10-20%
write	295M		0-10%

128K
read	368M		0-10%
write	240-245M	5-20%

256K
read	400M		0-3%
write	220M		10-25%

512K
read	360M		0-5%
write	107M		10-25%

1024K
read	290M		0-5%
write	31M		10-25%

My findings are broadly similar to those of Jon Nelson in July 2008
(<URL:http://blog.jamponi.net/2008/07/raid56-and-10-benchmarks-on-26255_10.html>).
We both found that with RAID 6 larger chunk sizes resulted in lower
write speeds and higher read speeds (in my case, only up to a certain
point).

The biggest single impact on performance turned out to be something
not discussed above. mdadm permits the addition to an existing array
of a write-intent bitmap for the array to make the process of
replacing a disk almost instantaneous. As the man page warns, however,
an internal bitmap "can have a drastic impact on performance." In my
case, I found that 250MB/s write speeds collapsed to 13MB/s (!) with
an internal bitmap. The man page warns against external bitmaps for
non-ext2/3 filesystems and I wouldn't want to depend on an external
disk for such a purpose anyway. So, if you have an internal bitmap in
your RAID array and want more speed, consider taking it off (adding
and subtraction are both almost instantaneous) and seeing what
happens.

NETWORK

I also tested remote access with CIFS to each array chunk size, but
abandoned the effort after finding read and write speeds consistently
in the 50-60MB/s range (unless the local speed was lower, as with the
1024K chunk size). Such figures aren't quite Gigabit's 125MB
theoretical maximum, but I know that in practice 40-50% is pretty
good. I also tested NFS but found it to be an order of magnitude
slower than CIFS, consistent with earlier tests years ago
(<URL:http://www.gossamer-threads.com/lists/mythtv/users/169825#169825>).

CONCLUSION

Any chunk size 256K or below is accceptable. I chose the 128KB chunk
size again; my choice in January 2007 was wiser than I knew. 128K
provides the second-highest read speed and a very close third on the
write speed.

-- 
MythTV FAQ Q: "Cheap frontend/backend?" A: Revo, $200-300 @ Newegg
Q: "Record HD cable/satellite?" A: Hauppauge HD-PVR, $200 @ Newegg
Q: "Can't change Live TV channels w/multirec!" A: Hit NEXTCARD key
More answers @ <URL:http://www.gossamer-threads.com/lists/mythtv/>


More information about the mythtv-users mailing list