OpenFiler Bonnie++ V1.03 Benchmark results

The following are the results of the Bonnie++ Disk Performance benchmark tests that I ran on my new OpenFiler V2.2 server before placing it into production. I was testing to prove that the disk drive/drive controller configuration I had come up with was cost effective and performed well.

I am replacing an old OpenFiler V1.1 server that has a LSI Logic MegaRAID i4 controller and five 80GB hard drives. This is also running on an older motherboard and has less than stellar performance.

The new hardware consists of:

• A SuperMicro X5DPE-G2 motherboard (scavenged for free)
• A 2.4GHz Xeon processor with hyperthreading (came with the motherboard)
• 1GB RAM (scavenged for free)
• A 4RU Server Class case ($185 at NewEgg)
• Three Promise Ultra100 TX2 IDE controllers (One free from a coworker - Thanks Nate!, two purchased on eBay for $8 and $16.) These are 66Mhz PCI ATA-100 Dual Channel IDE Controller cards. Each card can support four hard drives.
• A bunch of new IDE cables ($35 at NewEgg)
• A new CPU cooler ($20 at NewEgg)
• Five 250 GB IDE ATA133 hard drives - for the file storage (purchased from a Nate for $200 quite a while ago)
• Two 80 GB IDE hard drives - for the OS (scavenged for free)
• A SCSI controller and DLT40 tape drive (scavenged for free)

So you can see that it's pretty cheap (around $500). And it's a terabyte! (Well, almost. The drives show up as 238GB, so the total size is around 952GB, but it's all in how you count it and I'm going to call it a terabyte!) My current filer is only 320GB and it's filling up with MP3s and digital photos.

Notes:

• The hard drives are ATA133, but the IDE cards are ATA100. This is due to the fact that the Ultra100 cards were available cheaply on eBay (and the fact that Nate had one) and the Ultra133 cards were $35 each (now $30) at NewEgg. I had a very limited budget for this project.
• OpenFiler Version 2.2 is now based on rPath Linux. Version 1.1 was based on CentOS. rPath is specifically designed for building software applications and uses a repository system called 'conary' which seems to be a lot like 'yum'. rPath appears to be a subset of RedHat. (I saw some RedHat labels while I watched it boot.) So it's pretty familiar.

Test Conditions

The motherboard has two on-board IDE controllers and two dual-channel PCI bus controllers for a total of four separate PCI buses (and six PCI slots).

The Promise Ultra100 TX2 cards were flashed with the latest firmware update (2.20.0.15)

The motherboard PnP BIOS configuration was cleared each time the Ultra100 TX2 cards were added or removed.

The latest version of OpenFiler (2.2) was installed and "conary updateall" was performed.

bonnie++ V1.03 (the latest stable version) was installed using the following procedure found on the OpenFiler Forums:

• contrib.rpath.org@rpl:devel was added to the end of the installLabelPath line in /etc/conaryrc
• Run conary update bonnie++

All tests were performed with an md software RAID 5 array made of full disk partitions on five 250GB drives using the default 64K chunk size for a total RAID 5 size of ~1TB.

An LVM volume group was created out of the whole array.

A 200000MB XFS formatted volume was created to run the tests in.

All the RAID and volume creation was done using the OpenFiler web interface. Some of the volume removal had to be performed using the command line utilities as there appear to be some OpenFiler bugs related to this procedure.

The bonnie++ command line was:

bonnie++ -u root \
   -d /mnt/vol-huge/test-vol1/test \
   -s 2016M \
   -n 10:102400:1024:1024 \
   -m yakko \
   -q > /var/tmp/bonnie_test.csv 2> /var/tmp/bonnie_test.out

bonnie++ options explained:

• -u root - run as root (no other users on the system)
• -d /mnt/vol-huge/test-vol1/test - where to create the files
• -s 2016M - File size created for the Sequential tests - twice the system memory size so we don't test the cache
• -n 10:102400:1024:1024 - Options for the file creation tests - File Size (in MB):Max-Files:Min-Files:Number of directories
• -m yakko - Machine Name (for the output)
• -q - Quiet mode so that we can redirect the output to files - stdout is the csv format output and stderr is the human-readable format output

Each bonnie++ test was run 10 times with a 60 second sleep between each run to let everything settle and then the results were averaged.

Testing Configurations

Four different tests were performed. Additionally, after the new filer is moved into production a test will be run on the old filer for sake of comparison.

Test 1 Configuration

• 1 Ultra100 TX2 card on PCI bus 1b (133 MHz).
• The Ultra100 TX2 card had two drives per IDE channel.
• The secondary on-board IDE controller had the fifth RAID 5 drive and the second mirrored OS drive.
• The primary on-board IDE controller had the first mirrored OS drive.

Test 2 Configuration

• 2 Ultra100 TX2 cards on PCI Bus 1b (133 MHz) and PCI bus 2a (100 MHz).
• Each Ultra100 TX2 card had one hard drive per channel and the fifth RAID 5 hard drive was on the secondary on-board IDE controller shared with the second mirrored OS drive.
• The primary on-board IDE controller had the first mirrored OS drive.

Test 3 Configuration

• 3 Ultra100 TX2 cards on PCI bus 1a (100 Mhz), PCI Bus 1b (133 MHz) and PCI bus 2a (100 MHz).
• The Ultra100 TX2 cards on PCI buses 1b and 2a had one RAID 5 hard drive on each IDE channel.
• The Ultra100 TX2 card on PCI bus 1a had one RAID 5 hard drive on the primary IDE channel.
• The two on-board IDE controllers had the two mirrored OS drives.

Test 4 Configuration

• 3 Ultra100 TX2 cards on PCI bus 1a (100 Mhz), PCI Bus 1b (133 MHz) and PCI bus 2a (100 MHz).
• Each Ultra100 TX2 card had one RAID 5 hard drive on the primary IDE channel (three hard drives).
• The other two RAID 5 hard drives are on the two on-board IDE controllers - shared with the two mirrored OS drives.

Test Data Summary

The following table is a summary of the results. All the test results can be seen on the Full Results page. And are available in an Excel file and a CSV file in case you want to play with the numbers yourself.

Test Results Summary

I was surprised to see that Configuration 3 had slightly better performance on most of the read tests (which I feel is more important than the write tests.) I expected that Configuration 4 would be better, but it's not. So I will be building my filer following Configuration 3.

Other things I learned

Linux md software RAID is amazingly resilient.

I swapped the five RAID 5 drives around between all the controllers - changing the drive device names - and the array was still clean. Surprised the hell out of me.

The Linux kernel is smarter than the motherboard BIOS.

When I stuck the third Ultra100 TX2 card in the machine I was concerned by the fact that only two appeared to be recognised and that only 8 available drives were listed. Not to mention the fact that when I had five drives plugged into the three cards it only listed three drives!

But a bit of Googling led me to believe that it would work correctly under recent Linux kernels and when I booted into OpenFiler, sure enough, it saw all three controllers and all five drives.

Related Information

• All the OpenFiler tagged entries on my blog.
• The OpenFiler Project
• The OpenFiler Forums
• The bonnie++ website

This page and all content Copyright 2007 Timothy W. Foreman
Page last updated 06 January 2007