http://zackreed.me/articles/54-hp-smart-array-p410-controller-tuning

Tuning a HP Smart Array P400 with Linux – why tuning really matters….

 

Today I had to do some tuning on a HP Smart Array P400 controller with 8x SAS 300GB 10K RPM HDD. It was already determined that this controller was *really* *really* bad at RAID5. This was a system that needed some decent performance so it was decided to use RAID10. We set the cache ratio to 50/50, and used 256k stripes. The controller already had write cache enabled with the battery.

This system was running RedHat Enterprise Linux 5.3 with additional support for the XFS file system using the “extras” repository from CentOS. The XFS file system was setup to match with the stripe of the RAID10, and then we mounted it with the following options:

rw,noatime,nobarrier,logbufs=8,logbsize=256k

Sometimes you have all the time in the world to test out every scenario, but this wasn’t one of those times and I had to turn this around fairly quickly. So, I decided just to run a select few test cases with the iozone benchmark (my preferred benchmark tool). These are the specific test cases I ran:

iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2

 

For those not familiar with iozone, the “-i” indicates the test, with the numbers meaning:

0 : sequential write/re-write test
1 : sequential read/re-read test
2 : random read/write test

So, first order of business was to get a “baseline” run to see where we were at:

initial write: 299MB/s
re-write: 335MB/s
read: 123MB/s
re-read: 125MB/s
random read: 108MB/s
random write: 306MB/s

What immediately concerned me here are the read speeds (both sequential and random); those are pretty bad numbers for 8x HDD in RAID10! In particular, this system needed both sequential and random reads to be fast.

Whenever working with Linux on RAID controllers, the first thing I like to try out is to change the I/O scheduler to ‘noop’. Basically, Linux has a modular I/O scheduler architecture and you can choose from 4 different options. The default in RHEL5 is ‘cfq’, which is actually pretty good in many cases. But, if you’re on a RAID controller, sometimes it’s better to let the hardware take care of the I/O intelligence, and that’s where ‘noop’ comes to play. You can change the I/O scheduler via the /sys filesystem:

echo “noop” > /sys/block/cciss!c0d1/queue/scheduler

Re-running the iozone tests above with ‘noop’ yielded some good improvements in both sequential and random read speeds. There was also a small gain in initial write speeds, but other write tests actually lost a little speed. Though, the lost in some write tests seemed like it might be worth the trade-off considering the gains:

initial write: 338MB/s
re-write: 299MB/s
read: 201MB/s (from 123MB/s !!!)
re-read: 200MB/s (from 125MB/s !!!)
random read: 206MB/s (from 108MB/s !!!)
random write: 255MB/s

Looks like we’re headed in the right direction and gaining back some read speeds, even at the slight cost of some write speed tests.

To focus on tuning for read speeds, the next thing to tune is Linux’s read ahead cache. By default, this is set to 256k, but in my experience, this is really too little for every RAID controller I’ve worked with. Normally, you can incrementally increase this value and re-run the benchmark to see the gains (or loss). But I’ve been doing quite a bit of Linux tuning lately and my experience has told me that usually 8MB~32MB is where it’s worth playing. Also, I wasn’t given a lot of time to turn this around, so my first test was to increase the read ahead cache to 8MB:

/sbin/blockdev –setra 8192 /dev/cciss/c0d0

So, with ‘noop’ and 8MB of read ahead cache, here are the results:

initial write: 335MB/s
re-write: 305MB/s
read: 324MB/s (from 123MB/s !!! +263%)
re-read: 325MB/s (from 125MB/s !!! +261%)
random read: 417MB/s (from 108MB/s !!! +386%)
random write: 256MB/s

Pretty impressive gains on the read speeds. I also tried using 16MB of read ahead cache, but it didn’t really yield significant gains, so I decided that 8MB was enough.

Just two simple tuning parameters that can gain quite a bit! Here’s a chart of the results:

 

 

virtualix05 baseline w/ default read/write ratio settings in controller Bios : 29.6.2013

root@virtualix05:/var/lib/vz# iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Sat Jun 29 16:26:56 2013

        Record Size 4096 KB
        File size set to 8388608 KB
        Command line used: iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 6 processes
        Each process writes a 8388608 Kbyte file in 4096 Kbyte records

        Children see throughput for  6 initial writers  =  127186.71 KB/sec
        Parent sees throughput for  6 initial writers   =  109114.56 KB/sec
        Min throughput per process                      =   20453.02 KB/sec
        Max throughput per process                      =   23019.36 KB/sec
        Avg throughput per process                      =   21197.79 KB/sec
        Min xfer                                        = 7458816.00 KB

        Children see throughput for  6 rewriters        =  122600.47 KB/sec
        Parent sees throughput for  6 rewriters         =  115239.01 KB/sec
        Min throughput per process                      =   20321.65 KB/sec
        Max throughput per process                      =   20563.09 KB/sec
        Avg throughput per process                      =   20433.41 KB/sec
        Min xfer                                        = 8290304.00 KB

        Children see throughput for  6 readers          =  147034.55 KB/sec
        Parent sees throughput for  6 readers           =  147011.74 KB/sec
        Min throughput per process                      =   21902.88 KB/sec
        Max throughput per process                      =   26452.78 KB/sec
        Avg throughput per process                      =   24505.76 KB/sec
        Min xfer                                        = 6946816.00 KB

        Children see throughput for 6 re-readers        =  148429.68 KB/sec
        Parent sees throughput for 6 re-readers         =  148399.62 KB/sec
        Min throughput per process                      =   22414.61 KB/sec
        Max throughput per process                      =   25911.73 KB/sec
        Avg throughput per process                      =   24738.28 KB/sec
        Min xfer                                        = 7258112.00 KB

        Children see throughput for 6 random readers    =  111854.35 KB/sec
        Parent sees throughput for 6 random readers     =  111840.00 KB/sec
        Min throughput per process                      =   17777.72 KB/sec
        Max throughput per process                      =   19548.53 KB/sec
        Avg throughput per process                      =   18642.39 KB/sec
        Min xfer                                        = 7630848.00 KB

        Children see throughput for 6 random writers    =  114345.54 KB/sec
        Parent sees throughput for 6 random writers     =  104807.86 KB/sec
        Min throughput per process                      =   18730.79 KB/sec
        Max throughput per process                      =   19794.81 KB/sec
        Avg throughput per process                      =   19057.59 KB/sec
        Min xfer                                        = 7942144.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4096 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  127186.71

"        Rewrite "  122600.47

"           Read "  147034.55

"        Re-read "  148429.68

"    Random read "  111854.35

"   Random write "  114345.54

virtualix05 baseline w/ 50/50 read/write ratio settings in controller Bios : 29.6.2013

root@virtualix05:/var/lib/vz# iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Sat Jun 29 18:58:32 2013

        Record Size 4096 KB
        File size set to 8388608 KB
        Command line used: iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 6 processes
        Each process writes a 8388608 Kbyte file in 4096 Kbyte records

        Children see throughput for  6 initial writers  =  124040.12 KB/sec
        Parent sees throughput for  6 initial writers   =  112673.20 KB/sec
        Min throughput per process                      =   20060.90 KB/sec
        Max throughput per process                      =   21296.84 KB/sec
        Avg throughput per process                      =   20673.35 KB/sec
        Min xfer                                        = 7901184.00 KB

        Children see throughput for  6 rewriters        =  123174.21 KB/sec
        Parent sees throughput for  6 rewriters         =  115789.08 KB/sec
        Min throughput per process                      =   20370.70 KB/sec
        Max throughput per process                      =   20597.13 KB/sec
        Avg throughput per process                      =   20529.03 KB/sec
        Min xfer                                        = 8298496.00 KB

        Children see throughput for  6 readers          =  218050.79 KB/sec
        Parent sees throughput for  6 readers           =  217988.28 KB/sec
        Min throughput per process                      =   35256.20 KB/sec
        Max throughput per process                      =   37452.05 KB/sec
        Avg throughput per process                      =   36341.80 KB/sec
        Min xfer                                        = 7897088.00 KB

        Children see throughput for 6 re-readers        =  218356.21 KB/sec
        Parent sees throughput for 6 re-readers         =  218296.91 KB/sec
        Min throughput per process                      =   35303.81 KB/sec
        Max throughput per process                      =   37495.24 KB/sec
        Avg throughput per process                      =   36392.70 KB/sec
        Min xfer                                        = 7901184.00 KB

        Children see throughput for 6 random readers    =  111753.62 KB/sec
        Parent sees throughput for 6 random readers     =  111734.97 KB/sec
        Min throughput per process                      =   18135.00 KB/sec
        Max throughput per process                      =   19070.06 KB/sec
        Avg throughput per process                      =   18625.60 KB/sec
        Min xfer                                        = 7979008.00 KB

        Children see throughput for 6 random writers    =  112428.72 KB/sec
        Parent sees throughput for 6 random writers     =  105826.13 KB/sec
        Min throughput per process                      =   18506.08 KB/sec
        Max throughput per process                      =   19016.81 KB/sec
        Avg throughput per process                      =   18738.12 KB/sec
        Min xfer                                        = 8163328.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4096 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  124040.12

"        Rewrite "  123174.21

"           Read "  218050.79

"        Re-read "  218356.21

"    Random read "  111753.62

"   Random write "  112428.72

 

virtualix05 with noop io scheduler and w/ 50/50 read/write ratio settings in controller Bios : 29.6.2013

root@virtualix05:/var/lib/vz# iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Sun Jun 30 02:23:21 2013

        Record Size 4096 KB
        File size set to 8388608 KB
        Command line used: iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 6 processes
        Each process writes a 8388608 Kbyte file in 4096 Kbyte records

        Children see throughput for  6 initial writers  =  124310.84 KB/sec
        Parent sees throughput for  6 initial writers   =  115712.64 KB/sec
        Min throughput per process                      =   20586.12 KB/sec
        Max throughput per process                      =   20850.82 KB/sec
        Avg throughput per process                      =   20718.47 KB/sec
        Min xfer                                        = 8282112.00 KB

        Children see throughput for  6 rewriters        =  126704.35 KB/sec
        Parent sees throughput for  6 rewriters         =  118943.32 KB/sec
        Min throughput per process                      =   21020.16 KB/sec
        Max throughput per process                      =   21186.84 KB/sec
        Avg throughput per process                      =   21117.39 KB/sec
        Min xfer                                        = 8323072.00 KB

        Children see throughput for  6 readers          =  220193.77 KB/sec
        Parent sees throughput for  6 readers           =  220145.01 KB/sec
        Min throughput per process                      =   36492.48 KB/sec
        Max throughput per process                      =   36918.96 KB/sec
        Avg throughput per process                      =   36698.96 KB/sec
        Min xfer                                        = 8294400.00 KB

        Children see throughput for 6 re-readers        =  225211.62 KB/sec
        Parent sees throughput for 6 re-readers         =  225170.77 KB/sec
        Min throughput per process                      =   37312.09 KB/sec
        Max throughput per process                      =   37763.04 KB/sec
        Avg throughput per process                      =   37535.27 KB/sec
        Min xfer                                        = 8290304.00 KB

        Children see throughput for 6 random readers    =  111848.74 KB/sec
        Parent sees throughput for 6 random readers     =  111827.50 KB/sec
        Min throughput per process                      =   18459.26 KB/sec
        Max throughput per process                      =   18974.24 KB/sec
        Avg throughput per process                      =   18641.46 KB/sec
        Min xfer                                        = 8163328.00 KB

        Children see throughput for 6 random writers    =  113410.98 KB/sec
        Parent sees throughput for 6 random writers     =  107616.72 KB/sec
        Min throughput per process                      =   18809.50 KB/sec
        Max throughput per process                      =   18990.10 KB/sec
        Avg throughput per process                      =   18901.83 KB/sec
        Min xfer                                        = 8314880.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4096 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  124310.84

"        Rewrite "  126704.35

"           Read "  220193.77

"        Re-read "  225211.62

"    Random read "  111848.74

"   Random write "  113410.98

 

virtualix05 with deadline io scheduler, blockdev –setra 8192 /dev/sda and w/ 50/50 read/write ratio settings in controller Bios : 29.6.2013

 

blockdev --setra 8192 /dev/sda

 

root@virtualix05:/var/lib/vz# iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Sun Jun 30 12:28:08 2013

        Record Size 4096 KB
        File size set to 8388608 KB
        Command line used: iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 6 processes
        Each process writes a 8388608 Kbyte file in 4096 Kbyte records

        Children see throughput for  6 initial writers  =  123856.80 KB/sec
        Parent sees throughput for  6 initial writers   =  115624.39 KB/sec
        Min throughput per process                      =   20602.50 KB/sec
        Max throughput per process                      =   20692.75 KB/sec
        Avg throughput per process                      =   20642.80 KB/sec
        Min xfer                                        = 8351744.00 KB

        Children see throughput for  6 rewriters        =  126771.08 KB/sec
        Parent sees throughput for  6 rewriters         =  118854.80 KB/sec
        Min throughput per process                      =   21088.28 KB/sec
        Max throughput per process                      =   21174.65 KB/sec
        Avg throughput per process                      =   21128.51 KB/sec
        Min xfer                                        = 8355840.00 KB

        Children see throughput for  6 readers          =  221716.43 KB/sec
        Parent sees throughput for  6 readers           =  221640.65 KB/sec
        Min throughput per process                      =   36810.99 KB/sec
        Max throughput per process                      =   37041.74 KB/sec
        Avg throughput per process                      =   36952.74 KB/sec
        Min xfer                                        = 8339456.00 KB

        Children see throughput for 6 re-readers        =  222601.81 KB/sec
        Parent sees throughput for 6 re-readers         =  222498.44 KB/sec
        Min throughput per process                      =   37074.74 KB/sec
        Max throughput per process                      =   37108.47 KB/sec
        Avg throughput per process                      =   37100.30 KB/sec
        Min xfer                                        = 8384512.00 KB

        Children see throughput for 6 random readers    =  111680.99 KB/sec
        Parent sees throughput for 6 random readers     =  111658.95 KB/sec
        Min throughput per process                      =   17662.03 KB/sec
        Max throughput per process                      =   19127.40 KB/sec
        Avg throughput per process                      =   18613.50 KB/sec
        Min xfer                                        = 7749632.00 KB

        Children see throughput for 6 random writers    =  112984.59 KB/sec
        Parent sees throughput for 6 random writers     =  107439.23 KB/sec
        Min throughput per process                      =   18611.95 KB/sec
        Max throughput per process                      =   18954.59 KB/sec
        Avg throughput per process                      =   18830.76 KB/sec
        Min xfer                                        = 8237056.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4096 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  123856.80

"        Rewrite "  126771.08

"           Read "  221716.43

"        Re-read "  222601.81

"    Random read "  111680.99

"   Random write "  112984.59

 

virtualix05 with deadline io scheduler, blockdev –setra 65536 /dev/sda and w/ 50/50 read/write ratio settings in controller Bios : 29.6.2013

root@virtualix05:/var/lib/vz# blockdev --setra 65536 /dev/sda
root@virtualix05:/var/lib/vz# echo 512 > /sys/block/sda/queue/nr_requests
root@virtualix05:/var/lib/vz# echo 2048 > /sys/block/sda/queue/read_ahead_kb

 

root@virtualix05:/var/lib/vz# iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Sun Jun 30 13:09:02 2013

        Record Size 4096 KB
        File size set to 8388608 KB
        Command line used: iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 6 processes
        Each process writes a 8388608 Kbyte file in 4096 Kbyte records

        Children see throughput for  6 initial writers  =  124676.73 KB/sec
        Parent sees throughput for  6 initial writers   =  116643.22 KB/sec
        Min throughput per process                      =   20718.69 KB/sec
        Max throughput per process                      =   20860.29 KB/sec
        Avg throughput per process                      =   20779.45 KB/sec
        Min xfer                                        = 8335360.00 KB

        Children see throughput for  6 rewriters        =  126894.43 KB/sec
        Parent sees throughput for  6 rewriters         =  119005.01 KB/sec
        Min throughput per process                      =   21096.90 KB/sec
        Max throughput per process                      =   21238.43 KB/sec
        Avg throughput per process                      =   21149.07 KB/sec
        Min xfer                                        = 8335360.00 KB

        Children see throughput for  6 readers          =  220207.82 KB/sec
        Parent sees throughput for  6 readers           =  220115.41 KB/sec
        Min throughput per process                      =   36565.54 KB/sec
        Max throughput per process                      =   36892.97 KB/sec
        Avg throughput per process                      =   36701.30 KB/sec
        Min xfer                                        = 8318976.00 KB

        Children see throughput for 6 re-readers        =  222805.00 KB/sec
        Parent sees throughput for 6 re-readers         =  222765.91 KB/sec
        Min throughput per process                      =   37011.91 KB/sec
        Max throughput per process                      =   37292.94 KB/sec
        Avg throughput per process                      =   37134.17 KB/sec
        Min xfer                                        = 8327168.00 KB

        Children see throughput for 6 random readers    =  111137.16 KB/sec
        Parent sees throughput for 6 random readers     =  111121.99 KB/sec
        Min throughput per process                      =   18353.29 KB/sec
        Max throughput per process                      =   18817.59 KB/sec
        Avg throughput per process                      =   18522.86 KB/sec
        Min xfer                                        = 8183808.00 KB

        Children see throughput for 6 random writers    =  112366.90 KB/sec
        Parent sees throughput for 6 random writers     =  106570.36 KB/sec
        Min throughput per process                      =   18659.91 KB/sec
        Max throughput per process                      =   18797.96 KB/sec
        Avg throughput per process                      =   18727.82 KB/sec
        Min xfer                                        = 8327168.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4096 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  124676.73

"        Rewrite "  126894.43

"           Read "  220207.82

"        Re-read "  222805.00

"    Random read "  111137.16

"   Random write "  112366.90

iozone test complete.

 

cache disabled in controller and blockdev –setra 8192 /dev/sda

root@virtualix05:~# echo 512 > /sys/block/sda/queue/nr_requests
root@virtualix05:~# echo 2048 > /sys/block/sda/queue/read_ahead_kb
root@virtualix05:~# blockdev --setra 8192 /dev/sda

 

root@virtualix05:/var/lib/vz# iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Sun Jun 30 14:53:44 2013

        Record Size 4096 KB
        File size set to 8388608 KB
        Command line used: iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 6 processes
        Each process writes a 8388608 Kbyte file in 4096 Kbyte records

        Children see throughput for  6 initial writers  =  142557.77 KB/sec
        Parent sees throughput for  6 initial writers   =  132416.51 KB/sec
        Min throughput per process                      =   23524.44 KB/sec
        Max throughput per process                      =   24017.62 KB/sec
        Avg throughput per process                      =   23759.63 KB/sec
        Min xfer                                        = 8216576.00 KB

        Children see throughput for  6 rewriters        =  144666.08 KB/sec
        Parent sees throughput for  6 rewriters         =  136064.40 KB/sec
        Min throughput per process                      =   24000.16 KB/sec
        Max throughput per process                      =   24206.67 KB/sec
        Avg throughput per process                      =   24111.01 KB/sec
        Min xfer                                        = 8318976.00 KB

        Children see throughput for  6 readers          =  222012.70 KB/sec
        Parent sees throughput for  6 readers           =  221972.05 KB/sec
        Min throughput per process                      =   36623.76 KB/sec
        Max throughput per process                      =   37361.21 KB/sec
        Avg throughput per process                      =   37002.12 KB/sec
        Min xfer                                        = 8224768.00 KB

        Children see throughput for 6 re-readers        =  193011.65 KB/sec
        Parent sees throughput for 6 re-readers         =  192972.40 KB/sec
        Min throughput per process                      =   13777.51 KB/sec
        Max throughput per process                      =   36187.76 KB/sec
        Avg throughput per process                      =   32168.61 KB/sec
        Min xfer                                        = 3194880.00 KB

        Children see throughput for 6 random readers    =  111677.88 KB/sec
        Parent sees throughput for 6 random readers     =  111659.83 KB/sec
        Min throughput per process                      =   16241.89 KB/sec
        Max throughput per process                      =   19672.25 KB/sec
        Avg throughput per process                      =   18612.98 KB/sec
        Min xfer                                        = 6926336.00 KB

        Children see throughput for 6 random writers    =  126197.96 KB/sec
        Parent sees throughput for 6 random writers     =  117797.86 KB/sec
        Min throughput per process                      =   19615.35 KB/sec
        Max throughput per process                      =   21756.47 KB/sec
        Avg throughput per process                      =   21032.99 KB/sec
        Min xfer                                        = 7565312.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4096 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  142557.77

"        Rewrite "  144666.08

"           Read "  222012.70

"        Re-read "  193011.65

"    Random read "  111677.88

"   Random write "  126197.96

 

cache disabled in controller and blockdev –setra 8192/dev/sda

 virtualix06

root@virtualix06:/var/lib/vz# iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Sun Jun 30 15:12:55 2013

        Record Size 4096 KB
        File size set to 8388608 KB
        Command line used: iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 6 processes
        Each process writes a 8388608 Kbyte file in 4096 Kbyte records

        Children see throughput for  6 initial writers  =  144759.69 KB/sec
        Parent sees throughput for  6 initial writers   =  132716.50 KB/sec
        Min throughput per process                      =   23850.50 KB/sec
        Max throughput per process                      =   24708.73 KB/sec
        Avg throughput per process                      =   24126.61 KB/sec
        Min xfer                                        = 8093696.00 KB

        Children see throughput for  6 rewriters        =  143237.56 KB/sec
        Parent sees throughput for  6 rewriters         =  134912.85 KB/sec
        Min throughput per process                      =   23820.61 KB/sec
        Max throughput per process                      =   23955.29 KB/sec
        Avg throughput per process                      =   23872.93 KB/sec
        Min xfer                                        = 8343552.00 KB

        Children see throughput for  6 readers          =  220328.79 KB/sec
        Parent sees throughput for  6 readers           =  220251.93 KB/sec
        Min throughput per process                      =   36300.03 KB/sec
        Max throughput per process                      =   37686.62 KB/sec
        Avg throughput per process                      =   36721.46 KB/sec
        Min xfer                                        = 8081408.00 KB

        Children see throughput for 6 re-readers        =  222152.23 KB/sec
        Parent sees throughput for 6 re-readers         =  222096.77 KB/sec
        Min throughput per process                      =   36601.68 KB/sec
        Max throughput per process                      =   37950.48 KB/sec
        Avg throughput per process                      =   37025.37 KB/sec
        Min xfer                                        = 8093696.00 KB

        Children see throughput for 6 random readers    =  109889.88 KB/sec
        Parent sees throughput for 6 random readers     =  109877.32 KB/sec
        Min throughput per process                      =   18099.63 KB/sec
        Max throughput per process                      =   18490.03 KB/sec
        Avg throughput per process                      =   18314.98 KB/sec
        Min xfer                                        = 8212480.00 KB

        Children see throughput for 6 random writers    =  128001.76 KB/sec
        Parent sees throughput for 6 random writers     =  118392.95 KB/sec
        Min throughput per process                      =   21034.58 KB/sec
        Max throughput per process                      =   22218.07 KB/sec
        Avg throughput per process                      =   21333.63 KB/sec
        Min xfer                                        = 7942144.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4096 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  144759.69

"        Rewrite "  143237.56

"           Read "  220328.79

"        Re-read "  222152.23

"    Random read "  109889.88

"   Random write "  128001.76

 

cache disabled in controller and blockdev –setra 16384 /dev/sda

 virtualix05

root@virtualix05:/var/lib/vz# blockdev --setra 16384 /dev/sda
root@virtualix05:/var/lib/vz# iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Sun Jun 30 15:31:15 2013

        Record Size 4096 KB
        File size set to 8388608 KB
        Command line used: iozone -b results.xls -r 4m -s 8g -t 6 -i 0 -i 1 -i 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 6 processes
        Each process writes a 8388608 Kbyte file in 4096 Kbyte records

        Children see throughput for  6 initial writers  =  144278.33 KB/sec
        Parent sees throughput for  6 initial writers   =  134745.71 KB/sec
        Min throughput per process                      =   23946.20 KB/sec
        Max throughput per process                      =   24146.44 KB/sec
        Avg throughput per process                      =   24046.39 KB/sec
        Min xfer                                        = 8318976.00 KB

        Children see throughput for  6 rewriters        =  145186.16 KB/sec
        Parent sees throughput for  6 rewriters         =  136351.37 KB/sec
        Min throughput per process                      =   24044.97 KB/sec
        Max throughput per process                      =   24287.41 KB/sec
        Avg throughput per process                      =   24197.69 KB/sec
        Min xfer                                        = 8318976.00 KB

        Children see throughput for  6 readers          =  222705.52 KB/sec
        Parent sees throughput for  6 readers           =  222654.89 KB/sec
        Min throughput per process                      =   36967.43 KB/sec
        Max throughput per process                      =   37288.05 KB/sec
        Avg throughput per process                      =   37117.59 KB/sec
        Min xfer                                        = 8318976.00 KB

        Children see throughput for 6 re-readers        =  225033.54 KB/sec
        Parent sees throughput for 6 re-readers         =  224972.46 KB/sec
        Min throughput per process                      =   37370.41 KB/sec
        Max throughput per process                      =   37689.11 KB/sec
        Avg throughput per process                      =   37505.59 KB/sec
        Min xfer                                        = 8318976.00 KB

        Children see throughput for 6 random readers    =  111713.37 KB/sec
        Parent sees throughput for 6 random readers     =  111696.25 KB/sec
        Min throughput per process                      =   17991.62 KB/sec
        Max throughput per process                      =   19105.77 KB/sec
        Avg throughput per process                      =   18618.89 KB/sec
        Min xfer                                        = 7901184.00 KB

        Children see throughput for 6 random writers    =  127603.61 KB/sec
        Parent sees throughput for 6 random writers     =  122378.24 KB/sec
        Min throughput per process                      =   21126.16 KB/sec
        Max throughput per process                      =   21365.35 KB/sec
        Avg throughput per process                      =   21267.27 KB/sec
        Min xfer                                        = 8306688.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4096 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  144278.33

"        Rewrite "  145186.16

"           Read "  222705.52

"        Re-read "  225033.54

"    Random read "  111713.37

"   Random write "  127603.61

iozone test complete.

——————————————————————————

01.07.2013

virtualix05 w/o tuning

"  Initial write "  146166.36

"        Rewrite "  142903.57

"           Read "  216407.17

"        Re-read "  222941.70

"    Random read "  111485.41

"   Random write "  128887.30
--------------------------------
"  Initial write "  144666.84

"        Rewrite "  144737.20

"           Read "  222139.73

"        Re-read "  224575.34

"    Random read "  111395.25

"   Random write "  128379.38

virtualix06 w/o tuning

"  Initial write "  148530.15

"        Rewrite "  142020.55

"           Read "  160840.40

"        Re-read "  177377.35

"    Random read "  111926.32

"   Random write "  126882.84
--------------------------------
w/ tuning

"  Initial write "  144951.52

"        Rewrite "  143645.40

"           Read "  218866.55

"        Re-read "  219992.87

"    Random read "  110586.40

"   Random write "  127957.80
--------------------------------
+ /etc# vi sysctl.conf

# max open files by zumbi 14.02.2012
fs.file-max = 65535

# I/O tuning by zumbi
vm.dirty_background_ratio = 20
vm.dirty_ratio = 60
vm.overcommit_ratio = 2

"  Initial write "  162294.77

"        Rewrite "  160913.01

"           Read "  175933.54

"        Re-read "  201282.81

"    Random read "  111987.00

"   Random write "  135425.48
-----------------------------------------------

+ /etc# vi sysctl.conf

# max open files by zumbi 14.02.2012
fs.file-max = 65535

# I/O tuning by zumbi
vm.dirty_background_ratio = 20
vm.dirty_ratio = 60
vm.overcommit_ratio = 2

 blockdev --setra 16384 /dev/sda