Linux io scheduler comparison -- throughput

In the article change Linux io scheduler, I described the basic idea of each io scheduler, also how to change them to meet your application requirement.
CFQ is the default i/o scheduler, and it's the best bet In most of applications too.  Still, Linux allows you to tune it for your own application. In this article, I'll show you that how I tested i/o scheduler for file server. Most of file accesses are full file copy. If you can simulate your own application i/o, then there you go.

Env:

An old server, IBM 3650, 4 CPUs, 12GB memory, dual FC channels 4Gb each. SL5, kernel 2.6.18-194.11.4.el5, two SAN LUNs mounted on the node.
xfsprogs-2.9.4-1.sl5.x86_64, iozone version 3.327.
readahead buffer set to 16386.

Test1: 1 thread throughput writing and reading test.

test command(no other application running on the node)
/usr/bin/iozone -s24g -r512 -i 0 -i 1 -t 1 -+u
Output is in Kbytes/sec
   Write       re-write  read  re-read
 CFQ  462901.50  507647.25  388681.56  386001.81
 deadline  466868.50  512209.06  391834.41  387202.53
 anticipatory  460229.41  463203.00  399485.22  397976.94
 NOOP  464676.28  502374.19  389041.66  386453.16
As you can see, there is hardly different amoung them. The reason is simple, the test was just single thread sequencial i/o.
CPU cost
Out put is CPU%
   Write       re-write  read  re-read
 CFQ  65.07 71.85 27.72 27.80
 deadline  64.09 71.13 25.68 24.87
 anticipatory  67.37 73.01 28.18 28.30
 NOOP  57.47  60.36  26.13  25.26
As you can see, NOOP takes less CPU cost, no pause, no optimization, make sense.

Test2: 10 threads test

Same condition with the first one, except number of threads.
/usr/bin/iozone -s24g -r512 -i 0 -i 1 -t 10 -+u
Output is in Kbytes/sec
   Write       re-write  read  re-read
 CFQ 171273.15 163021.82 274956.17 295029.39
 deadline 203954.23 196002.21 320392.37 325140.25
 anticipatory 189632.70 183137.74 323864.17 333810.14
 NOOP 72295.27 60461.09 304580.19 300524.52

From the data above, you can see there is not much different on reading, which is the main area of io scheduler tunning for. So, for sequencial i/o, i/o scheduler only does a little on reading. While they made different on writing, where deadline does the best.
CPU cost
Out put is CPU%
   Write       re-write  read  re-read
 CFQ 49.90 65.08 20.50 21.72
 deadline 61.49 79.87 23.85 23.76
 anticipatory 57.59 74.41 23.75 24.81
 NOOP 16.78 20.72 22.61 22.27
Same as 1 thread test, NOOP takes less CPU cost, while deadline io scheduler takes the most CPU on most of write/read tests.

Overall, let's say, CFQ is a compromised i/o scheduler on sequencial i/o, between I/O performance and CPU cost. Deadline doest the best if CPU is not a cost consideration on your application.