iozone is a file system benchmark tool, which is a very good measurement tool for file system based applications, servers. etc.. Even database applications. Togegher with network performance tools, CPU benchmark tools, they basically give you a baseline of how well you server can do, or how well your server tunned.
From the link below, you can download it's full documentation, as well as source and examples
http://www.iozone.org/
Here I'm going to just show most popular functions to get you easy start.
Before get started, one thing has to mentioned is that it tests filesystem, each filesystem has buffer cache, so if you want to measure sustenable rate, then probably you need to specify a enough big file to eliminate cache effect.

Basic output.

Each time you run iozone, there is a basic info session

 #/usr/bin/iozone -s24m -r64 -i 0 -i 1 -t 1 -b 24m.xls
    Iozone: Performance Test of File I/O
            Version $Revision: 3.327 $
        Compiled for 64 bit mode.
        Build: linux

    Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                 Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                 Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                 Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                 Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
                 Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

Below the info session is the test description of your own.

    File size set to 24576 KB
    Record Size 64 KB
    Command line used: /usr/bin/iozone -s24m -r64 -i 0 -i 1 -t 1 -b 24m.xls
    Output is in Kbytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 1024 Kbytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
    Throughput test with 1 process
    Each process writes a 24576 Kbyte file in 64 Kbyte records

The session below shows detailed results.
...

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 64 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  422137.78

"        Rewrite " 1028377.50

"           Read " 2072368.75

"        Re-read " 2137207.00

Note: In the example, I tested one of my server's /root filesystem, which is sitting on a mirrored SATA disks. iozone tells that it got 2GB/sec readiing on it(recall I mentioned caching effect for small files).

Automatic test mode

-a option used  to  select  full automatic mode. Produces output that covers all tested file operations for record sizes of 4k to 16M for file sizes of 64k to 512M.
Usually you combine it with -g or -s option to specify larger files.

    Auto Mode
    Command line used: iozone -s 2g -a
                                                 ndom  random    bkwd   record   stride     
KB  reclen write rewrite    read   reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
2097152   4  860348 1201573  2081567  2173982 1826295 1132584 1960767  3962311  1936269   843183  1176150 2125247  2182763
2097152   8  933899 1252630  2297032  2373218 2134208 1257601 2216773  5434625  2204942   906740  1254996 2224330  2363552

    1st column KB: Indicates the file size that was used for the testing.
    2nd column reclen: Indicates the record length that was used for the testing.
    3rd column until the last column: Indicates the various tests that are performed and its output values in per second.

-n and  -g option, specify file size range

   Command line used: iozone -n 1g -g 2g -a
                                                   random  random    bkwd   record   stride      
     KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
1048576      64 1008942 1428607  2263735  2513172 2471750 1516633 2342705  5436863  2472584  1011489  1322385 2291308  2510891
...

-s specify only one file size, while -n,-g give a range of file size for test. -n defualt is 64k.

Note, -n and -g have to work with -a option, while -s is not the case.

3 -i option, used to specify which tests to run

Here are list of tests you run, you can run one or more in one test, 0 must be specified for it creates a file, auto mode does them all.

    0=write/rewrite
    1=read/re-read
    2=random-read/write
    3=Read-backwards
    4=Re-write-record
    5=stride-read
    6=fwrite/re-fwrite
    7=fread/Re-fread,
    8=random mix
    9=pwrite/Re-pwrite
    10=pread/Re-pread
    11=pwritev/Re-pwritev
    12=preadv/Re-preadv

Here is one example

    Command line used: iozone -a -i 0 -i 1 -s 2g
                                                    random  random    bkwd   record   stride      
      KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
2097152       4  867286 1212607  2085904  2200394                                                
..
2097152   16384  802502 1021890  1235077  1252524

-r option, to  specify the record size

It's in Kbytes, to test. One may also specify -r #k (size in Kbytes) or -r #m (size in Mbytes) or -r #g (size in Gbytes).
This option is particular important, especially you are testing a filesystem for database, then you need to specify the record size equal to db block size, mostly 4k. Also, it also helps when you test a storage throughput.

For example.
If you have an array, raid6 8+2, segment size 64k size, then you probably want to get the best throughput by using -r 512. Here is a test output from one of my old SAN storage. 8+2 raid6, dual 4Gb FC connection

    Command line used: /usr/bin/iozone -s24g -r512 -i 0 -i 1 -t 1 -b iozone_lun1_2013-11-01-23-49.xls
    Output is in Kbytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 1024 Kbytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
    Throughput test with 1 process
    Each process writes a 25165824 Kbyte file in 512 Kbyte records

    Children see throughput for  1 initial writers     =  464439.69 KB/sec
    Parent sees throughput for  1 initial writers     =  371337.60 KB/sec
    Min throughput per process             =  464439.69 KB/sec
    Max throughput per process             =  464439.69 KB/sec
    Avg throughput per process             =  464439.69 KB/sec
    Min xfer                     = 25165824.00 KB

    Children see throughput for  1 rewriters     =  469936.88 KB/sec
    Parent sees throughput for  1 rewriters     =  375833.57 KB/sec
    Min throughput per process             =  469936.88 KB/sec
    Max throughput per process             =  469936.88 KB/sec
    Avg throughput per process             =  469936.88 KB/sec
    Min xfer                     = 25165824.00 KB

    Children see throughput for  1 readers         =  381264.19 KB/sec
    Parent sees throughput for  1 readers         =  381205.64 KB/sec
    Min throughput per process             =  381264.19 KB/sec
    Max throughput per process             =  381264.19 KB/sec
    Avg throughput per process             =  381264.19 KB/sec
    Min xfer                     = 25165824.00 KB

    Children see throughput for 1 re-readers     =  378713.69 KB/sec
    Parent sees throughput for 1 re-readers     =  378627.10 KB/sec
    Min throughput per process             =  378713.69 KB/sec
    Max throughput per process             =  378713.69 KB/sec
    Avg throughput per process             =  378713.69 KB/sec
    Min xfer                     = 25165824.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 512 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  464439.69

"        Rewrite "  469936.88

"           Read "  381264.19

"        Re-read "  378713.69
iozone test complete.

 -t  Run Iozone in a throughput mode

This option allows the user to specify how many threads or processes to have active during the measurement.
Already showed the test output in previous test throughput, I use it almost all the time. To simulate multiple threads test, specify -t 10 for example.

-F Specify  each of the temporary file names to be used in the throughput testing.

This option allows you to combine multiple tests into one, very useful if you want to know a host i/o capability on multiple LUNs.

Here is an example:
    Command line used: /usr/bin/iozone -s96g -i 0 -i 1 -r2048 -S2048 -t 2 -F /lun0/test1 /lun1/test1
    Output is in Kbytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 2048 Kbytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
    Throughput test with 2 processes
    Each process writes a 100663296 Kbyte file in 2048 Kbyte records

    Children see throughput for  2 initial writers     = 1514665.19 KB/sec
    Parent sees throughput for  2 initial writers     = 1404630.06 KB/sec
    Min throughput per process             =  739716.50 KB/sec
    Max throughput per process             =  774948.69 KB/sec
    Avg throughput per process             =  757332.59 KB/sec
    Min xfer                     = 96083968.00 KB

Note: The number of names should be equal to the number of processes or threads that are specified.

-l -u options Set  the  lower and upper  limit  on number of processes to run

similar with -t, but with flexible range.
When running throughput tests this option allows the user to specify the greatest number of processes or threads to start

Set minimum record size  and max record size

you can play with options -y and -g

-+m and  -+t    Enable network performance test.

Use with -+m
              Used  to  specify  a filename that will be used to specify the clients in a distributed measurement. The file contains one line for each client. The fields are space delimited. Field 1 is the client name. Field 2 is the working directory, on  the  client, where Iozone will run. Field 3 is the path to the executable Iozone on the client.

See another article How to use iozone to do distributed measurement

Report CPU utilization time, -+u

    Command line used: /usr/bin/iozone -s24g -r512 -i 0 -i 1 -t 1 -+u
    Children see throughput for  1 initial writers     =  470748.72 KB/sec
...
    CPU Utilization: Wall time   53.459    CPU time   42.402    CPU utilization  79.32 %
    Children see throughput for  1 rewriters     =  470614.66 KB/sec
...
    CPU utilization: Wall time   53.474    CPU time   51.333    CPU utilization  96.00 %
...
"Output is in CPU%"
"  Initial write "      79.32
"        Rewrite "      96.00
"           Read "      25.46
"        Re-read "      25.70