iostat is one of mostly used performance tools for troubleshooting on Linux, it reports CPU statistics and input/output statistics for devices, partitions and NFS.
Here are some useful command examples:
Basic command examples
Display a single history since boot report for all CPU and Devices
iostat
Display a continuous device report at two second intervals.
iostat -d 2
Display six reports at two second intervals for all devices.
iostat -d 2 6
Omit the first report in multiple report mode.
In multiple reports, the first one always shows the statistics since boot, to omit the first report, try
iostat -y -d 2 6
In the case above, the total number of output is 5
Display six reports of extended statistics at two second intervals for devices hda and hdb.
iostat -x hda hdb 2 6
Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)
iostat -p sda 2 6
Report output explanation
The iostat command generates two types of reports, the CPU Utilization report and the Device Utilization report.
CPU Utilization Report
The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the following format:
%user
Show the percentage of CPU utilization that occurred while executing at the user level
%nice
Show the percentage of CPU utilization that occurred while executing at the user level
with nice priority.
%system
Show the percentage of CPU utilization that occurred while executing at the system level
%iowait
Show the percentage of time that the CPU or CPUs were idle during which the system had
an outstanding disk I/O request.
%steal
Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the
hypervisor was servicing another virtual processor.
%idle
Show the percentage of time that the CPU or CPUs were idle and the system did not have
an outstanding disk I/O request.
Device Utilization Report
The device report provides statistics on a per physical device or partition basis. Block devices and partitions for which statistics are to be displayed may be entered on the command line. If no device nor partition is entered, then statistics are displayed for every device used by the system, and providing that the kernel maintains statistics for it. If the ALL keyword is given on the command line, then statistics are displayed for every device defined by the system, including those that have never been used. Transfer rates are shown in 1K blocks by default, unless the environment variable POSIXLY_CORRECT is set, in which case 512-byte blocks are used. The report may show the following fields, depending on the flags used:
Device:
This column gives the device (or partition) name as listed in the /dev directory.
tps
Indicate the number of transfers per second that were issued to the device.
A transfer is an I/O request to the device. Multiple logical requests can be combined
into a single I/O request to the device. A transfer is of indeterminate size.
Blk_read/s (kB_read/s, MB_read/s)
Indicate the amount of data read from the device expressed in a number of blocks
per second. Blocks are equivalent to sectors and therefore have a size of 512 bytes.
Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s)
Indicate the amount of data written to the device expressed in a number of blocks per second.
Blk_read (kB_read, MB_read)
The total number of blocks (kilobytes, megabytes) read.
Blk_wrtn (kB_wrtn, MB_wrtn)
The total number of blocks (kilobytes, megabytes) written.
rrqm/s
The number of read requests merged per second that were queued to the device.
wrqm/s
The number of write requests merged per second that were queued to the device.
r/s
The number (after merges) of read requests completed per second for the device.
w/s
The number (after merges) of write requests completed per second for the device.
rsec/s (rkB/s, rMB/s)
The number of sectors (kilobytes, megabytes) read from the device per second.
wsec/s (wkB/s, wMB/s)
The number of sectors (kilobytes, megabytes) written to the device per second.
avgrq-sz
The average size (in sectors) of the requests that were issued to the device.
avgqu-sz
The average queue length of the requests that were issued to the device.
await
The average time (in milliseconds) for I/O requests issued to the device to be served.
This includes the time spent by the requests in queue and the time spent servicing them.
r_await
The average time (in milliseconds) for read requests issued to the device to be served.
This includes the time spent by the requests in queue and the time spent servicing them.
w_await
The average time (in milliseconds) for write requests issued to the device to be served.
This includes the time spent by the requests in queue and the time spent servicing them.
svctm
The average service time (in milliseconds) for I/O requests that were issued to the device
Warning!
Do not trust this field any more. This field will be removed in a future sysstat version.
%util
Percentage of elapsed time during which I/O requests were issued to the device
(bandwidth utilization for the device).
Device saturation occurs when this value is close to 100%.
Display only CPU statistics
# iostat -c 2 2
Linux 2.6.32-504.12.2.el6.x86_64 07/18/15 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.73 0.00 0.99 0.49 0.00 97.78
avg-cpu: %user %nice %system %iowait %steal %idle
0.44 0.00 0.61 0.15 0.00 98.81
Display only NFS statistics
iostat -n
Linux 2.6.32-504.1.3.el6.x86_64 07/18/15 _x86_64_ (4 CPU)
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s wBlk_dir/s rBlk_svr/s wBlk_svr/s ops/s rops/s wops/s
Display only disk statistics
# iostat -d
Linux 2.6.32-504.12.2.el6.x86_64 07/18/15 _x86_64_ (24 CPU)
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 6.97 655.35 407.92 4928340558 3067622213
sdc 10.00 12727.00 7164.52 95709420584 53878546848
sdj 7.37 10625.38 1629.56 79904801512 12254573904
...
Display one or multiple disks io statistics
iostat -p sda,sdc
Linux 2.6.32-504.12.2.el6.x86_64 07/18/15 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.73 0.00 0.99 0.49 0.00 97.78
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 6.97 655.33 407.90 4928434150 3067640781
sda1 0.00 0.00 0.00 12809 1109
sda2 0.63 29.94 0.16 225149943 1191880
sda3 1.85 34.05 14.01 256090276 105332840
sda4 4.49 591.33 393.73 4447179714 2961114952
sdc 10.00 12727.05 7164.27 95714993512 53879554624
Note: with option '-p', iostat displays statistics for block devices and all their partitions, sda in the example above
Display one or multiple disks by persistent disk name
Especially useful if you have multpath devices,here is the syntax
-j { ID | LABEL | PATH | UUID | ... } [ device [...] | ALL ]
Display persistent device names. Options ID, LABEL, etc. specify the type of the persistent name. These options are not limited, only prerequisite is that directory with required persistent names is present in /dev/disk. Optionally, multiple devices can be specified in the chosen persistent name type.
# iostat -m -j ID dm-name-lun8 dm-name-lun0
Linux 2.6.32-504.12.2.el6.x86_64 07/18/15 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.73 0.00 0.99 0.49 0.00 97.79
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
dm-name-lun0 15.73 10.70 2.01 80508087 15138946
dm-name-lun8 14.56 10.78 1.56 81079650 11733230
Note: The persistent name is from /dev/disk/by-id/
iostat Display I/O data in group
# iostat -g OSdisk sda sdi 6 2
Linux 2.6.32-504.12.2.el6.x86_64 07/18/15 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.07 0.08 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 28.11 1782.01 0.85 1953498023 932957
sdi 28.09 1781.82 0.85 1953294311 934109
OSdisk 56.20 3563.83 1.70 3906792334 1867066
Don't want show individual disks in the group output? Try this:
# iostat -g OSdisk sda sdi 6 2
Linux 2.6.32-504.12.2.el6.x86_64 07/18/15 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.07 0.08 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
OSdisk 56.20 3563.83 1.70 3906792334 1867066
iostat Display I/O data in MB/second
Default output is in KB/sec, this command shows output in MB/sec
iostat -m -j ID dm-name-lun8 dm-name-lun0 2 2
..
avg-cpu: %user %nice %system %iowait %steal %idle
0.29 0.00 0.99 0.00 0.00 98.72
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
dm-name-lun0 0.00 0.00 0.00 0 0
dm-name-lun8 36.00 72.00 0.00 144 0
iostat Display with timestamp
Want timestamp in the output? here is it
iostat -t -p sdc 2 2
Linux 2.6.32-504.12.2.el6.x86_64 07/18/15 _x86_64_ (24 CPU)
07/18/15 13:40:56
avg-cpu: %user %nice %system %iowait %steal %idle
0.73 0.00 0.99 0.49 0.00 97.79
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sdc 10.00 12725.99 7162.96 95724608304 53879612960
07/18/15 13:40:58
avg-cpu: %user %nice %system %iowait %steal %idle
0.19 0.00 0.65 0.00 0.00 99.16
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sdc 0.00 0.00 0.00 0 0
Display extended statistics info
iostat -j ID dm-name-lun8 dm-name-lun0 -x -m 2 2
Linux 2.6.32-504.12.2.el6.x86_64 07/18/15 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.73 0.00 0.99 0.49 0.00 97.79
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
dm-name-lun0 4.47 1.04 13.24 1.92 10.57 1.94 1689.57 0.15 9.86 2.31 3.51
dm-name-lun8 4.65 0.85 12.87 1.68 10.78 1.56 1736.35 0.02 1.49 2.49 3.63
Where
rrqm/s : The number of read requests merged per second that were queued to the hard disk
wrqm/s : The number of write requests merged per second that were queued to the hard disk
r/s : The number of read requests per second
w/s : The number of write requests per second
rsec/s : The number of sectors read from the hard disk per second
wsec/s : The number of sectors written to the hard disk per second
avgrq-sz : The average size (in sectors) of the requests that were issued to the device.
avgqu-sz : The average queue length of the requests that were issued to the device
await : The average time (in milliseconds) for I/O requests issued to the device to be served.
This includes the time spent by the requests in queue and the time spent servicing them.
svctm : The average service time (in ms) for I/O requests that were issued to the device
%util : Percentage of CPU time during which I/O requests were issued to the device .
Device saturation occurs when this value is close to 100%.
