It has been a lot changed since I started using it many years ago, so I'd like to share something in general rather than particular version or on particular platform. Once you know what and how multipath works, it's easier for you to manage multipath configuration over versions and for different types of devices.

Device mapper multipathing (DM-Multipath) is a tool that allows you to configure multiple I/O paths between server and storage arrays into a single device, mostly used in external storage systems(SAN, iSCSI, tape etc..). Multipathing aggregates the I/O paths, creating a new device that consists of the aggregated paths.

However, there are many different types of external devices, as well hosts and HBAs, so there has to have functions to manage them, this is also one of the reason it keeps changing in configuration.


let's get started.

Installation packages

#yum install device-mapper-multipath

Quick check

Use tool mpathconf

# /sbin/mpathconf 
multipath is disabled
find_multipaths is disabled
user_friendly_names is enabled
dm_multipath module is loaded
multipathd is chkconfiged off

Quick and basic configuration

# /sbin/mpathconf --enable --find_multipaths y --user_friendly_names y --with_chkconfig y --with_multipathd y
Starting multipathd daemon:                                [  OK  ]

# /sbin/mpathconf
multipath is enabled
find_multipaths is enabled
user_friendly_names is enabled
dm_multipath module is loaded
multipathd is chkconfiged on

Multipath output list
# multipath -ll
mpathr (360001ff00d620e940f020100003a3818) dm-14 IBM,DCS9550
size=1.8T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 3:0:1:17 sdbv 68:144 active ready  running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 4:0:1:17 sdcj 69:112 active ready  running
..

Further configuration

Using /sbin/mpathconf just made a basic configuration, /etc/multipath.conf, with some examples, for device mapper

For a complete list of the default configuration values, see file
/usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults, or multipath.conf under the same directory on RHEL7/SL7

The default configuration file is a big one, but only has four sections:

Default, blacklist, multipaths and devices

And, all parameters explicitly set in /etc/multipath.conf will overwrite the default values. Usually devices section can be merged into default section if only one type of device on the host.

Let's go through 3 sections one by one.

Device section in multipath.conf

     This session usually doesn't show in /etc/multipath.conf unless you want to set your own setting for the particular device. For most of cases, my experience is that find your device setting in .conf.default, then put it to default section in your /etc/multipath.conf.

To identify you device, multipath uses both Vendor and production info, so you can get the info by scsi_id

#  /sbin/scsi_id --page=0x80 --whitelisted --device=/dev/sdb
SIBM     1818      FAStTSV14408879
#  /sbin/scsi_id --page=0x80 --whitelisted --device=/dev/sda
SLSI     Logical Volume   394949227760212329

Note: on RHEL7/SL7, scsi_id is located at /lib/udev/scsi_id

So, device sdb Vendor info is IBM and product is 1818, while sda from Vendor LSI and product description is Logical Volume(local disk)
So in .conf.default file, you can easily find devices section

#       device {
#               vendor "IBM"
#               product "^1818"
#               product_blacklist "Universal Xport"
#               path_grouping_policy group_by_prio
#               getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
#               path_selector "round-robin 0"
#               path_checker rdac
#               features "0"
#               hardware_handler "1 rdac"
#               prio rdac
#               failback immediate
#               rr_weight uniform
#               no_path_retry queue
#               rr_min_io 1000
#               rr_min_io_rq 1
#       }

Then, copy and paste the setting into default section in /etc/multipath.conf, you don't have to, but easier for you later check, or you can have devices section in /etc/multipath.conf too particularly you want you own setting for the device. Most of time settings from default do the best for the device.

Note: some of the parameters from devices section are not compatible with default section, just remove them out.

See Multipath configuration parameters explain for each parameters.

Default section in multipath.conf

Default session covers parameters not set in device session, of couse, also for unknown devices.

defaults {
    udev_dir                /dev
    polling_interval        15
    path_selector "round-robin 0"
    path_grouping_policy group_by_prio
    path_checker rdac
    getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
    rr_min_io 1000
    rr_min_io_rq 1
    rr_weight uniform
    failback immediate
    no_path_retry 3
    user_friendly_names     yes
    features "0"
    prio rdac
    max_fds                 4096
    find_multipaths yes
    log_checker_err always
}

Blacklist section in multipath.conf

Blacklist session tells multipathd which device you don't want to use multipath, for example, local disk, usb devices etc..

You can blacklist devices by wwid, device name, devnode or by vendor etc..

Here I have an example to blacklist local disk, good for large number of servers using an identical multipath.conf in a large SAN environment

blacklist {
        device {
                vendor "LSI"
                product "Logical Volume"
        }
    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
}

Multipaths section in multipath.conf

Note: this section works only you set user_friendly_names to yes.

Define you real device name here.

multipaths {
          multipath {
                     wwid    360080e50002d26520000067a4f7a3fb8
                     alias   dcsunit01_lun0
          }
...
}

With the configuration above, the device /dev/mapper/dcsunit01_lun0 is the device you want to use in your application.

You can use scsi_id to get each device wwid, or get them from storage management tool.

Start to use multipathd

As we mentioned, mpathconf can be used to start/stop multipath daemon, module and chkconfig, you can  also do it with command line

Runservice start multipathd to start mulltipath daemon, then check devices

#multipath -v2
# multipath -ll
dcsunit03_lun10 (360080e50002d12b4000005814f35cb40) dm-34 IBM,1818      FAStT
size=35T features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| |- 5:0:3:10 sdav 66:240  active ready running
| `- 6:0:3:10 sddp 71:112  active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
  |- 5:0:4:10 sdbh 67:176  active ready running
  `- 6:0:4:10 sdeb 128:48  active ready running
...

To see one device info

# multipath -ll dcsunit02_lun1
dcsunit02_lun1 (360080e50002d029c000005a74f7a402b) dm-13 IBM,1818      FAStT
size=35T features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| |- 5:0:2:1  sdaa 65:160  active ready running
| `- 6:0:2:1  sdcu 70:32   active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
  |- 5:0:1:1  sdo  8:224   active ready running
  `- 6:0:1:1  sdci 69:96   active ready running

Flush device map away(device in use can't be flushed away)

# multipath -F
Create map with verbose output
# multipath -v2
...

To flush all devices, umount file systems which using multipath devices, or stop applications using multipath devices, then run multipath -F

Debug

multipathd supports interactive mode for debug.
       -k     multipathd will enter interactive mode. From this mode, the available commands can be viewed by entering "help".  When  you  are finished entering commands, press CTRL-D to quit.

# multipathd -k
multipathd> help
multipath-tools v0.4.9 (04/04, 2009)
CLI commands reference:
 list|show paths
 list|show paths format $format
 list|show status
...


The most useful part I used is to check current configuration, blacklist devices, devices not monitored etc..

multipathd> show blacklist
device node rules:
...
wwid rules:
- blacklist:
        <empty>
- exceptions:
        <empty>
device rules:
- blacklist:
        (config file rule) LSI:Logical Volume
- blacklist:
        (config file rule) LSI:Logical Volume
        (default rule)     DGC:LUNZ
...

Or, show devices multipathd found

multipathd> show device
available block devices:
    ram0 devnode blacklisted, unmonitored
...
    loop7 devnode blacklisted, unmonitored
    sda
    sdb devnode whitelisted, monitored
...

Show configuration

multipathd> show config
..

Extra for device parameters settings.

polling_interval
interval between two path checks in seconds For properly functioning paths, the interval between checks will gradually increase to (4 * polling_interval); default is 5
user_friendly_names
If set to yes , using the bindings file /etc/multipath/bindings to assign a persistent and unique alias to the multipath, in the form of mpath<n>. If set to no use the WWID as the alias. In either case this be will be overriden by any specific aliases in the multipaths section. Default is no

max_fds
Specify the maximum number of file descriptors that can be opened by multipath and multipathd. This is equivalent to ulimit -n. A value of max will set this to the system limit from /proc/sys/fs/nr_open. If this is not set, the maximum number of open fds is taken from the calling process. It is usually 1024. To be safe, this should be set to the maximum number of paths plus 32, if that number is greated than 1024.

find_multipaths
The Red Hat Enterprise Linux 6 release provides a new mode for setting up multipath devices, which you set with the find_multipaths configuration file parameter. In previous releases of Red Hat Enterprise Linux, multipath always tried to create a multipath device for every path that was not explicitly blacklisted. In Red Hat Enterprise Linux 6, however, if the find_multipaths configuration parameter is set to yes, then multipath will create a device only if one of three conditions are met:

    There are at least two non-blacklisted paths with the same WWID.
    The user manually forces the device creation, by specifying a device with the multipath command.
    A path has the same WWID as a multipath device that was previously created (even if that multipath device does not currently exist). For instructions on the procedure to follow if you have previously created multipath devices when the find_multipaths parameter was not set, see

features "0" and no_path_retry

Specify the number of retries until disable queueing, or fail for immediate failure (no queueing), queue for never stop queueing. Default is 0.
Issues with queue_if_no_path feature(this part from RHEL portal)
If features "1 queue_if_no_path" is specified in the /etc/multipath.conf file, then any process that issues I/O will hang until one or more paths are restored. To avoid this, set the no_path_retry N parameter in the /etc/multipath.conf file (where N is the number of times the system should retry a path).

When you set the no_path_retry parameter, remove the features "1 queue_if_no_path" option from the /etc/multipath.conf file as well. If, however, you are using a multipathed device for which the features "1 queue_if_no_path" option is set as a compiled-in default, as it is for many SAN devices, you must explicitly add features "0" to override this default. You can do this by copying the existing devices section for your device from /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults into /etc/multipath.conf and editing it to suit your needs.
If you need to use the features "1 queue_if_no_path" option and you experience the issue noted here, use the dmsetup command to edit the policy at runtime for a particular LUN (that is, for which all the paths are unavailable). For example, if you want to change the policy on the multipath device mpathc from "queue_if_no_path" to "fail_if_no_path", execute the following command.