step by step replacing disks in Solaris Volume Manager environment

This article describes how to replace disks in a Solaris Volume Manager environment. I use a real case happened on Sun Fire 4540 server, c0t0, one of the system disk failed. I've been used to use script to capture the screen output during the disk replacement, quite useful.

Step #1  identify the failed disk

        a. Alert from ILOM SNMP
        b. Our own daily raid monitoring script
        c. you can also identify the failed disk by examining the /var/adm/messages file and the metastat command output
        d. fmadm faulty

Example of fmadm faulty output

       # fmadm faulty
--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Aug 20 15:38:58 78675c38-dfe1-60ab-c41d-fa3cd7f9a47c  DISK-8000-0X   Major

Host        : spool12
Platform    : Sun-Fire-X4540    Chassis_id  : 0939AMR058
Product_sn  :

Fault class : fault.io.disk.predictive-failure
Affects     : dev:///:devid=id1,sd@n5000c50015c7e84f//pci@0,0/pci10de,377@a/pci1000,1000@0/sd@0,0
                  faulted but still in service
FRU         : "bay0" (hc://:product-id=Sun-Fire-X4540:server-id=spool12:chassis-id=0939AMR058:serial=9QJ51SC8:part=ATA-SEAGATE-ST31000N:revision=SU0E/bay=0/disk=0)
                  faulty

Description : SMART health-monitoring firmware reported that a disk
              failure is imminent.
              Refer to http://sun.com/msg/DISK-8000-0X for more information.

Response    : None.

Impact      : It is likely that the continued operation of
              this disk will result in data loss.

Action      : Schedule a repair procedure to replace the affected disk.
              Use fmdump -v -u <EVENT_ID> to identify the disk.

Step #2 Locate any state database replicas that might have been placed on the failed disk.

        # metadb -i
        flags           first blk       block count
      Wm  p  l          16              8192            /dev/dsk/c0t0d0s6
      W   p  l          8208            8192            /dev/dsk/c0t0d0s6
      W   p  l          16400           8192            /dev/dsk/c0t0d0s6
     a   p  luo        16              8192            /dev/dsk/c1t0d0s6
     a   p  luo        8208            8192            /dev/dsk/c1t0d0s6
     a   p  luo        16400           8192            /dev/dsk/c1t0d0s6

The output shows three state database replicas on each slice 6 of the local disks, c0t0d0 and c1t0d0. The W in the flags field of the c0t0d0s6 slice indicates that the device has write errors. Three replicas on the c1t0d0s6 slice are still good.

Step #3 delete the state database replicas

Record the slice name where the state database replicas reside and the number of state database replicas. Then, delete the state database replicas(If you also like using script to capture you screen output, then it's easy)

# metadb -d /dev/dsk/c0t0d0s6
# metadb
        flags           first blk       block count
     a    p  luo        16              8192            /dev/dsk/c1t0d0s6
     a    p  luo        8208            8192            /dev/dsk/c1t0d0s6
     a    p  luo        16400           8192            /dev/dsk/c1t0d0s6

        You can see that the state replicase on c0t0d0 are gone

Step #4 Replace the failed disk

(If you have hot spares on the failed disk, remove them first by using 'metahs -d', I don't have in my case).

Step #5 Update the Solaris Volume Manager state database with the device ID for the new disk

#metadevadm -u c0t0d0

Step #6 Repartition the new disk.

#  prtvtoc /dev/rdsk/c1t0d0s6 |  fmthard -s - /dev/rdsk/c0t0d0s6
fmthard: Partition 2 specifies the full disk and is not equal
full size of disk.  The full disk capacity is 1953439740 sectors.
fmthard:  New volume table of contents now in place.
#  prtvtoc /dev/rdsk/c0t0d0s6
* /dev/rdsk/c0t0d0s6 partition map
*
* Dimensions:
*     512 bytes/sector
*     126 sectors/track
*     255 tracks/cylinder
*   32130 sectors/cylinder
*   60800 cylinders
*   60798 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
* Unallocated space:
*       First     Sector    Last
*       Sector     Count    Sector
*           0  83762910  83762909
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      0    00   83762910   2120580  85883489
       1      0    00      32130  83730780  83762909
       2      5    00          0 1953407610 1953407609
       3      0    00   85883490  10249470  96132959
       4      0    00   96132960  20498940 116631899
       5      0    00  116631900   8193150 124825049
       6      0    00  124825050     32130 124857179
       7      0    00  124857180 1828550430 1953407609
       8      1    01          0     32130     32129

In the case above, I used the good disk's partition table to create the second one's

Step #7 Create state database replicas on newly replaced disks

# metadb
        flags           first blk       block count
     a    p  luo        16              8192            /dev/dsk/c1t0d0s6
     a    p  luo        8208            8192            /dev/dsk/c1t0d0s6
     a    p  luo        16400           8192            /dev/dsk/c1t0d0s6
#  metadb -a -c 3 c0t0d0s6
# metadb
        flags           first blk       block count
     a        u         16              8192            /dev/dsk/c0t0d0s6
     a        u         8208            8192            /dev/dsk/c0t0d0s6
     a        u         16400           8192            /dev/dsk/c0t0d0s6
     a    p  luo        16              8192            /dev/dsk/c1t0d0s6
     a    p  luo        8208            8192            /dev/dsk/c1t0d0s6
     a    p  luo        16400           8192            /dev/dsk/c1t0d0s6

Step #8 Rsync data to new slices

run the metareplace -e command for each slice to rsync data to new slices.

# metareplace -e d10 c0t0d0s0
d10: device c0t0d0s0 is enabled
# metareplace -e d20 c0t0d0s1
d20: device c0t0d0s1 is enabled
# metareplace -e d30 c0t0d0s3
d30: device c0t0d0s3 is enabled

# metareplace -e d40 c0t0d0s4
d40: device c0t0d0s4 is enabled

# metareplace -e d50 c0t0d0s5
d50: device c0t0d0s5 is enabled
# metareplace -e d70 c0t0d0s7
d70: device c0t0d0s7 is enabled
Note: If you have any soft partitions are built directly on slices on the replaced disk, run the metarecover -m -p command on each slice that contains soft partitions. This command regenerates the extent headers on disk. it is not my case.

Step #9 double check if everything is ok.

# metadb
        flags           first blk       block count
     a        u         16              8192            /dev/dsk/c0t0d0s6
     a        u         8208            8192            /dev/dsk/c0t0d0s6
     a        u         16400           8192            /dev/dsk/c0t0d0s6
     a    p  luo        16              8192            /dev/dsk/c1t0d0s6
     a    p  luo        8208            8192            /dev/dsk/c1t0d0s6
     a    p  luo        16400           8192            /dev/dsk/c1t0d0s6
# metastat | grep Resyncing
      State: Resyncing
    State: Resyncing
        c0t0d0s7          0     No       Resyncing   Yes
      State: Resyncing
    State: Resyncing
        c0t0d0s5          0     No       Resyncing   Yes
      State: Resyncing
    State: Resyncing
        c0t0d0s3          0     No       Resyncing   Yes
      State: Resyncing
    State: Resyncing
        c0t0d0s1          0     No       Resyncing   Yes
      State: Resyncing
    State: Resyncing
        c0t0d0s0          0     No       Resyncing   Yes
      State: Resyncing
    State: Resyncing
        c0t0d0s4          0     No       Resyncing   Yes

Step #10 mark the problem repaired by fmadm

Use fmadm to mark the problem repaired, so the service led on the disk will be switched off.

# fmadm faulty -r
dev:///:devid=id1,sd@n5000c50015c7e84f//pci@0,0/pci10de,377@a/pci1000,1000@0/sd@0,0 degraded
# fmadm repaired dev:///:devid=id1,sd@n5000c50015c7e84f//pci@0,0/pci10de,377@a/pci1000,1000@0/sd@0,0
fmadm: recorded repair to of dev:///:devid=id1,sd@n5000c50015c7e84f//pci@0,0/pci10de,377@a/pci1000,1000@0/sd@0,0

I don't have idea why replace option does not work, never looked into it, let me know if you know, thanks.

# fmadm replaced dev:///:devid=id1,sd@n5000c50015c7e84f//pci@0,0/pci10de,377@a/pci1000,1000@0/sd@0,0
fmadm: failed to record replacement of dev:///:devid=id1,sd@n5000c50015c7e84f//pci@0,0/pci10de,377@a/pci1000,1000@0/sd@0,0: specified resource has not been replaced