Mdadm is an utility to manage Software array on Linux. Here is an example show you how to fix an array that is inactive state. For more infor about mdadm, see Mdadm, a tool for software array on linux

Check the status and detail info of the mdadm array

Suppose the inactive array name is md7, 12 drives RAID6 array.

Here is the detail info looks like

# mdadm --detail /dev/md7
/dev/md7:
        Version : 1.2
     Raid Level : raid0
  Total Devices : 12
    Persistence : Superblock is persistent

          State : inactive

           Name : fibrevillage:7  (local to host fibrevillage.com)
           UUID : 675bbb1f:1538972a:fdc81e47:acc7bf3d
         Events : 217856

    Number   Major   Minor   RaidDevice

       -       8       96        -        /dev/sdg
       -       8      112        -        /dev/sdh
       -       8      224        -        /dev/sdo
       -       8      240        -        /dev/sdp
       -      65       96        -        /dev/sdw
       -      65      112        -        /dev/sdx
       -      65      224        -        /dev/sdae
       -      65      240        -        /dev/sdaf
       -      66       96        -        /dev/sdam
       -      66      112        -        /dev/sdan
       -      66      224        -        /dev/sdau
       -      66      240        -        /dev/sdav

As you can see the array state is 'inactive' and raid type is wrong.

Check the array running state

# cat /run/mdadm/map
md7 1.2 1fbb5b67:2a973815:471ec8fd:3dbfc7ac /dev/md/7
...

Seems ok

Scan the array in verbose mode

# mdadm --detail --scan -v /dev/md7
INACTIVE-ARRAY /dev/md7 num-devices=12 metadata=1.2 name=fibrevillage.com:7 UUID=675bbb1f:1538972a:fdc81e47:acc7bf3d
   devices=/dev/sdae,/dev/sdaf,/dev/sdam,/dev/sdan,/dev/sdau,/dev/sdav,/dev/sdg,/dev/sdh,/dev/sdo,/dev/sdp,/dev/sdw,/dev/sdx

You can see it's state is inactive

Try to start it in readonly mode

mdadm --run --readonly /dev/md7

failed

Exam each individual drive

# mdadm --examine /dev/sdg
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 675bbb1f:1538972a:fdc81e47:acc7bf3d
           Name : fibrevillage.com:7  (local to host fibrevillage.com)
  Creation Time : Tue Jan 31 12:41:54 2017
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
     Array Size : 9766314240 (9313.88 GiB 10000.71 GB)
  Used Dev Size : 1953262848 (931.39 GiB 1000.07 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=176 sectors
          State : active
    Device UUID : 6695c9c6:f2ac3032:87a75d4d:3cf3154c

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Mar 20 23:45:59 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 49d25b0a - correct
         Events : 217856

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 0
   Array State : AAA.AA.AAAAA ('A' == active, '.' == missing, 'R' == replacing)

So, you can see some drives are missing in drive's meta data.

Let's see other drives

# mdadm --examine /dev/sdh
/dev/sdh:

   Array State : AAA.AA.AAAAA ('A' == active, '.' == missing, 'R' == replacing)
...
   Array State : AAA.AA.AAAAA ('A' == active, '.' == missing, 'R' == replacing)

...

See few more

# mdadm --examine /dev/sdp
/dev/sdp:
    ...

   Device Role : Active device 3
   Array State : AAAAAA.AAAAA ('A' == active, '.' == missing, 'R' == replacing)
...
# mdadm --examine /dev/sde
/dev/sde:
 ...
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
...

Haha, you can see the drives info in different drive meta are inconsistent. This is why the whole array became inactive

Stop the array

Run commmand below to stop the array

# mdadm -S /dev/md7
mdadm: stopped /dev/md7
# mdadm --detail /dev/md7
mdadm: cannot open /dev/md7: No such file or directory

Re-assemble the array

To re-assemble the array, use existing uuid, run command

# mdadm --assemble /dev/md7 --uuid=675bbb1f:1538972a:fdc81e47:acc7bf3d 
mdadm: /dev/md7 assembled from 10 drives - not enough to start the array while not clean - consider --force.

As you can see, when re-assembling the array, it only can detect 10 drives, 2 are missing

So, check it again, then run re-assemble in force mode

# mdadm --detail /dev/md7
mdadm: cannot open /dev/md7: No such file or directory
# mdadm --assemble /dev/md7 --force --uuid=675bbb1f:1538972a:fdc81e47:acc7bf3d
mdadm: Marking array /dev/md7 as 'clean'
mdadm: /dev/md7 has been started with 10 drives (out of 12).

Check the re-assembled array


# mdadm --detail /dev/md7
/dev/md7:
        Version : 1.2
  Creation Time : Tue Jan 31 12:41:54 2017
     Raid Level : raid6
     Array Size : 9766314240 (9313.88 GiB 10000.71 GB)
  Used Dev Size : 976631424 (931.39 GiB 1000.07 GB)
   Raid Devices : 12
  Total Devices : 10
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Mar 20 23:45:59 2017
          State : clean, degraded
 Active Devices : 10
Working Devices : 10
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

           Name : fibrevillage.com:7  (local to host fibrevillage.com)
           UUID : 675bbb1f:1538972a:fdc81e47:acc7bf3d
         Events : 217856

    Number   Major   Minor   RaidDevice State
       0       8       96        0      active sync   /dev/sdg
       1       8      112        1      active sync   /dev/sdh
       2       8      224        2      active sync   /dev/sdo
       6       0        0        6      removed
       4      65       96        4      active sync   /dev/sdw
       5      65      112        5      active sync   /dev/sdx
      12       0        0       12      removed
       7      65      240        7      active sync   /dev/sdaf
       8      66       96        8      active sync   /dev/sdam
       9      66      112        9      active sync   /dev/sdan
      10      66      224       10      active sync   /dev/sdau
      11      66      240       11      active sync   /dev/sdav

Replace removed drives

Replace the two drives that marked removed with new ones, they are the trouble makers.

Add new drives into the array

After drive replacement, add them back

# mdadm --re-add /dev/md7 /dev/sdp
mdadm: re-added /dev/sdp
# mdadm --re-add /dev/md7 /dev/sdae
mdadm: re-added /dev/sdae

Check the array status

# mdadm --detail /dev/md7
/dev/md7:
        Version : 1.2
  Creation Time : Tue Jan 31 12:41:54 2017
     Raid Level : raid6
     Array Size : 9766314240 (9313.88 GiB 10000.71 GB)
  Used Dev Size : 976631424 (931.39 GiB 1000.07 GB)
   Raid Devices : 12
  Total Devices : 12
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Mar 30 11:17:45 2017
          State : clean, degraded, recovering
 Active Devices : 10
Working Devices : 12
 Failed Devices : 0
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 128K

 Rebuild Status : 50% complete

           Name : fibrevillage.com:7  (local to host fibrevillage.com)
           UUID : 675bbb1f:1538972a:fdc81e47:acc7bf3d
         Events : 217872

    Number   Major   Minor   RaidDevice State
       0       8       96        0      active sync   /dev/sdg
       1       8      112        1      active sync   /dev/sdh
       2       8      224        2      active sync   /dev/sdo
       3       8      240        3      spare rebuilding   /dev/sdp
       4      65       96        4      active sync   /dev/sdw
       5      65      112        5      active sync   /dev/sdx
       6      65      224        6      spare rebuilding   /dev/sdae
       7      65      240        7      active sync   /dev/sdaf
       8      66       96        8      active sync   /dev/sdam
       9      66      112        9      active sync   /dev/sdan
      10      66      224       10      active sync   /dev/sdau
      11      66      240       11      active sync   /dev/sdav

Repair the file system

You are luck if you don't see any problem when mounting the file system back, if there is, try to fix it

# xfs_repair -n /dev/md7
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
sb_ifree 1787, counted 1783
sb_fdblocks 1686315926, counted 1686319481
        - 11:30:11: scanning filesystem freespace - 32 of 32 allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - 11:30:11: scanning agi unlinked lists - 32 of 32 allocation groups done
        - process known inodes and perform inode discovery...
        - agno = 15
        - agno = 30
        - agno = 0
        - agno = 16
        - agno = 31
        - agno = 1
        - agno = 17
        - agno = 2
        - agno = 18
        - agno = 3
        - agno = 4
        - agno = 19
        - agno = 5
        - agno = 20
        - agno = 6
        - agno = 21
        - agno = 7
        - agno = 22
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 23
        - agno = 24
        - agno = 25
        - agno = 26
        - agno = 27
        - agno = 28
        - agno = 29
        - 11:30:12: process known inodes and inode discovery - 9024 of 9024 inodes done
        - process newly discovered inodes...
        - 11:30:12: process newly discovered inodes - 32 of 32 allocation groups done
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - 11:30:12: setting up duplicate extent list - 32 of 32 allocation groups done
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 8
        - agno = 12
        - agno = 11
        - agno = 15
        - agno = 6
        - agno = 7
        - agno = 18
        - agno = 2
        - agno = 10
        - agno = 3
        - agno = 4
        - agno = 13
        - agno = 14
        - agno = 5
        - agno = 16
        - agno = 26
        - agno = 9
        - agno = 19
        - agno = 20
        - agno = 30
        - agno = 22
        - agno = 23
        - agno = 24
        - agno = 25
        - agno = 17
        - agno = 27
        - agno = 28
        - agno = 29
        - agno = 21
        - agno = 31
        - 11:30:12: check for inodes claiming duplicate blocks - 9024 of 9024 inodes done
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

Above is a dry run, no changes made, for more example of repairing a xfs file system, see How to repair a xfs filesystem