How To Raid In Linux With Mdadm?

Hi, today I am gonna show you how to make raid ( redundant array of inexpensive disks ). Why wee need it? Think about that you have customer data stored in a server. Say one day disk is dead and then all of your data about customer are gone. This is a very bad new.  What if disk dies but the data remains the same.

This will be very good. You can think that will disk dies frequently. No but it is not impossible, especially in enterprise side or in personal use in home. RAID is a technology to backup disks with extra disks. Say we have 3 disks which are used separate projects. And we want to backup them with high availibility in particular working hours. RAID can be done with separate hardware, firmware or software. Making raid with hardware is the best, fast and reliable way but it may be expensive or too much for project. Firmware raid is generally provided by motherboard manufacturers as a cheap option for hardware raid. Software raid is the cheapest and least reliable way to mail raid. But it can be suitable for home or unprofessional usage. I will not delve into details of raid levels. But you can find more about in wikipedia. In linux software raid consist of kernel module and user space programs.

3 disks are used actively to store data but 4. disk the or spare disk  is used to backup for 3 disks. It doesn’t store all data of the 3 disks as it is impossible if 3 disks are near to full.

First list disks. We will make array with vdb,vdc,vdd and vde disks. I make this in my vm so your disk names mabe different like sda, sdb etc. All disk have 1 GB.

$ ls /dev/vd* /dev/vda  /dev/vda1  /dev/vdb  /dev/vdc  /dev/vdd  /dev/vde

1 2	$ ls /dev/vd* /dev/vda  /dev/vda1  /dev/vdb  /dev/vdc  /dev/vdd  /dev/vde

Install mdadm in ubuntu with this command

$ apt-get install mdadm Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libpam-systemd postfix ssl-cert Suggested packages: procmail postfix-mysql postfix-pgsql postfix-ldap postfix-pcre sasl2-bin dovecot-common postfix-cdb mail-reader postfix-doc openssl-blacklist Recommended packages: default-mta mail-transport-agent The following NEW packages will be installed: mdadm postfix ssl-cert The following packages will be upgraded: libpam-systemd 1 upgraded, 3 newly installed, 0 to remove and 3 not upgraded. 13 not fully installed or removed. Need to get 1,464 kB/1,490 kB of archives. After this operation, 4,904 kB of additional disk space will be used. Do you want to continue? [Y/n] Y

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

$ apt-get install mdadm Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libpam-systemd postfix ssl-cert Suggested packages: procmail postfix-mysql postfix-pgsql postfix-ldap postfix-pcre sasl2-bin dovecot-common postfix-cdb mail-reader postfix-doc openssl-blacklist Recommended packages: default-mta mail-transport-agent The following NEW packages will be installed: mdadm postfix ssl-cert The following packages will be upgraded: libpam-systemd 1 upgraded, 3 newly installed, to remove and 3 not upgraded. 13 not fully installed or removed. Need to get 1,464 kB/1,490 kB of archives. After this operation, 4,904 kB of additional disk space will be used. Do you want to continue? [Y/n] Y

If not load mdadm kernel module

$ modprobe raid456

1	$ modprobe raid456

Here we create raid. We should give count of disk with raid-devices level with leve and the disk to be used in raid. Here meta data is optional it sets raid metada version After the command is executed we got a message that says our new disk which we named md0 is created . If you wab to create raid0 use level=stripe and raid1 level=mirror

$ mdadm --create /dev/md0 --metadata 1.2 --level=4 --raid-devices=4 /dev/vdb /dev/vdc /dev/vdd /dev/vde mdadm: array /dev/md0 started.

1 2	$ mdadm --create /dev/md0 --metadata 1.2 --level=4 --raid-devices=4 /dev/vdb /dev/vdc /dev/vdd /dev/vde mdadm: array /dev/md0 started.

Also we got a mail about our raid operation. As you can see in message the pyshical disk are nuımbered and possibile future raid levels are provided with Personalities row.

$ cat /var/mail/root From root@ubuntu  Mon Jul 21 13:48:15 2014 Return-Path: <root@ubuntu> X-Original-To: root Delivered-To: root@ubuntu Received: by ubuntu (Postfix, from userid 0) id 8E4CC208B9; Mon, 21 Jul 2014 13:48:15 +0300 (EEST) From: mdadm monitoring <root@ubuntu> To: root@ubuntu Subject: DegradedArray event on /dev/md0:ubuntu Message-Id: <20140721104815.8E4CC208B9@ubuntu> Date: Mon, 21 Jul 2014 13:48:15 +0300 (EEST) This is an automatically generated mail message from mdadm running on ubuntu A DegradedArray event had been detected on md device /dev/md0. Faithfully yours, etc. P.S. The /proc/mdstat file currently contains the following: Personalities : [raid6] [raid5] [raid4] md0 : active raid4 vde[4] vdd[2] vdc[1] vdb[0] 3142656 blocks super 1.2 level 4, 512k chunk, algorithm 0 [4/3] [UUU_] [>....................]  recovery =  0.0% (0/1047552) finish=1091.2min speed=0K/sec unused devices: <none>

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

$ cat /var/mail/root From root@ubuntu  Mon Jul 21 13:48:15 2014 Return-Path: <root@ubuntu> X-Original-To: root Delivered-To: root@ubuntu Received: by ubuntu (Postfix, from userid ) id 8E4CC208B9; Mon, 21 Jul 2014 13:48:15 +0300 (EEST) From: mdadm monitoring <root@ubuntu> To: root@ubuntu Subject: DegradedArray event on /dev/md0:ubuntu Message-Id: <20140721104815.8E4CC208B9@ubuntu> Date: Mon, 21 Jul 2014 13:48:15 +0300 (EEST)   This is an automatically generated mail message from mdadm running on ubuntu   A DegradedArray event had been detected on md device /dev/md0.   Faithfully yours, etc.   P.S. The /proc/mdstat file currently contains the following:   Personalities : [raid6] [raid5] [raid4] md0 : active raid4 vde[4] vdd[2] vdc[1] vdb[] 3142656 blocks super 1.2 level 4, 512k chunk, algorithm [4/3] [UUU_] [>....................]  recovery =  0.0% (/1047552) finish=1091.2min speed=0K/sec   unused devices: <none>

To see raid devices status.Shows total disk count and total useable size which is 3/4 of the total disk size. As you can see version is 1.2 where we set it while creating the array

$ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid4 vde[4] vdd[2] vdc[1] vdb[0] 3142656 blocks super 1.2 level 4, 512k chunk, algorithm 0 [4/4] [UUUU] unused devices: <none> $ mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Mon Jul 21 13:48:15 2014 Raid Level : raid4 Array Size : 3142656 (3.00 GiB 3.22 GB) Used Dev Size : 1047552 (1023.17 MiB 1072.69 MB) Raid Devices : 4 Total Devices : 4 Persistence : Superblock is persistent Update Time : Mon Jul 21 13:48:28 2014 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Chunk Size : 512K Name : ubuntu:0  (local to host ubuntu) UUID : 515af24c:86bb12f7:015dfebf:6bd9b35b Events : 18 Number   Major   Minor   RaidDevice State 0     253       16        0      active sync   /dev/vdb 1     253       32        1      active sync   /dev/vdc 2     253       48        2      active sync   /dev/vdd 4     253       64        3      active sync   /dev/vde

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

$ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid4 vde[4] vdd[2] vdc[1] vdb[] 3142656 blocks super 1.2 level 4, 512k chunk, algorithm [4/4] [UUUU]   unused devices: <none>   $ mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Mon Jul 21 13:48:15 2014 Raid Level : raid4 Array Size : 3142656 (3.00 GiB 3.22 GB) Used Dev Size : 1047552 (1023.17 MiB 1072.69 MB) Raid Devices : 4 Total Devices : 4 Persistence : Superblock is persistent   Update Time : Mon Jul 21 13:48:28 2014 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : Spare Devices :   Chunk Size : 512K   Name : ubuntu:  (local to host ubuntu) UUID : 515af24c:86bb12f7:015dfebf:6bd9b35b Events : 18   Number   Major   Minor   RaidDevice State      253       16              active sync   /dev/vdb 1     253       32        1      active sync   /dev/vdc 2     253       48        2      active sync   /dev/vdd 4     253       64        3      active sync   /dev/vde

You can save raid configuration in file.

$ mdadm --detail --scan >> /etc/mdadm/mdadm.conf $ cat /etc/mdadm/mdadm.conf # mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. # # by default (built-in), scan all partitions (/proc/partitions) and all # containers for MD superblocks. alternatively, specify devices to scan, using # wildcards if desired. #DEVICE partitions containers # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes # automatically tag new arrays as belonging to the local system HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts MAILADDR root # definitions of existing MD arrays # This file was auto-generated on Mon, 21 Jul 2014 13:43:42 +0300 # by mkconf $Id$ ARRAY /dev/md/ubuntu:0 metadata=1.2 name=ubuntu:0 UUID=515af24c:86bb12f7:015dfebf:6bd9b35b

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

$ mdadm --detail --scan >> /etc/mdadm/mdadm.conf $ cat /etc/mdadm/mdadm.conf # mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. #   # by default (built-in), scan all partitions (/proc/partitions) and all # containers for MD superblocks. alternatively, specify devices to scan, using # wildcards if desired. #DEVICE partitions containers   # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes   # automatically tag new arrays as belonging to the local system HOMEHOST <system>   # instruct the monitoring daemon where to send mail alerts MAILADDR root   # definitions of existing MD arrays   # This file was auto-generated on Mon, 21 Jul 2014 13:43:42 +0300 # by mkconf $Id$ ARRAY /dev/md/ubuntu: metadata=1.2 name=ubuntu: UUID=515af24c:86bb12f7:015dfebf:6bd9b35b

To see a disk status of array

$ mdadm -E /dev/vdd /dev/vdd: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 515af24c:86bb12f7:015dfebf:6bd9b35b Name : ubuntu:0  (local to host ubuntu) Creation Time : Mon Jul 21 13:48:15 2014 Raid Level : raid4 Raid Devices : 4 Avail Dev Size : 2096128 (1023.67 MiB 1073.22 MB) Array Size : 3142656 (3.00 GiB 3.22 GB) Used Dev Size : 2095104 (1023.17 MiB 1072.69 MB) Data Offset : 1024 sectors Super Offset : 8 sectors State : clean Device UUID : e5cc5bd9:decd247a:16c39662:abeb38bb Update Time : Mon Jul 21 14:07:48 2014 Checksum : c07c9d94 - correct Events : 18 Chunk Size : 512K Device Role : Active device 2 Array State : AAAA ('A' == active, '.' == missing)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

$ mdadm -E /dev/vdd /dev/vdd: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 515af24c:86bb12f7:015dfebf:6bd9b35b Name : ubuntu:  (local to host ubuntu) Creation Time : Mon Jul 21 13:48:15 2014 Raid Level : raid4 Raid Devices : 4   Avail Dev Size : 2096128 (1023.67 MiB 1073.22 MB) Array Size : 3142656 (3.00 GiB 3.22 GB) Used Dev Size : 2095104 (1023.17 MiB 1072.69 MB) Data Offset : 1024 sectors Super Offset : 8 sectors State : clean Device UUID : e5cc5bd9:decd247a:16c39662:abeb38bb   Update Time : Mon Jul 21 14:07:48 2014 Checksum : c07c9d94 - correct Events : 18   Chunk Size : 512K   Device Role : Active device 2 Array State : AAAA ('A' == active, '.' == missing)

Create file system for new disk

$ mkfs.ext4 /dev/md0 mke2fs 1.42.9 (4-Feb-2014) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=128 blocks, Stripe width=384 blocks 196608 inodes, 785664 blocks 39283 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=805306368 24 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

$ mkfs.ext4 /dev/md0 mke2fs 1.42.9 (4-Feb-2014) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=128 blocks, Stripe width=384 blocks 196608 inodes, 785664 blocks 39283 blocks (5.00%) reserved for the super user First data block= Maximum filesystem blocks=805306368 24 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912   Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done

Mount the disk and create some files. Also show status of mounted disks

$ mount /dev/md0 /mnt $ mkdir /mnt/ismail $ ls /mnt/ ismail  lost+found $ df -lh Filesystem      Size  Used Avail Use% Mounted on /dev/vda1        18G  1.7G   16G  10% / none            4.0K     0  4.0K   0% /sys/fs/cgroup udev            235M  4.0K  235M   1% /dev tmpfs            49M  268K   49M   1% /run none            5.0M     0  5.0M   0% /run/lock none            245M     0  245M   0% /run/shm none            100M     0  100M   0% /run/user /dev/md0        2.9G  4.6M  2.8G   1% /mnt

1 2 3 4 5 6 7 8 9 10 11 12 13 14

$ mount /dev/md0 /mnt $ mkdir /mnt/ismail $ ls /mnt/ ismail  lost+found $ df -lh Filesystem      Size  Used Avail Use% Mounted on /dev/vda1        18G  1.7G   16G  10% / none            4.0K       4.0K   % /sys/fs/cgroup udev            235M  4.0K  235M   1% /dev tmpfs            49M  268K   49M   1% /run none            5.0M       5.0M   % /run/lock none            245M       245M   % /run/shm none            100M       100M   % /run/user /dev/md0        2.9G  4.6M  2.8G   1% /mnt

Say disk vdd is corrupted. And we need to remove it.

$ mdadm --remove /dev/md0 /dev/vde

1	$ mdadm --remove /dev/md0 /dev/vde

or add some disks

$ mdadm --add /dev/md0 /dev/vdf

1	$ mdadm --add /dev/md0 /dev/vdf

To use raid after restart. Raid devies must reassembled every time to use.

<strong>mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 </strong>More easy way to reassembly

1 2 3 4

<strong>mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1     </strong>More easy way to reassembly

<strong>mdadm --assemble --scan</strong>

1	<strong>mdadm --assemble --scan</strong>

Resources:

https://raid.wiki.kernel.org/index.php/RAID_setup

Ubuntu 14.04 Server Guide