How To Raid In Linux With Mdadm?

Posted on 12/11/2016 by İsmail Baydan

https://www.poftut.com/raid-linux-mdadm/

Hi, today I am gonna show you how to make raid ( redundant array of inexpensive disks ). Why wee need it? Think about that you have customer data stored in a server. Say one day disk is dead and then all of your data about customer are gone. This is a very bad new. What if disk dies but the data remains the same.

This will be very good. You can think that will disk dies frequently. No but it is not impossible, especially in enterprise side or in personal use in home. RAID is a technology to backup disks with extra disks. Say we have 3 disks which are used separate projects. And we want to backup them with high availibility in particular working hours. RAID can be done with separate hardware, firmware or software. Making raid with hardware is the best, fast and reliable way but it may be expensive or too much for project. Firmware raid is generally provided by motherboard manufacturers as a cheap option for hardware raid. Software raid is the cheapest and least reliable way to mail raid. But it can be suitable for home or unprofessional usage. I will not delve into details of raid levels. But you can find more about in wikipedia. In linux software raid consist of kernel module and user space programs.

3 disks are used actively to store data but 4. disk the or spare disk is used to backup for 3 disks. It doesn’t store all data of the 3 disks as it is impossible if 3 disks are near to full.

First list disks. We will make array with vdb,vdc,vdd and vde disks. I make this in my vm so your disk names mabe different like sda, sdb etc. All disk have 1 GB.

$ ls /dev/vd* /dev/vda /dev/vda1 /dev/vdb /dev/vdc /dev/vdd /dev/vde
1
2
$ ls /dev/vd*
/dev/vda /dev/vda1 /dev/vdb /dev/vdc /dev/vdd /dev/vde

Install mdadm in ubuntu with this command

$ apt-get install mdadm Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libpam-systemd postfix ssl-cert Suggested packages: procmail postfix-mysql postfix-pgsql postfix-ldap postfix-pcre sasl2-bin dovecot-common postfix-cdb mail-reader postfix-doc openssl-blacklist Recommended packages: default-mta mail-transport-agent The following NEW packages will be installed: mdadm postfix ssl-cert The following packages will be upgraded: libpam-systemd 1 upgraded, 3 newly installed, 0 to remove and 3 not upgraded. 13 not fully installed or removed. Need to get 1,464 kB/1,490 kB of archives. After this operation, 4,904 kB of additional disk space will be used. Do you want to continue? [Y/n] Y
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ apt-get install mdadm
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libpam-systemd postfix ssl-cert
Suggested packages:
procmail postfix-mysql postfix-pgsql postfix-ldap postfix-pcre sasl2-bin
dovecot-common postfix-cdb mail-reader postfix-doc openssl-blacklist
Recommended packages:
default-mta mail-transport-agent
The following NEW packages will be installed:
mdadm postfix ssl-cert
The following packages will be upgraded:
libpam-systemd
1 upgraded, 3 newly installed, to remove and 3 not upgraded.
13 not fully installed or removed.
Need to get 1,464 kB/1,490 kB of archives.
After this operation, 4,904 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y

If not load mdadm kernel module

$ modprobe raid456
1
$ modprobe raid456

Here we create raid. We should give count of disk with raid-devices level with leve and the disk to be used in raid. Here meta data is optional it sets raid metada version After the command is executed we got a message that says our new disk which we named md0 is created . If you wab to create raid0 use level=stripe and raid1 level=mirror

$ mdadm --create /dev/md0 --metadata 1.2 --level=4 --raid-devices=4 /dev/vdb /dev/vdc /dev/vdd /dev/vde mdadm: array /dev/md0 started.
1
2
$ mdadm --create /dev/md0 --metadata 1.2 --level=4 --raid-devices=4 /dev/vdb /dev/vdc /dev/vdd /dev/vde
mdadm: array /dev/md0 started.

Also we got a mail about our raid operation. As you can see in message the pyshical disk are nuımbered and possibile future raid levels are provided with Personalities row.

$ cat /var/mail/root From root@ubuntu Mon Jul 21 13:48:15 2014 Return-Path: <root@ubuntu> X-Original-To: root Delivered-To: root@ubuntu Received: by ubuntu (Postfix, from userid 0) id 8E4CC208B9; Mon, 21 Jul 2014 13:48:15 +0300 (EEST) From: mdadm monitoring <root@ubuntu> To: root@ubuntu Subject: DegradedArray event on /dev/md0:ubuntu Message-Id: <20140721104815.8E4CC208B9@ubuntu> Date: Mon, 21 Jul 2014 13:48:15 +0300 (EEST) This is an automatically generated mail message from mdadm running on ubuntu A DegradedArray event had been detected on md device /dev/md0. Faithfully yours, etc. P.S. The /proc/mdstat file currently contains the following: Personalities : [raid6] [raid5] [raid4] md0 : active raid4 vde[4] vdd[2] vdc[1] vdb[0] 3142656 blocks super 1.2 level 4, 512k chunk, algorithm 0 [4/3] [UUU_] [>....................] recovery = 0.0% (0/1047552) finish=1091.2min speed=0K/sec unused devices: <none>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
$ cat /var/mail/root
From root@ubuntu Mon Jul 21 13:48:15 2014
Return-Path: <root@ubuntu>
X-Original-To: root
Delivered-To: root@ubuntu
Received: by ubuntu (Postfix, from userid )
id 8E4CC208B9; Mon, 21 Jul 2014 13:48:15 +0300 (EEST)
From: mdadm monitoring <root@ubuntu>
To: root@ubuntu
Subject: DegradedArray event on /dev/md0:ubuntu
Message-Id: <20140721104815.8E4CC208B9@ubuntu>
Date: Mon, 21 Jul 2014 13:48:15 +0300 (EEST)

This is an automatically generated mail message from mdadm
running on ubuntu

A DegradedArray event had been detected on md device /dev/md0.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid4 vde[4] vdd[2] vdc[1] vdb[]
3142656 blocks super 1.2 level 4, 512k chunk, algorithm [4/3] [UUU_]
[>....................] recovery = 0.0% (/1047552) finish=1091.2min speed=0K/sec

unused devices: <none>

To see raid devices status.Shows total disk count and total useable size which is 3/4 of the total disk size. As you can see version is 1.2 where we set it while creating the array

$ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid4 vde[4] vdd[2] vdc[1] vdb[0] 3142656 blocks super 1.2 level 4, 512k chunk, algorithm 0 [4/4] [UUUU] unused devices: <none> $ mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Mon Jul 21 13:48:15 2014 Raid Level : raid4 Array Size : 3142656 (3.00 GiB 3.22 GB) Used Dev Size : 1047552 (1023.17 MiB 1072.69 MB) Raid Devices : 4 Total Devices : 4 Persistence : Superblock is persistent Update Time : Mon Jul 21 13:48:28 2014 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Chunk Size : 512K Name : ubuntu:0 (local to host ubuntu) UUID : 515af24c:86bb12f7:015dfebf:6bd9b35b Events : 18 Number Major Minor RaidDevice State 0 253 16 0 active sync /dev/vdb 1 253 32 1 active sync /dev/vdc 2 253 48 2 active sync /dev/vdd 4 253 64 3 active sync /dev/vde
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid4 vde[4] vdd[2] vdc[1] vdb[]
3142656 blocks super 1.2 level 4, 512k chunk, algorithm [4/4] [UUUU]

unused devices: <none>

$ mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon Jul 21 13:48:15 2014
Raid Level : raid4
Array Size : 3142656 (3.00 GiB 3.22 GB)
Used Dev Size : 1047552 (1023.17 MiB 1072.69 MB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jul 21 13:48:28 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices :
Spare Devices :

Chunk Size : 512K

Name : ubuntu: (local to host ubuntu)
UUID : 515af24c:86bb12f7:015dfebf:6bd9b35b
Events : 18

Number Major Minor RaidDevice State
 253 16 active sync /dev/vdb
1 253 32 1 active sync /dev/vdc
2 253 48 2 active sync /dev/vdd
4 253 64 3 active sync /dev/vde

You can save raid configuration in file.

$ mdadm --detail --scan >> /etc/mdadm/mdadm.conf $ cat /etc/mdadm/mdadm.conf # mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. # # by default (built-in), scan all partitions (/proc/partitions) and all # containers for MD superblocks. alternatively, specify devices to scan, using # wildcards if desired. #DEVICE partitions containers # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes # automatically tag new arrays as belonging to the local system HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts MAILADDR root # definitions of existing MD arrays # This file was auto-generated on Mon, 21 Jul 2014 13:43:42 +0300 # by mkconf $Id$ ARRAY /dev/md/ubuntu:0 metadata=1.2 name=ubuntu:0 UUID=515af24c:86bb12f7:015dfebf:6bd9b35b
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ mdadm --detail --scan >> /etc/mdadm/mdadm.conf
$ cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Mon, 21 Jul 2014 13:43:42 +0300
# by mkconf $Id$
ARRAY /dev/md/ubuntu: metadata=1.2 name=ubuntu: UUID=515af24c:86bb12f7:015dfebf:6bd9b35b

To see a disk status of array

$ mdadm -E /dev/vdd /dev/vdd: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 515af24c:86bb12f7:015dfebf:6bd9b35b Name : ubuntu:0 (local to host ubuntu) Creation Time : Mon Jul 21 13:48:15 2014 Raid Level : raid4 Raid Devices : 4 Avail Dev Size : 2096128 (1023.67 MiB 1073.22 MB) Array Size : 3142656 (3.00 GiB 3.22 GB) Used Dev Size : 2095104 (1023.17 MiB 1072.69 MB) Data Offset : 1024 sectors Super Offset : 8 sectors State : clean Device UUID : e5cc5bd9:decd247a:16c39662:abeb38bb Update Time : Mon Jul 21 14:07:48 2014 Checksum : c07c9d94 - correct Events : 18 Chunk Size : 512K Device Role : Active device 2 Array State : AAAA ('A' == active, '.' == missing)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ mdadm -E /dev/vdd
/dev/vdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 515af24c:86bb12f7:015dfebf:6bd9b35b
Name : ubuntu: (local to host ubuntu)
Creation Time : Mon Jul 21 13:48:15 2014
Raid Level : raid4
Raid Devices : 4

Avail Dev Size : 2096128 (1023.67 MiB 1073.22 MB)
Array Size : 3142656 (3.00 GiB 3.22 GB)
Used Dev Size : 2095104 (1023.17 MiB 1072.69 MB)
Data Offset : 1024 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e5cc5bd9:decd247a:16c39662:abeb38bb

Update Time : Mon Jul 21 14:07:48 2014
Checksum : c07c9d94 - correct
Events : 18

Chunk Size : 512K

Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing)

Create file system for new disk

$ mkfs.ext4 /dev/md0 mke2fs 1.42.9 (4-Feb-2014) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=128 blocks, Stripe width=384 blocks 196608 inodes, 785664 blocks 39283 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=805306368 24 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ mkfs.ext4 /dev/md0
mke2fs 1.42.9 (4-Feb-2014)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=384 blocks
196608 inodes, 785664 blocks
39283 blocks (5.00%) reserved for the super user
First data block=
Maximum filesystem blocks=805306368
24 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

Mount the disk and create some files. Also show status of mounted disks

$ mount /dev/md0 /mnt $ mkdir /mnt/ismail $ ls /mnt/ ismail lost+found $ df -lh Filesystem      Size Used Avail Use% Mounted on /dev/vda1        18G 1.7G   16G 10% / none            4.0K     0 4.0K   0% /sys/fs/cgroup udev            235M 4.0K 235M   1% /dev tmpfs            49M 268K   49M   1% /run none            5.0M     0 5.0M   0% /run/lock none            245M     0 245M   0% /run/shm none            100M     0 100M   0% /run/user /dev/md0        2.9G 4.6M 2.8G   1% /mnt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ mount /dev/md0 /mnt
$ mkdir /mnt/ismail
$ ls /mnt/
ismail lost+found
$ df -lh
Filesystem      Size Used Avail Use% Mounted on
/dev/vda1        18G 1.7G   16G 10% /
none            4.0K     4.0K   % /sys/fs/cgroup
udev            235M 4.0K 235M   1% /dev
tmpfs            49M 268K   49M   1% /run
none            5.0M     5.0M   % /run/lock
none            245M     245M   % /run/shm
none            100M     100M   % /run/user
/dev/md0        2.9G 4.6M 2.8G   1% /mnt

Say disk vdd is corrupted. And we need to remove it.

$ mdadm --remove /dev/md0 /dev/vde
1
$ mdadm --remove /dev/md0 /dev/vde

or add some disks

$ mdadm --add /dev/md0 /dev/vdf
1
$ mdadm --add /dev/md0 /dev/vdf

To use raid after restart. Raid devies must reassembled every time to use.

mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 More easy way to reassembly
1
2
3
4
mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

More easy way to reassembly
mdadm --assemble --scan
1
mdadm --assemble --scan

Resources:

https://raid.wiki.kernel.org/index.php/RAID_setup

Ubuntu 14.04 Server Guide

https://www.poftut.com/raid-linux-mdadm/

LEARN MORE Linux Mint

How To Raid In Linux With Mdadm?

One thought on “How To Raid In Linux With Mdadm?”

Leave a Reply Cancel reply

Enjoy this blog? Please spread the word :)