md (software RAID) and lvm (logical volume management)
Contents
md
Building a RAID array using mdadm - two primary steps:
- "mdadm --create" to build the array using available resources
- "mdadm --detail --scan" to build config string for /etc/mdadm/mdadm.conf
Simple examples:
RAID6 array
Set up partitions to be used (in this case the whole disk):
# for x in /dev/sd{b,c,d,e,f}1 ; do fdisk $x ; doneCreate the array (in this case, with one hot-spare):
# mdadm --create /dev/md0 --level=6 --raid-devices=4 --spare-devices=1 /dev/sd{b,c,d,e,f}1Configure the array for reboot (append to the end of /etc/mdadm/mdadm.conf):
# mdadm --detail --scan ARRAY /dev/md/0 metadata=1.2 spares=1 name=debian6-vm:0 UUID=9b42abcd:309fabcd:6bfbabcd:298dabcd
Considerations when setting up the partitions might be that any replacement disks will need to support that same size partition. Unconfirmed but it sounds like it might be a reasonable concern: "Enter a value smaller than the free space value minus 2% or the disk size to make sure that when you will later install a new disk in replacement of a failed one, you will have at least the same capacity even if the number of cylinders is different." (http://www.jerryweb.org/settings/raid/)
RAID1 array
Pretty much the same process - but for reference here's the simpler RAID1 config without any spare:
# mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sd{g,h}1
And here's how the config looks:
ARRAY /dev/md1 metadata=1.2 name=debian6-vm:1 UUID=fa94abcd:317fabcd:3a61abcd:4932abcd
/proc/mdstat
The current state of the RAID arrays:
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdh1[1] sdg1[0]
10481285 blocks super 1.2 [2/2] [UU]
md0 : active raid6 sdb1[0] sdf1[4](S) sde1[3] sdd1[2] sdc1[1]
16769024 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
unused devices: <none>
replacing a failed disk
Some handy notes here:
http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array
http://consultancy.edvoncken.net/index.php/HOWTO_Replace_a_failing_disk_on_Linux_Software_RAID-5
lvm
Regardless of using RAID or plain disks, lvm allows much more flexibility in allocating that space. Not any use whatsoever on a smaller machine which is using a single large partition for everything (but even then, use of lvm could allow new disk to be added to extend a filesystem if space is running low).
For some very details documentation: http://tldp.org/HOWTO/LVM-HOWTO/
The steps:
- "pvcreate" to initialise partitions to be used by lvm
- "vgcreate" to create volume groups (these contain physical devices)
- "lvcreate" to define logical volumes (these are then used as disk partitions)
pvcreate
Partitions can't be used until they're initialised:
(this is run in testmode with -t to show what would happen)
# pvcreate -t -v /dev/md0
Test mode: Metadata will NOT be updated.
Test mode: Wiping internal cache
Wiping internal VG cache
The physical volume (PV) will now be listed as available (along with those already allocated to a group). In this example all PVs are already allocated to groups:
# pvs PV VG Fmt Attr PSize PFree /dev/md0 brad_group1 lvm2 a- 15.99g 9.35g /dev/md1 brad_group1 lvm2 a- 9.99g 9.02g
vgcreate (and vgextend)
In order to use PVs they need to be allocated to a volume group (VG) within lvm.
When a group is created it starts with a number of initialised PVs:
# vgcreate brad_group1 /dev/md0
Later on more PVs can be added to a VG with vgextend:
# vgextend /dev/brad_group1 /dev/md1
Something determined at this stage is the "physical extent size" (defaulting to 4MB) - this is the volume chunk size of all PVs within the group. Space is allocated to logical volumes in whole physical extent chunks.
lvcreate (and lvextend)
Now that we have PVs and VGs it's possible to create logical volumes (LVs) and use them.
There's lots of options here, but here's a simple example which uses extents, though it's also possible to ask for a specific size:
# lvcreate -l 250 /dev/brad_group1
It's also possible to ask for a specific LV name (if not, one is auto-generated).
This LV can now be formatted and mounted:
# mkfs.xfs /dev/brad_group1/lvol0 # mount /dev/brad_group1/lvol0 /mnt/test
If the size of an LV needs to be changed the lvextend program can be used:
# lvextend -l +1000 /dev/brad_group1/lvol0
Note: that this only changes the size of the LV, not the filesystem. Various filesystems have different approaches (see http://tldp.org/HOWTO/LVM-HOWTO/extendlv.html), xfs happens to be quite easy for growth but it doesn't support shrinking:
# lvextend -l +1000 /dev/brad_group1/lvol0 # xfs_growfs /mnt/test
There's some other stuff going on here as well, for instance it's possible to set up a LV to stripe across PVs in the VG. This cane be seen using lvdisplay:
# lvcreate --extents 500 --stripes 2 /dev/brad_group1
# lvdisplay -m /dev/brad_group1/lvol2
--- Logical volume ---
LV Name /dev/brad_group1/lvol2
VG Name brad_group1
LV UUID 20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh
LV Write Access read/write
LV Status available
# open 1
LV Size 1.95 GiB
Current LE 500
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 512
Block device 253:2
--- Segments ---
Logical extent 0 to 499:
Type striped
Stripes 2
Stripe size 64.00 KiB
Stripe 0:
Physical volume /dev/md0
Physical extents 1450 to 1699
Stripe 1:
Physical volume /dev/md1
Physical extents 0 to 249
displaying information
Various commands show info about the lvm configuration, examples follow.
pvs
# pvs PV VG Fmt Attr PSize PFree /dev/md0 brad_group1 lvm2 a- 15.99g 9.35g /dev/md1 brad_group1 lvm2 a- 9.99g 9.02g
pvdisplay
# pvdisplay --- Physical volume --- PV Name /dev/md0 VG Name brad_group1 PV Size 15.99 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 4093 Free PE 2393 Allocated PE 1700 PV UUID OFyENQ-lRD8-jPoW-aEr4-MY8j-WMqN-pZYy2H --- Physical volume --- PV Name /dev/md1 VG Name brad_group1 PV Size 10.00 GiB / not usable 3.63 MiB Allocatable yes PE Size 4.00 MiB Total PE 2558 Free PE 2308 Allocated PE 250 PV UUID 1WsYxg-YnSN-VKme-E0iw-gX6Y-KEcI-VUckAb
vgs
# vgs VG #PV #LV #SN Attr VSize VFree brad_group1 2 3 0 wz--n- 25.98g 18.36g
vgdisplay
# vgdisplay --- Volume group --- VG Name brad_group1 System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 7 VG Access read/write VG Status resizable MAX LV 0 Cur LV 3 Open LV 3 Max PV 0 Cur PV 2 Act PV 2 VG Size 25.98 GiB PE Size 4.00 MiB Total PE 6651 Alloc PE / Size 1950 / 7.62 GiB Free PE / Size 4701 / 18.36 GiB VG UUID BA5Vkh-SrmB-JOr6-vAz0-48Eh-YI6o-m3Ghvr
lvs
# lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert lvol0 brad_group1 -wi-ao 4.69g lvol1 brad_group1 -wi-ao 1000.00m lvol2 brad_group1 -wi-ao 1.95g
lvdisplay
# lvdisplay --- Logical volume --- LV Name /dev/brad_group1/lvol0 VG Name brad_group1 LV UUID 06M0bJ-RaDR-fVNz-weRo-pVAz-wTMl-17xr5G LV Write Access read/write LV Status available # open 1 LV Size 4.69 GiB Current LE 1200 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 4096 Block device 253:0 --- Logical volume --- LV Name /dev/brad_group1/lvol1 VG Name brad_group1 LV UUID b0a2x6-Qxkl-sAX1-uz2y-KVFK-qiJu-IBodH7 LV Write Access read/write LV Status available # open 1 LV Size 1000.00 MiB Current LE 250 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 4096 Block device 253:1 --- Logical volume --- LV Name /dev/brad_group1/lvol2 VG Name brad_group1 LV UUID 20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh LV Write Access read/write LV Status available # open 1 LV Size 1.95 GiB Current LE 500 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 512 Block device 253:2