= md (software RAID) and lvm (logical volume management) = [[TableOfContents()]] == md == Building a RAID array using mdadm - two primary steps: 1. "mdadm --create" to build the array using available resources 2. "mdadm --detail --scan" to build config string for /etc/mdadm/mdadm.conf Simple examples: === RAID6 array === * Set up partitions to be used (in this case the whole disk): {{{ # for x in /dev/sd{b,c,d,e,f}1 ; do fdisk $x ; done }}} * Create the array (in this case, with one hot-spare): {{{ # mdadm --create /dev/md0 --level=6 --raid-devices=4 --spare-devices=1 /dev/sd{b,c,d,e,f}1 }}} * Configure the array for reboot (append to the end of /etc/mdadm/mdadm.conf): {{{ # mdadm --detail --scan ARRAY /dev/md/0 metadata=1.2 spares=1 name=debian6-vm:0 UUID=9b42abcd:309fabcd:6bfbabcd:298dabcd }}} Considerations when setting up the partitions might be that any replacement disks will need to support that same size partition. Unconfirmed but it sounds like it might be a reasonable concern: "Enter a value smaller than the free space value minus 2% or the disk size to make sure that when you will later install a new disk in replacement of a failed one, you will have at least the same capacity even if the number of cylinders is different." (http://www.jerryweb.org/settings/raid/) === RAID1 array === Pretty much the same process - but for reference here's the simpler RAID1 config without any spare: {{{ # mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sd{g,h}1 }}} And here's how the config looks: {{{ ARRAY /dev/md1 metadata=1.2 name=debian6-vm:1 UUID=fa94abcd:317fabcd:3a61abcd:4932abcd }}} === /proc/mdstat === The current state of the RAID arrays: {{{ # cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid1 sdh1[1] sdg1[0] 10481285 blocks super 1.2 [2/2] [UU] md0 : active raid6 sdb1[0] sdf1[4](S) sde1[3] sdd1[2] sdc1[1] 16769024 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU] unused devices: }}} == lvm == Regardless of using RAID or plain disks, lvm allows much more flexibility in allocating that space. Not any use whatsoever on a smaller machine which is using a single large partition for everything (but even then, use of lvm could allow new disk to be added to extend a filesystem if space is running low). The steps: 1. "pvcreate" to initialise partitions to be used by lvm 2. "vgcreate" to create volume groups (these contain physical devices) 3. "lvcreate" to define logical volumes (these are then used as disk partitions) === pvcreate === Partitions can't be used until they're initialised: (this is run in testmode with -t to show what would happen) {{{ # pvcreate -t -v /dev/md0 Test mode: Metadata will NOT be updated. Test mode: Wiping internal cache Wiping internal VG cache }}} The physical volume (PV) will now be listed as available (along with those already allocated to a group). In this example all PVs are already allocated to groups: {{{ # pvs PV VG Fmt Attr PSize PFree /dev/md0 brad_group1 lvm2 a- 15.99g 9.35g /dev/md1 brad_group1 lvm2 a- 9.99g 9.02g }}} === vgcreate (and vgextend) === In order to use PVs they need to be allocated to a volume group (VG) within lvm. When a group is created it starts with a number of initialised PVs: {{{ # vgcreate brad_group1 /dev/md0 }}} Later on more PVs can be added to a VG with vgextend: {{{ # vgextend /dev/brad_group1 /dev/md1 }}} Something determined at this stage is the "physical extent size" (defaulting to 4MB) - this is the volume chunk size of all PVs within the group. Space is allocated to logical volumes in whole physical extent chunks. === lvcreate (and lvextend) === Now that we have PVs and VGs it's possible to create logical volumes (LVs) and use them. There's lots of options here, but here's a simple example which uses extents, though it's also possible to ask for a specific size: {{{ # lvcreate -l 250 /dev/brad_group1 }}} It's also possible to ask for a specific LV name (if not, one is auto-generated). This LV can now be formatted and mounted: {{{ # mkfs.xfs /dev/brad_group1/lvol0 # mount /dev/brad_group1/lvol0 /mnt/test }}} If the size of an LV needs to be changed the lvextend program can be used: {{{ # lvextend -l +1000 /dev/brad_group1/lvol0 }}} '''Note:''' that this only changes the size of the LV, not the filesystem. Various filesystems have different approaches (see http://tldp.org/HOWTO/LVM-HOWTO/extendlv.html), xfs happens to be quite easy for growth but it doesn't support shrinking: {{{ # lvextend -l +1000 /dev/brad_group1/lvol0 # xfs_growfs /mnt/test }}} There's some other stuff going on here as well, for instance it's possible to set up a LV to stripe across PVs in the VG. This cane be seen using lvdisplay: {{{ # lvcreate --extents 500 --stripes 2 /dev/brad_group1 # lvdisplay -m /dev/brad_group1/lvol2 --- Logical volume --- LV Name /dev/brad_group1/lvol2 VG Name brad_group1 LV UUID 20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh LV Write Access read/write LV Status available # open 1 LV Size 1.95 GiB Current LE 500 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 512 Block device 253:2 --- Segments --- Logical extent 0 to 499: Type striped Stripes 2 Stripe size 64.00 KiB Stripe 0: Physical volume /dev/md0 Physical extents 1450 to 1699 Stripe 1: Physical volume /dev/md1 Physical extents 0 to 249 }}} === displaying information === Various commands show info about the lvm configuration, examples follow. ==== pvs ==== {{{{ # pvs PV VG Fmt Attr PSize PFree /dev/md0 brad_group1 lvm2 a- 15.99g 9.35g /dev/md1 brad_group1 lvm2 a- 9.99g 9.02g }}}} ==== pvdisplay ==== {{{{ # pvdisplay --- Physical volume --- PV Name /dev/md0 VG Name brad_group1 PV Size 15.99 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 4093 Free PE 2393 Allocated PE 1700 PV UUID OFyENQ-lRD8-jPoW-aEr4-MY8j-WMqN-pZYy2H --- Physical volume --- PV Name /dev/md1 VG Name brad_group1 PV Size 10.00 GiB / not usable 3.63 MiB Allocatable yes PE Size 4.00 MiB Total PE 2558 Free PE 2308 Allocated PE 250 PV UUID 1WsYxg-YnSN-VKme-E0iw-gX6Y-KEcI-VUckAb }}}} ==== vgs ==== {{{{ # vgs VG #PV #LV #SN Attr VSize VFree brad_group1 2 3 0 wz--n- 25.98g 18.36g }}}} ==== vgdisplay ===== {{{{ # vgdisplay --- Volume group --- VG Name brad_group1 System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 7 VG Access read/write VG Status resizable MAX LV 0 Cur LV 3 Open LV 3 Max PV 0 Cur PV 2 Act PV 2 VG Size 25.98 GiB PE Size 4.00 MiB Total PE 6651 Alloc PE / Size 1950 / 7.62 GiB Free PE / Size 4701 / 18.36 GiB VG UUID BA5Vkh-SrmB-JOr6-vAz0-48Eh-YI6o-m3Ghvr }}}} ==== lvs ==== {{{{ # lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert lvol0 brad_group1 -wi-ao 4.69g lvol1 brad_group1 -wi-ao 1000.00m lvol2 brad_group1 -wi-ao 1.95g }}}} ==== lvdisplay ==== {{{{ # lvdisplay --- Logical volume --- LV Name /dev/brad_group1/lvol0 VG Name brad_group1 LV UUID 06M0bJ-RaDR-fVNz-weRo-pVAz-wTMl-17xr5G LV Write Access read/write LV Status available # open 1 LV Size 4.69 GiB Current LE 1200 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 4096 Block device 253:0 --- Logical volume --- LV Name /dev/brad_group1/lvol1 VG Name brad_group1 LV UUID b0a2x6-Qxkl-sAX1-uz2y-KVFK-qiJu-IBodH7 LV Write Access read/write LV Status available # open 1 LV Size 1000.00 MiB Current LE 250 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 4096 Block device 253:1 --- Logical volume --- LV Name /dev/brad_group1/lvol2 VG Name brad_group1 LV UUID 20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh LV Write Access read/write LV Status available # open 1 LV Size 1.95 GiB Current LE 500 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 512 Block device 253:2 }}}}