Expanding OS partition – LVM on top of MD RAID – without downtime
Replacing failing physical disks on the servers hosted at the data center could be just another routine task with no surprises. Sadly, it’s often a headache since you need to rely on DC support and there’s no way to tell if the person on the other end is newly hired and still learning, or maybe is distracted? or simply doesn’t care much? complications are pretty rare but severe. I’ve faced massive data loss as the DC support pulled the wrong drive (different server!) during a hot-swap attempt. Another day and another data center – failing drive’s LED are blinking as a sign to replace exactly this one and.. healthy disk from near slot gets pulled instead, mistakenly.
So unless you’re absolutely sure about the DC support – I recommend offline drive swaps/additions. It’s 5-15minutes of scheduled downtime outside business hours VS potential data loss.
In this example, I’m using Debian distribution installed on LVM on top of software RAID1 array, made of 2x12GB disks.
Two new spare disks (2x20GB) are installed on the server already as well (in case your server doesn’t have any more free slots to add new disks – the procedure will need to be slightly modified, more on that a bit later [1]).
Current situation:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
sda 8:0 0 12G 0 disk └─sda1 8:1 0 12G 0 part └─md0 9:0 0 12G 0 raid1 ├─VG_Debian-boot 253:0 0 380M 0 lvm /boot ├─VG_Debian-root 253:1 0 9.8G 0 lvm / └─VG_Debian-swap 253:2 0 1.9G 0 lvm [SWAP] sdb 8:16 0 12G 0 disk └─sdb1 8:17 0 12G 0 part └─md0 9:0 0 12G 0 raid1 ├─VG_Debian-boot 253:0 0 380M 0 lvm /boot ├─VG_Debian-root 253:1 0 9.8G 0 lvm / └─VG_Debian-swap 253:2 0 1.9G 0 lvm [SWAP] sdc 8:32 0 20G 0 disk sdd 8:48 0 20G 0 disk |
Objectives:
-
make extra space available for Debian;
-
ensure the server remains online – no outages allowed.
Getting started.
First is the RAID array, sda and sdb needs to be replaced with sdc and sdd.
1. Create partition on both new disks, no filesystem needed:
1 2 3 4 5 |
~# parted /dev/sdc mklabel gpt mkpart primary 2048s 100% ~# parted /dev/sdd mklabel gpt mkpart primary 2048s 100% ~# fdisk -l /dev/sdc|tail -1 /dev/sdc1 2048 41943039 41940992 20G 83 Linux |
2. Check array’s members:
1 2 3 |
~# cat /proc/mdstat md0 : active raid1 sdb1[1] sda1[0] 12571648 blocks super 1.2 [2/2] [UU] |
3. Remove the old disk:
1 2 3 4 |
~# mdadm --manage /dev/md0 --fail /dev/sdb1 mdadm: set /dev/sdb1 faulty in /dev/md0 ~# mdadm --manage /dev/md0 --remove /dev/sdb1 mdadm: hot removed /dev/sdb1 from /dev/md0 |
4. Add new one:
1 2 |
~# mdadm --manage /dev/md0 --add /dev/sdc1 mdadm: added /dev/sdc1 |
Take a look on the newly added array member, the Avail size should match the size of the partition created at the beginning:
1 2 3 4 |
~# mdadm --examine /dev/sdc1|grep Size Avail Dev Size : 41922560 (19.99 GiB 21.46 GB) Array Size : 12571648 (11.99 GiB 12.87 GB) Used Dev Size : 25143296 (11.99 GiB 12.87 GB) |
The array is now synchronizing, it might take long time, depending on your array’s size.
1 2 3 4 |
~# cat /proc/mdstat md0 : active raid1 sdc1[2] sdd1[3] 12571648 blocks super 1.2 [2/1] [U_] [>....................] recovery = 0.7% (94784/12571648) finish=2.1min speed=94784K/sec |
So let it finish, the array will be ready for resizing once synchronization is done.
5. Expanding the array:
1 2 |
~# mdadm --grow /dev/md0 --size=max mdadm: component size of /dev/md0 has been set to 20961280K |
6. Replace remaining old drive (sda) in the array with new one (sdd) – simply repeat step 3 & 4, adjusting the drive names.
The RAID array is now using all available space on the new disks:
1 2 3 4 5 |
~# mdadm --detail /dev/md0|egrep '(Size|active)' Array Size : 20961280 (19.99 GiB 21.46 GB) Used Dev Size : 20961280 (19.99 GiB 21.46 GB) 3 8 49 0 active sync /dev/sdd1 2 8 33 1 active sync /dev/sdc1 |
Done, there’s nothing more to do with the array except waiting for synchronization to finish.
Expanding LVM is now doable, lets check the current status:
1 2 3 4 5 |
~# df -h / ; pvs Filesystem Size Used Avail Use% Mounted on /dev/mapper/VG_Debian-root 9.6G 9.2G 276M 98% / PV VG Fmt Attr PSize PFree /dev/md0 VG_Debian lvm2 a-- <11.99g 0 |
7. Expand physical volume, volume group and logical volume(s):
1 2 3 4 5 6 7 8 9 10 11 |
~# pvresize /dev/md0 Physical volume "/dev/md0" changed 1 physical volume(s) resized or updated / 0 physical volume(s) not resized PV and and Volume Group are now both ready: ~# pvs ; vgs PV VG Fmt Attr PSize PFree /dev/md0 VG_Debian lvm2 a-- <19.99g 8.00g VG #PV #LV #SN Attr VSize VFree VG_Debian 1 3 0 wz--n- <19.99g 8.00g |
Logical Volumes can be now expanded:
1 2 3 4 5 |
~# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert boot VG_Debian -wi-ao---- 380.00m root VG_Debian -wi-ao---- 9.75g swap VG_Debian -wi-ao---- <1.87g |
I’m going to max out the ‘root’ LV but any LV within VG_Debian group can be expanded:
1 2 3 4 5 6 |
~# lvextend -l +100%FREE /dev/VG_Debian/root Size of logical volume VG_Debian/root changed from 9.75 GiB (2496 extents) to 17.75 GiB (4544 extents). Logical volume VG_Debian/root successfully resized. ~# lvs|grep root root VG_Debian -wi-ao---- 17.75g |
8. Final touch – grow the OS filesystem to use all the free space available on the LV:
1 2 3 4 5 |
~# resize2fs /dev/mapper/VG_Debian-root resize2fs 1.45.6 (20-Mar-2020) Filesystem at /dev/mapper/VG_Debian-root is mounted on /; on-line resizing required old_desc_blocks = 2, new_desc_blocks = 3 The filesystem on /dev/mapper/VG_Debian-root is now 4653056 (4k) blocks long. |
Verify the filesystem:
1 2 3 |
~# df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/VG_Debian-root 18G 9.2G 8.2G 53% / |
Objective completed.