Expanding OS partition – LVM on top of MD RAID – without downtime

 

Replacing failing physical disks on the servers hosted at the data center could be just another routine task with no surprises.  Sadly, it’s often a headache since you need to rely on DC support and there’s no way to tell if the person on the other end is newly hired and still learning, or maybe is distracted? or simply doesn’t care much? complications are pretty rare but severe. I’ve faced massive data loss as the DC support pulled the wrong drive (different server!) during a hot-swap attempt. Another day and another data center – failing drive’s LED are blinking as a sign to replace exactly this one and.. healthy disk from near slot gets pulled instead, mistakenly.

So unless you’re absolutely sure about the DC support – I recommend offline drive swaps/additions. It’s 5-15minutes of scheduled downtime outside business hours VS potential data loss.

In this example, I’m using Debian distribution installed on LVM on top of software RAID1 array, made of 2x12GB disks.

Two new spare disks (2x20GB) are installed on the server already as well (in case your server doesn’t have any more free slots to add new disks – the procedure will need to be slightly modified, more on that a bit later [1]).

Current situation:

 

Objectives:

• make extra space available for Debian;

• ensure the server remains online – no outages allowed.

Getting started.

First is the RAID array, sda and sdb needs to be replaced with sdc and sdd.

1. Create partition on both new disks, no filesystem needed:

2. Check array’s members:

3. Remove the old disk:

4. Add new one:

Take a look on the newly added array member, the Avail size should match the size of the partition created at the beginning:

The array is now synchronizing, it might take long time, depending on your array’s size.

So let it finish, the array will be ready for resizing once synchronization is done.

5. Expanding the array:

6. Replace remaining old drive (sda) in the array with new one (sdd) – simply repeat step 3 & 4, adjusting the drive names.

The RAID array is now using all available space on the new disks:

Done, there’s nothing more to do with the array except waiting for synchronization to finish.

Expanding LVM is now doable, lets check the current status:

7. Expand physical volume, volume group and logical volume(s):

Logical Volumes can be now expanded:

I’m going to max out the ‘root’ LV but any LV within VG_Debian group can be expanded:

8. Final touch – grow the OS filesystem to use all the free space available on the LV:

Verify the filesystem:

Objective completed.