KVM guests – compressed storages – enabling TRIM support on virtio disks

Trim is commonly used as a way to notify SSD drive that some part of the data it holds is garbage and can be discarded, it extends SSD lifetime and releases otherwise reserved space on the disk. Turns out it’s not the only scenario where trim comes handy. Compression is a popular way to save on space usage when deploying virtual machines and it seems to be a good option especially that the performance penalty is marginal.

But there’s one compression-specific issue some admins wasn’t aware of or misunderstood. It might cause serious problems impacting all KVM guests on such storage if it’s not addressed. I will describe it in the example scenario below, note that it’s not ZFS-specific issue.

Let’s say we:

• defined ZFS pool ‘t1’:

• created some sparse (thin provisioned) datasets in that pool, one per KVM guest:

• created KVM guest which will use compressed dataset we just made:

 

Initial space usage on the pool and inside the KVM guest:

 

Now amount of data (let it be 10GB of random data) deleted inside KVM guest will be the amount of free space immediately available back for that KVM guest.

Outside the guest the dataset still shows either  the same space usage as it was before deletion – or released amount is so much smaller than the size of the data deleted:

this is because the data wasn’t really deleted but only marked as available for overwriting whenever the OS needs to write data again. As a result, there is now 10GB of garbage data on the VM’s storage, from VM’s OS perspective there is no problem as it perceives marked garbage as 10GB of available space  –  however outside the VM it’s a problem – the ZFS pool with compressed dataset have no idea it now holds 10GB of garbage because from the outside of KVM guest there is no way to tell which data is garbage and which isn’t. It’s 10GB less space available on the ZFS pool, affecting remaining KVM guests.

That’s where TRIM would come handy.

Bummer. In many cases the virtual controller defaults to virtio-blk (for example – current version of Softaculous’s Virtualizor) which doesn’t support discard feature (actually virtio-blk supports discard since kernel 5.0 but that version might be not an option for many reasons) so the workaround needs to be used.

In order to release the 10GB of garbage data from ZFS dataset without TRIM, the Guest’s OS would need to overwrite garbage with zeros (or any other stream with high compression ratio), so then the ZFS’s dataset can start compressing 10GB of data made just from a single character, therefore reducing it to – wild guess – let’s say few bytes/kilobytes:

After that the 10GB on the dataset has been reclaimed:

However TRIM is the preferred way.

For the TRIM to work inside KVM guest, the virtual disk must support ‘discard’ feature, here’s how:

1. Create a new file named new-virtio-scsi-ctl.xml with SCSI controller definition, SCSI drives will be attached to it, add the content below and save the file:

2. Define new SCSI drive – create new file new-virtio-scsi-drive.xml, fill it as below, note that the ‘controller’ number must match the one we just defined above:

3. Import both controller and new disk definition, to our example KVM guest named C6-guest-13:

The controller:

Inside the KVM guest, dmesg will show something like:

The drive:

And the guest’s dmesg will show:

Lets quickly test the drive for fstrim support, execute as below inside KVM guest (bc and lsscsi needed):

The output should look as below:

That’s it.

In order to convert existing ‘virtio’ storage to ‘virtio-scsi’ just add SCSI controller and adjust existing storage definition to match the new controller as an example above.