Moving KVM guests to a new host and: How to shrink a KVM image

This was written in February 2021 at about the time Debian 10.8 was released.

As I had ordered a new Hetzner server, it was necessary to move three KVM guests running on the old server to the new host. The old host also served as a mail server and for simplicity, I had decided to use https://github.com/docker-mailserver/docker-mailserver for the new server. This is an important detail as Docker modifies your iptables rules, leading to your KVM guests losing internet access. More on that later.

Introduction

Let’s assume that our new server’s IP address is 1.2.3.4 with a gateway of 5.6.7.8. For the virtual machines, I had ordered a subnet with 6 additional addresses, let’s assume they are 9.10.11.12/29. All machines (host and guests) are running Debian, version 10.8 at the point this was written.

Setting up networking

Based on Hetzner’s own documentation, I had to route the guest networks through the host. A direct bridge doesn’t work as they don’t provide MAC addresses for each address in the IPv4 block. The configuration ended up as follows (doing this for IPv6 is left as an exercise to you):

Host

/etc/network/interfaces

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto enp5s0
iface enp5s0 inet static
  address 1.2.3.4
  netmask 255.255.255.255
  pointopoint 5.6.7.8
  gateway 5.6.7.8

/etc/network/interfaces.d/virbr1

auto virbr1
iface virbr1 inet static
   address 1.2.3.4
   netmask 255.255.255.255
   # putting a gateway/pointopoint entry here leads to the server not being bootable anymore.
   bridge_ports none
   bridge_stp off
   bridge_fd 0
   pre-up brctl addbr virbr1
   post-down brctl delbr virbr1
   up ip route add 9.10.11.12/29 dev virbr1 scope host
   down ip route del 9.10.11.12/29 dev virbr1 scope host

Guest

Our guest VM has the IP address 9.10.11.13.

/etc/network/interfaces

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet static
  address 9.10.11.13
  netmask 255.255.255.255
  gateway 1.2.3.4
  pointopoint 1.2.3.4

If you’re using your own image, make sure to configure the name servers in /etc/resolv.conf. (Hetzner provides this in their install images, you can copy the file from the host.)

Problems with Docker

At this point, I spent hours trying to find out why my guests still didn’t have internet access. I finally realized it was because Docker modifies iptables, setting FORWARD to DROP (you can google this), resulting in no traffic forwarded to/from the virtual machines anymore. There are instructions on the web on how to turn Docker’s modifications off and recreating the rules from scratch. But it’s too likely I was going to make mistakes as these rules are quite complex.

Modifying /etc/iptables/rules.v4 also didn’t help as those rules were simply overwritten. Almost all advice I found (including Docker’s own documentation) simply listed a command like this: iptables -I DOCKER-USER -o virbr1 -i virbr1 -j ACCEPT. It (sort of) works but it’s not permanent. Your VMs will have no internet again after a reboot of the host.

I finally settled on a cron script which runs once a minute:

/usr/sbin/iptables -D DOCKER-USER -o virbr1 -j ACCEPT
/usr/sbin/iptables -D DOCKER-USER -i virbr1 -j ACCEPT
/usr/sbin/iptables -I DOCKER-USER -o virbr1 -j ACCEPT
/usr/sbin/iptables -I DOCKER-USER -i virbr1 -j ACCEPT

It’s important to delete the rules first so they don’t stack up over time. (The first time you try to delete them, there’s an error message because they don’t exist yet, but that can be /dev/null-ed.)

Transferring the KVM guests to the new host

We had three VMs on the old host and the transfer method was different for each one.

Setting up a new KVM guest and copying the data over

This was fairly straightforward. How to install a new KVM guest is something that is thoroughly documented. I wanted to start with a minimal Debian installation so I opted for a minimal CD-ROM image which I downloaded into the /vm directory. I installed it with a HD size of 200GB, 2 CPUs, and 16GB of RAM:

virt-install -n firstguest --vcpus 2 -r 16384 --network=bridge=virbr1,model=virtio --location /vm/debian-10.7.0-amd64-netinst.iso --os-variant
=debian10 --graphics none -x console=ttyS0,115200 --disk /vm/firstguest.img,format=raw,size=200

You can start the guest with virsh start firstguest and open a console into it with virsh console firstguest. Debian will guide you through the installation process.

Then I transferred the data from the old guest with a mix of rsync and sftp.

Moving a KVM guest to the new host

This is also quite simple and there are plenty of instructions on how to do this on the web:

Shrinking a KVM guest image

Transferring the third guest was by far the most complicated. The new server had less space than the old one (it’s an SSD which, at this point, is more expensive than the HDD on the old server) so there wasn’t enough space for the guest image. We wanted to avoid having to set up everything from scratch so the only option was to somehow shrink the image from 700GB to something like 200GB. It looked doable because only about 100GB were used in the old image.

The first step was to copy the image and XML definition to the new server as we did for secondguest above. You can’t simply run resize2fs on that image because it is not just one partition but it includes everything, the partition table, the boot sector, a swap partition etc. And because this is all headless, I couldn’t run GParted. I couldn’t even run the command-line program GNU parted because we’re dealing with a KVM image that cannot be easily mounted onto the host (at least I’m not aware of how).

Shrinking the ext4 partition

You can list the file systems contained in the image with virt-list-filesystems -al thirdguest.img. Those need to be recreated later so it’s good to write them down. In my case, there was only one ext4 partition (/dev/sda1) and one swap partition (/dev/sda5). Log into the guest and write down the UUID of the swap partition which you’ll find in /etc/fstab. It makes things easier later.

Then I downloaded the ext4 partition into its own file:

guestfish -i -a thirdguest.img download /dev/sda1 thirdguest_dev_sda1.img

This image file was now something I could work with. First, to be on the safe side, I checked its integrity:

e2fsck -f thirdguest_dev_sda1.img

It looked good so I proceeded to shrink it:

resize2fs thirdguest_dev_sda1.img 200G

Afterwards, the file can be truncated to this exact size:

truncate -s 209715200K thirdguest_dev_sda1.img

Reassembling the KVM image

Now I needed to put the whole thing together again so it would run in the KVM host. The following are the commands I used in the guestfish program. Basically, I’m creating the two partitions, upload the resized file system into the first one and make a swap partition out of the second one.

I also had to reinstall grub2 or the virtual machine wouldn’t boot. The grub-install command is invoked on the host so I needed to install it there first:

/usr/bin/env DEBIAN_FRONTEND=noninteractive apt-get install -y grub2

Here are the guestfish steps:

alloc thirdguest.img 215G
run
part-init /dev/sda mbr
part-add /dev/sda primary 2048 419432447
part-add /dev/sda primary 419432448 -1
pvcreate /dev/sda1
pvcreate /dev/sda2
upload thirdguest_dev_sda1.img /dev/sda1
mkswap /dev/sda2 uuid:4a6b77e1-5b61-460c-a794-452f8a168310
mount /dev/sda1 /
write /boot/grub/device.map "(hd0) /dev/sda"
command "grub-install /dev/sda"
command "update-grub"
write /boot/grub/device.map "(hd0) /dev/vda"

Now I was able to add the image to the KVM host:

virsh define thirdguest.xml
virsh start thirdguest
virsh console thirdguest

The last command was simply used to log into the machine and verify that it’s working (and set up networking). And sure enough, everything worked! (After many frustrating hours of googling and trying different things.)