Moving KVM guests to a new host and: How to shrink a KVM image
This was written in February 2021 at about the time Debian 10.8 was released.
As I had ordered a new Hetzner server, it was necessary to move three KVM guests running on the old server to the new host. The old host also served as a mail server and for simplicity, I had decided to use https://github.com/docker-mailserver/docker-mailserver for the new server. This is an important detail as Docker modifies your
iptables rules, leading to your KVM guests losing internet access. More on that later.
Let’s assume that our new server’s IP address is
126.96.36.199 with a gateway of
188.8.131.52. For the virtual machines, I had ordered a subnet with 6 additional addresses, let’s assume they are
184.108.40.206/29. All machines (host and guests) are running Debian, version 10.8 at the point this was written.
Setting up networking
Based on Hetzner’s own documentation, I had to route the guest networks through the host. A direct bridge doesn’t work as they don’t provide MAC addresses for each address in the IPv4 block. The configuration ended up as follows (doing this for IPv6 is left as an exercise to you):
source /etc/network/interfaces.d/* auto lo iface lo inet loopback auto enp5s0 iface enp5s0 inet static address 220.127.116.11 netmask 255.255.255.255 pointopoint 18.104.22.168 gateway 22.214.171.124
auto virbr1 iface virbr1 inet static address 126.96.36.199 netmask 255.255.255.255 # putting a gateway/pointopoint entry here leads to the server not being bootable anymore. bridge_ports none bridge_stp off bridge_fd 0 pre-up brctl addbr virbr1 post-down brctl delbr virbr1 up ip route add 188.8.131.52/29 dev virbr1 scope host down ip route del 184.108.40.206/29 dev virbr1 scope host
Our guest VM has the IP address
source /etc/network/interfaces.d/* auto lo iface lo inet loopback auto enp1s0 iface enp1s0 inet static address 220.127.116.11 netmask 255.255.255.255 gateway 18.104.22.168 pointopoint 22.214.171.124
If you’re using your own image, make sure to configure the name servers in
/etc/resolv.conf. (Hetzner provides this in their install images, you can copy the file from the host.)
Problems with Docker
At this point, I spent hours trying to find out why my guests still didn’t have internet access. I finally realized it was because Docker modifies
iptables, setting FORWARD to DROP (you can google this), resulting in no traffic forwarded to/from the virtual machines anymore. There are instructions on the web on how to turn Docker’s modifications off and recreating the rules from scratch. But it’s too likely I was going to make mistakes as these rules are quite complex.
/etc/iptables/rules.v4 also didn’t help as those rules were simply overwritten. Almost all advice I found (including Docker’s own documentation) simply listed a command like this:
iptables -I DOCKER-USER -o virbr1 -i virbr1 -j ACCEPT. It (sort of) works but it’s not permanent. Your VMs will have no internet again after a reboot of the host.
I finally settled on a cron script which runs once a minute:
/usr/sbin/iptables -D DOCKER-USER -o virbr1 -j ACCEPT /usr/sbin/iptables -D DOCKER-USER -i virbr1 -j ACCEPT /usr/sbin/iptables -I DOCKER-USER -o virbr1 -j ACCEPT /usr/sbin/iptables -I DOCKER-USER -i virbr1 -j ACCEPT
It’s important to delete the rules first so they don’t stack up over time. (The first time you try to delete them, there’s an error message because they don’t exist yet, but that can be
Transferring the KVM guests to the new host
We had three VMs on the old host and the transfer method was different for each one.
Setting up a new KVM guest and copying the data over
This was fairly straightforward. How to install a new KVM guest is something that is thoroughly documented. I wanted to start with a minimal Debian installation so I opted for a minimal CD-ROM image which I downloaded into the
/vm directory. I installed it with a HD size of 200GB, 2 CPUs, and 16GB of RAM:
virt-install -n firstguest --vcpus 2 -r 16384 --network=bridge=virbr1,model=virtio --location /vm/debian-10.7.0-amd64-netinst.iso --os-variant =debian10 --graphics none -x console=ttyS0,115200 --disk /vm/firstguest.img,format=raw,size=200
You can start the guest with
virsh start firstguest and open a console into it with
virsh console firstguest. Debian will guide you through the installation process.
Then I transferred the data from the old guest with a mix of
Moving a KVM guest to the new host
This is also quite simple and there are plenty of instructions on how to do this on the web:
- Shut down the guest on the old host:
virsh shutdown secondguest(resort to
virsh destroy secondguestif this doesn’t work at all). You can also
shutdown nowon the guest itself.
- Export the guest definition into a file:
virsh dumpxml secondguest > secondguest.xml
- Copy/move both
secondguest.xmland the image file (e.g.
secondguest.img) to the new host.
secondguest.xmlmanually so it matches the new configuration you want (e.g. number of CPUs, RAM).
- Initialize the guest on the new host:
virsh define secondguest.xml
- Start it:
virsh start secondguest
- To configure the network from inside the guest:
virsh console secondguest
Shrinking a KVM guest image
Transferring the third guest was by far the most complicated. The new server had less space than the old one (it’s an SSD which, at this point, is more expensive than the HDD on the old server) so there wasn’t enough space for the guest image. We wanted to avoid having to set up everything from scratch so the only option was to somehow shrink the image from 700GB to something like 200GB. It looked doable because only about 100GB were used in the old image.
The first step was to copy the image and XML definition to the new server as we did for
secondguest above. You can’t simply run
resize2fs on that image because it is not just one partition but it includes everything, the partition table, the boot sector, a swap partition etc. And because this is all headless, I couldn’t run GParted. I couldn’t even run the command-line program GNU parted because we’re dealing with a KVM image that cannot be easily mounted onto the host (at least I’m not aware of how).
You can list the file systems contained in the image with
virt-list-filesystems -al thirdguest.img. Those need to be recreated later so it’s good to write them down. In my case, there was only one
ext4 partition (
/dev/sda1) and one swap partition (
/dev/sda5). Log into the guest and write down the UUID of the swap partition which you’ll find in
/etc/fstab. It makes things easier later.
Then I downloaded the
ext4 partition into its own file:
guestfish -i -a thirdguest.img download /dev/sda1 thirdguest_dev_sda1.img
This image file was now something I could work with. First, to be on the safe side, I checked its integrity:
e2fsck -f thirdguest_dev_sda1.img
It looked good so I proceeded to shrink it:
resize2fs thirdguest_dev_sda1.img 200G
Afterwards, the file can be truncated to this exact size:
truncate -s 209715200K thirdguest_dev_sda1.img
Reassembling the KVM image
Now I needed to put the whole thing together again so it would run in the KVM host. The following are the commands I used in the
guestfish program. Basically, I’m creating the two partitions, upload the resized file system into the first one and make a swap partition out of the second one.
I also had to reinstall
grub2 or the virtual machine wouldn’t boot. The
grub-install command is invoked on the host so I needed to install it there first:
/usr/bin/env DEBIAN_FRONTEND=noninteractive apt-get install -y grub2
Here are the
alloc thirdguest.img 215G run part-init /dev/sda mbr part-add /dev/sda primary 2048 419432447 part-add /dev/sda primary 419432448 -1 pvcreate /dev/sda1 pvcreate /dev/sda2 upload thirdguest_dev_sda1.img /dev/sda1 mkswap /dev/sda2 uuid:4a6b77e1-5b61-460c-a794-452f8a168310 mount /dev/sda1 / write /boot/grub/device.map "(hd0) /dev/sda" command "grub-install /dev/sda" command "update-grub" write /boot/grub/device.map "(hd0) /dev/vda"
- The first partition is offset by 2048 blocks which is a common thing to do.
- The start of the second partition is 419432448 which is calculated as follows: 2048 + 200 * 1024 * 1024 * 1024 / 512. It’s the size of the
ext4image file in bytes, divided by 512 (the size of one sector).
4a6b77e1-5b61-460c-a794-452f8a168310is the UUID of the swap partition which I wrote down above. This is not strictly necessary and can be edited in
/etc/fstablater but it saved me some time.
- From inside the guest (via
fdisk -c -u -l /dev/vda), I found that the swap partition’s type is now listed as “Linux” instead of “swap” and instead of
/dev/vda5it is now
/dev/vda2but that doesn’t seem to be a problem. Swapping works as it should.
- From the outside, the hard drive device is
/dev/sda(inside the guest it’s
grub-installneeds to know it so I had to write it in
Now I was able to add the image to the KVM host:
virsh define thirdguest.xml virsh start thirdguest virsh console thirdguest
The last command was simply used to log into the machine and verify that it’s working (and set up networking). And sure enough, everything worked! (After many frustrating hours of googling and trying different things.)