Migrating to KVM

Background

A few years ago, the services I provide (web server, mail server and so on) used to run on the same machine that I used as my desktop. It was powerful, spacious but it was also rather hungry on power. So, after some deliberation, I bought myself a fit-PC2i which was quite a power-miser. It had a 1.6Ghz CPU and 1G of RAM and was plenty enough for day-to-day work and serving, and freed up the bigger machine for pure desktop usage. However, I started to notice that occasionally it would freeze for a few seconds. My suspicion is that this is being caused by the SSD which doesn’t support TRIM¹. So, I am now looking into moving my server onto my housemate’s server as a KVM instance.

Virtualisation

Virtualisation is, of course, the big thing in servers these days. Why fill your data centre with big computers which may be sitting mostly idle? Why not pack those servers with “virtual machines” and make better use of that expensive hardware?

I have played around with VirtualBox in the past and that is generally a really good way to get started with desktop virtualisation. You start the application, define your machine, install from a CD and then run the new machine in a window. Now, while VirtualBox does have a “headless” mode, running machines “as a service” isn’t really it’s forté.

A couple of months ago, I also tried out Xen as a virtualisation solution. Xen is much more like the “Big Boys” run virtualisation. It can provide accelerated networking, disk access and the like. You can set virtual machines to start and stop with the host and so on. However, the way Xen works is that you need to boot the physical hardware into the Xen hypervisor. The Xen hypervisor then boots an initial guest system which is nominated as privileged (in Xen terms, this is Domain-0, Dom0). The Dom0 system can then start all the other guests (Domain-1, Domain-2… collectively, DomU). This is great for security as only Xen has direct access to the hardware and it works closely with Dom0; the DomUs may or may not be allowed direct access as required. The downside, though, is that issue of booting into Xen. Not terribly kind when you’re piggy-backing on a friend’s already-set-up server.

So I’ve chosen to give QEMU-KVM a try. KVM is part of the Linux Kernel that, in a way, allows Linux to work like Xen, but without the explicit super-server. kvm.ko is a loadable module which allows programs to access the virtualisation capabilities of modern processors. In this way, the Linux kernel accesses the hardware (as usual) and the guest system is treated as just another process. When the guest system wants to use the CPU (or the disk or any other hardware), then this request is passed to the Linux kernel’s scheduler just as if a browser or whatever wanted the CPU. Of course, the guest kernel will have its own scheduler but this is really not much different to the threads in an application.

While KVM is the kernel side of things (allowing an application access to the virtualisation functions), QEMU is the application that accesses KVM. QEMU is a wonderfully capable program. I first came across it before KVM existed, when it was marketed as a sort of generic emulator. It had the ability to run an OS for almost any instruction set on any other (so you could run ARM binaries on an x86 or amd64 binaries on a sparc). It was, however, rather slow (due to the need for translating the instructions as it goes). QEMU can, however, be paired with the KVM system as a sort of “accelerator”. When emulating one instruction set on hardware of the same type (so x86 on x86), then there’s no need for translation; the instructions are passed to KVM with runs them natively. As a result, QEMU+KVM achieves “near native” performance (that is, there is a slight performance hit, but its fractions of a percent).

Preparation

My friend runs Debian, so getting qemu installed was just a matter of:

# aptitude install qemu-kvm libvirt-bin

I plan to administer the virtual machine from a laptop, so on that machine do:

# aptitude install virt-manager

I started by connecting Virt-Manager to the VM Host (choosing KVM over SSH). However, this resulted in “nc” producing a “connect failed: permission denied” error. The solution was, on the VM Host, to add my user to the “libvirt” (so I can talk to the libvirt daemon) and “kvm” (so I can create/manage VMs) groups:

# adduser paul libvirt
# adduser paul kvm

Cloning

One of the tasks involved in moving to a virtual machine is the “Physical to Virtual” (P2V) conversion. This involves capturing the spec of the machine (Number of CPUs, amount of RAM, etc), making an image of the hard drive and then turning one off and the other on. Now, as I’m only moving one machine, making this partly manual isn’t a great problem. So I’ll set up the new virtual machine manually, do the cloning and then do the switch over. For the cloning of the disks, I could use the wonderful rsync to copy files over, but the problem with that is, even though rsync can do very quick “catch-up” syncs, there’s always got to be some period of time between doing the final sync and shutting down. So, what I want is some system whereby I can power the system off and then synchronise the disk.

So, for this, I’m going to look at Clonezilla. Again, as I’m looking for a quick-and-dirty method of cloning, I’ll not bother with setting up a Clonezilla server, but instead, I’ll make a Live USB stick from Clonezilla, boot off that and use SSH to transfer the image.

So, taking a blank USB stick (formatted as FAT32 as usual), download the Clonezilla ZIP file, and unzip it to the stick’s root.

$ wget http://sourceforge.net/projects/clonezilla/files/clonezilla_live_stable/2.2.0-31/clonezilla-live-2.2.0-31-i686-pae.zip/download
$ unzip clonezilla-live-2.2.0-31-i686.pae -d /media/usb

Then, from the utils/linux directory of the stick, run “makeboot.sh {USB Device}”²

$ cd /media/usb/utils/linux
$ bash makeboot.sh /dev/disk/by-id/usb--part1

and follow the prompts. I had to install “parted” (the partition editor) before continuing.

Now that your USB stick is prepared, this is a good time to set up the VM ready to recieve the cloned image. That’s in the next section. When that’s ready, we’ll reboot the guest and boot from the Clonezilla stick.

You should be presented with a yellow splash screen at boot, press enter to select the default option “Clonezilla Live”. Select English, then pick the UK keyboard. Select Start_Clonezilla, then “device-image” to copy from the partition to an image. I’m using a spare partition on the USB stick to accept the image so, on the next page select “local_dev” and then pick the partition on the USB stick. There’s no need to copy all the images (we can skip swap, for example) so select “Saveparts” and pick the partitions to clone. From here the defaults (name of the image etc) are good enough for us, so click through and let the cloning happen. I managed a little under 1GB/s, so the cloning took about 10 minutes.

When it’s done, select power off and move the USB stick to another computer. Clonzilla will have put the image in a series of compress parts. To reconstitute the image, we do the following:

# cat sda1.xfs-ptcl-img.gz.* | gzip -d -c | partclone.restore -C -s - -O sda1.xfs-ptcl.img
$ qemu-img convert -c -p -O qcow2 sda1.xfs-ptcl.img sda1.xfs-ptcl.qcow2

(adjust, of course, for your clonezilla image name). This will result in a compressed qcow2 image which the VM can be booted from.

VM Creation

OK. So we have an image of the old machine, time to create the new machine. On the host, install the qemu-kvm and libvirt-bin packages. Now, in my fiddling, I found a permissions issue so, to fix it, add your user to the kvm, libvirt, and libvirt-qemu groups:

$ adduser paul kvm     # and so on

Next, check that the kvm device has the right permissions:

$ ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 Dec 30 13:28 /dev/kvm

Note that the device is readable and writable by the kvm group. Finally, reload the kvm module appropriate for your CPU (kvm_amd or kvm_intel):

$ modprobe -r kvm_amd
$ modprobe kvm_amd

This should allow your user to use the KVM acceleration.

Next up, the following seems to be the “most Debian” way of setting up networking in my situation. I already have a bridge on the host machine, so I made an ethernet tap to which qemu can connect, at added that to the bridge. In /etc/network/interfaces modify the bridge to read:

auto br0
iface br0 inet static
    pre-up ip tuntap add dev tap0 mode tap user darac
    pre-up ip link set tap0 up
    mtu 1500
    address 192.168.101.254
    network 192.168.101.0
    netmask 255.255.255.0 
    broadcast 192.168.101.255
    bridge_ports eth0.1 eth1 eth2 tap0
    bridge_fd 5
    bridge_maxwait 5
    bridge_stp on
    post-up brctl sethello br0 5
    post-up brctl setfd br0 5
    post-down ip link set tap0 down
    post-down ip tuntap del dev tap0 mode tap

The highlighted lines (pre-up, post-down and bridge-ports) are what I modified. The pre-up and post-down bring the tap up and down with the bridge and the tap is added to the bridge-ports line. Note that tap0 (on the host) will be eth0 on the guest.

And now, I’m ready to actually create the virtual machine. Start Virt-Manager and connect it to the VM host (using SSH). Add a new Virtual Machine, naming it as appropriate (the hostname seems the most obvious choice). Choose “Import existing disk image” as the installation medium. Now, virt-manager is expecting the hard drive image there, but we’re not ready with that just yet, so let’s create a dummy for the moment. Switch back to the Virt-Manager window and select “Edit” > “Connection Details”. On the Storage tab, click “New Volume”. Give it a name and a capacity, but leave the allocation as 0 (so it doesn’t actually take up much space for now). Set the format to qcow2 and click finish. Now switch back to the wizard and, on the second page, click the Browse button to add the disk we just created. Set the OS type to Linux / Debian Wheezy. On the next page, allocate some RAM and CPU and then, on the last page be sure to click “Advanced Options”. Here, you should be able to choose “tap0” as the NIC and set the “Virt Type” to kvm.

Right, with that all done, the machine should boot, but stop because the disk you provided was empty. Time to do the cloning.

When you have the cloned image file, remove the blank disk, attach the image disk, et voila.

You may need to boot in single-user mode to adjust /etc/fstab, /etc/network/interfaces and so on.