This is a guide for KVM Virtualization on the ARMv8 architecture and illustrates how to set up a KVM development environment on ARM64 processors. It showcases how to run a guest virtual machine on top of an arm hikey960 board host.
Requirements
The steps given in this guide are based on ubuntu 16.04 system, however all the tools used herewith are also available for other Linux distributions
-
64-bit ARM Cross Compiler
-
Buildroot
-
ARM64 Host and Guest Kernels 64-bit ARM kernels with KVM capability, which is mandatory for the Host system
-
Host and Guest 64-bit Userspace Images Userspace images for the systems we are going to run.
-
Kvmtool
-
usbstick
Prepare Guest kernel image, Guest disk image, kvmtool
Build guest kernel
The guest kernel has to be built with the following configuration:
For the default console output:
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=yFor running 32bit images on 64bit hosts:
CONFIG_IA32_EMULATION=yProper FS options according to image FS (e.g. CONFIG_EXT2_FS, CONFIG_EXT4_FS).
For all virtio devices listed below:
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=yFor virtio-blk devices (–disk, -d):
CONFIG_VIRTIO_BLK=yFor virtio-net devices ([–network, -n] virtio):
CONFIG_VIRTIO_NET=yFor virtio-9p devices (–virtio-9p):
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_9P_FS=yFor virtio-balloon device (–balloon):
CONFIG_VIRTIO_BALLOON=yFor virtio-console device (–console virtio):
CONFIG_VIRTIO_CONSOLE=yFor virtio-rng device (–rng):
CONFIG_HW_RANDOM_VIRTIO=yFor vesa device (–sdl or --vnc):
CONFIG_FB_VESA=y
rootfs disk Image for the Guest
Create the disk image for the guest using the following steps:
$ dd if=/dev/zero of=./disk_oe64.img bs=1MiB count=512
$ mkfs.ext3 ./disk_oe64.img
$ mkdir mnt.disk
$ sudo mount -o loop disk_oe64.img mnt.disk
$ sudo tar -xf linaro-image-minimal-genericarmv8-20170127-888.rootfs.tar.gz
-C mnt.disk/
$ sudo umount mnt.disk && rmdir mnt.disk
Copy guest kernel image and guest disk image into a USB stick.
Building KVMtool for arm64
kvmtool depend on libfdt-dev, libbfd, zlib, aio, to simplify cross compile we will us buildroot to build kvmtool.
Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation.
-
Download buildroot from release Buildroot - Making Embedded Linux Easy
-
make menuconfig and select aarch64 toolchain
-
make arm_juno_defconfig, select juno default configuration
-
tweak kvmtool build cmd to build static version of kvmtool, in
buildroot/package/kvmtool/kvmtool.mk add lkvm-static
define KVMTOOL_BUILD_CMDS
$(TARGET_MAKE_ENV) ARCH=$(KERNEL_ARCH) $(MAKE) -C $(@D) $(KVMTOOL_MAKE_OPTS) lkvm-static
endef
- make kvmtool
Start kvm hypervisor to EL2
Due to hikey960 default bootloader can not support boot kernel from EL2 like u-boot, we will uses UEFI to boot kernel from EL2 and start kvm hypervisor.
Download prebuilt UEFI image
http://builds.96boards.org/snapshots/reference-platform/components/uefi-staging/55/hikey960/release/
Or you can build UEFI load from source
for details you can reference: https://github.com/96boards-hikey/tools-images-hikey960/blob/master/build-from-source/README-ATF-UEFI-build-from-source.md
Recover flash UEFI
Config board to recovery mode (1, 2 ON, 3 OFF)
This Hardware User Manual shows the meaning of different pins and components of this board.
Using tools in this dir to flash our new loader:
$ sudo ./hikey_idt -c config -p /dev/ttyUSB0, (ttyUSB0 is the one your type-c is connecting). After this step, the board will run a fastboot program which means we can flash other files.
partition table
$sudo fastboot flash ptable ${UEFI_BUILD_PATH}/prm_ptable.img
bootloader
$sudo fastboot flash xloader ${UEFI_BUILD_PATH}/sec_xloader.img
$sudo fastboot flash fastboot ${UEFI_BUILD_PATH}/l-loader.bin
$sudo fastboot flash fip ${UEFI_BUILD_PATH}/fip.bin
Update UEFI boot.img and rootfs.img
download UEFI boot.img and rootfs.img from rbp release:
boot-0.0+AUTOINC+ba45819943-ea12986b87-r0-hikey960-20180202071634-128.uefi.img → boot.img (disk boot sector)
rpb-console-image-hikey960-20180202071634-128.rootfs.img.gz → rootfs.img (rootfs disk image)
http://snapshots.linaro.org/reference-platform/embedded/morty/hikey960/latest/rpb/
$sudo fastboot flash boot uefi.img
$sudo fastboot flash system rootfs.img
Boot Debian from UEFI on hikey960
Config board to autoboot mode (1 ON, 2, 3 OFF)
UEFI will find disk boot sector and load Grub bootload
Grob will load kernel image from disk rootfs path /boot/Image (If you want to update kernel image, replace /boot/Image)
Make sure Hyp mode initialized successfully
/----------------------------------------------------------------------------
|*CE Reference Platform (HiKey960 rpb) |
| Fastboot |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
----------------------------------------------------------------------------/Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, `e' to edit the commands before booting or `c' for a command-line. ESC to return previous menu.The highlighted entry will be executed automatically in 0s.
Booting `CE Reference Platform (HiKey960 rpb)’hikey960:/home/linaro# dmesg|grep -i kvm
[ 0.294921] kvm [1]: 8-bit VMID
[ 0.294924] kvm [1]: IDMAP page: a71000
[ 0.294926] kvm [1]: HYP VA range: 800000000000:ffffffffffff
[ 0.295566] kvm [1]: Hyp mode initialized successfully
[ 0.295758] kvm [1]: vgic-v2@e82b4000
[ 0.295839] kvm [1]: vgic interrupt IRQ1
[ 0.295849] kvm [1]: virtual timer IRQ4
Run KVM on hikey960
mount usb stick and run KVM
hikey960:~# mount /dev/sde2 /mnt/
hikey960:/mnt/home/root/kvm# kvmtool-bed2bd9e1fbef5819090feeada7b86eed97ca5e2/lkvm-static run --kernel boot/Image -d disk_oe64.img -m 512 --console virtio –
params “earlyprintk=shm console=hvc0 root=/dev/vda”lkvm run -k boot/Image -m 512 -c 8 --name guest-3395
Info: Loaded kernel to 0x80080000 (13773312 bytes)
Info: Placing fdt at 0x8fe00000 - 0x8fffffff
Info: virtio-mmio.devices=0x200@0x10000:36Info: virtio-mmio.devices=0x200@0x10200:37
Info: virtio-mmio.devices=0x200@0x10400:38
[ 1.718158] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 1.720656] brd: module loaded
[ 1.723143] loop: module loaded
[ 1.726397] mtdoops: mtd device (mtddev=name/number) must be supplied
[ 1.726729] libphy: Fixed MDIO Bus: probed
[ 1.729531] sky2: driver version 1.30
[ 1.729779] ehci_hcd: USB 2.0 ‘Enhanced’ Host Controller (EHCI) Driver
[ 1.729896] ehci-pci: EHCI PCI platform driver
[ 1.730034] ehci-platform: EHCI generic platform driver
[ 1.730143] ohci_hcd: USB 1.1 ‘Open’ Host Controller (OHCI) Driver
[ 1.730272] ohci-pci: OHCI PCI platform driver
[ 1.730427] usbcore: registered new interface driver usb-storage
[ 1.730868] IR NEC protocol handler initialized
[ 1.730985] IR RC5(x/sz) protocol handler initialized
[ 1.731064] IR RC6 protocol handler initialized
[ 1.731268] IR JVC protocol handler initialized
[ 1.731368] IR Sony protocol handler initialized
[ 1.731685] IR SANYO protocol handler initialized
[ 1.731775] IR Sharp protocol handler initialized
[ 1.732072] IR MCE Keyboard/mouse protocol handler initialized
[ 1.732186] IR XMP protocol handler initialized
[ 1.732790] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com
[ 1.733391] usbcore: registered new interface driver usbhid
[ 1.733482] usbhid: USB HID core driver
[ 1.734476] Initializing XFRM netlink socket
[ 1.734604] NET: Registered protocol family 17
[ 1.734710] NET: Registered protocol family 15
[ 1.734848] 9pnet: Installing 9P2000 support
[ 1.734974] Key type dns_resolver registered
[ 1.736390] Btrfs loaded, crc32c=crc32c-generic
[ 1.736581] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.736969] hctosys: unable to open rtc device (rtc0)
[ 1.737115] ALSA device list:
[ 1.737385] No soundcards found.
[ 1.738247] EXT4-fs (vda): mounting ext3 file system using the ext4 subsystem
[ 1.738977] EXT4-fs (vda): INFO: recovery required on readonly filesystem
[ 1.739157] EXT4-fs (vda): write access will be enabled during recovery[ 6.091847] EXT4-fs (vda): recovery complete
[ 6.562293] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[ 6.566264] VFS: Mounted root (ext3 filesystem) readonly on device 254:0.
[ 6.567145] devtmpfs: mounted
[ 6.570496] Freeing unused kernel memory: 832K
INIT: version 2.88 booting
Starting udev
/etc/rcS.d/S03systemd-udevd: line 76: /proc/sys/kernel/hotplug: No such file or directory
starting version 232
[ 7.677081] EXT4-fs (vda): re-mounted. Opts: data=ordered
hwclock: can’t open ‘/dev/misc/rtc’: No such file or directory
Fri Jan 27 07:36:34 UTC 2017
hwclock: can’t open ‘/dev/misc/rtc’: No such file or directory
INIT: Entering runlevel: 5
Configuring network interfaces… udhcpc (v1.24.1) started
Sending discover…
Sending select for 192.168.33.15…
Lease of 192.168.33.15 obtained, lease time 14400
done.
Starting OpenBSD Secure Shell server: sshd
done.
Starting rpcbind daemon…done.
starting statd: done
hwclock: can’t open ‘/dev/misc/rtc’: No such file or directory
exportfs: can’t open /etc/exports for reading
modprobe: can’t change directory to ‘/lib/modules’: No such file or directory
NFS daemon support not enabled in kernel
Starting syslogd/klogd: done
No makedumpfile found.
Starting auto-serial-console: done
INIT: no more processes left in this runlevelLast login: Fri Jan 27 07:36:35 UTC 2017 on tty1
root@genericarmv8:~#
