Xen on ROCKPro64

On this page I am going to document the steps I went through to install the Xen hypervisor on my Pine ROCKPro64 single board computer.

A quick reminder of the key features of the ROCKPro64 which are relevant to me:

  • Hex core ARM processor – The ARM port of Xen is generally considered more secure than the x86 version as the code base is considerably smaller. Six cores leaves me lots of room to isolate VMs on their own cores for increased security.
  • 4GB RAM – A relatively large amount of RAM for a SBC which is required to support multiple VMs.
  • Gb Ethernet
  • A real PCIe 4x slot – Will allow me to fit a SATA controller card to efficiently drive a small RAID array.
  • Small and relatively inexpensive.
  • The RK3399 SOC has already been used to run Xen on some other boards.

There is also a powerful GPU, but I won’t be using that.

A few key points about my installation which will be relevant to anyone trying to follow along:

  • I will be using Gentoo for dom0 for no other reason than that I am used to it.
  • All control will be via the serial console. I won’t be switching on the GPU at all.
  • I will be booting from an SD card.
  • I have a Raspberry Pi 4 with a 64 bit version of Gentoo installed which I used for much of the preparation. As an alternative, I suggest using your RockPro64 with two SD cards; one with a default Linux installation and the other which you will install Xen onto via a USB adapter.

Stage 1 – Getting u-boot working

The first step is to get the board working with one of the default Linux installations. This not only checks that the hardware is working OK, but also installs a bootloader which you can then use in subsequent stages.

We will be booting Xen directly from the u-boot bootloader so it is simplest to first get Linux to boot in the same way. Unfortunately, most instructions online involve using an intermediary between u-boot and the Linux kernel (such as Grub) and I found that this just complicates things.

The first step is therefore to create the two files which form the u-boot first and second stage loaders. I am most grateful to Andrius Štikonas for his handy post whose instructions I urge you to follow. Remember that even though you may be building the software on an ARM64 system, you will still need to install crossdev to provide an ARM32 compiler.

Andrius suggests creating two 4MB partitions to hold the two u-boot images. In practice I found it tricky to get things working properly with the first partition starting at sector 64 as gpt partition tables seem reluctant to allow partitions before sector 2048. In the end I compromised and used a partition only for the second u-boot module at sector 16384 and used the dd seek option to install the first module at sector 64 as follows.

dd if=idbloader.img of=/dev/sda seek=64
dd if=u-boot.itb of=/dev/sda1

I then created two more partitions to hold /boot and the root partition both formatted as ext4. My resulting gpt partition table looked as follows:

Disk /dev/sda: 14.86 GiB, 15931539456 bytes, 31116288 sectors
Disk model: STORAGE DEVICE  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 40AA7963-4AF1-4548-8167-37D8D0A56C04

Device      Start      End  Sectors  Size Type
/dev/sda1  16384    24575     8192    4M BIOS boot
/dev/sda2  24576   548863   524288  256M Linux filesystem
/dev/sda3 548864 17326079 16777216    8G Linux filesystem

Finally, u-boot needs to find a boot script which it normally looks for in the first partition. In our case it will be in the second partition (/boot) and we indicate this by setting the legacy boot flag for this partition. In fdisk you set this by selecting ‘x’ for expert mode and then using the ‘A’ command.

If you try to boot from the card now, you should see u-boot start up and be able to interrupt the boot sequence with a key press and end up at the u-boot command line.

Stage 2 – Booting into Gentoo ‘Stage 3’

The next step is to switch over to running the Gentoo Stage 3 install image. For this we will need a suitable kernel image, a device tree file and possibly an initrd RAM disk image. Note that I don’t tend to use initrd images and just make sure that I build my kernel with everything needed for booting built in. You will of course also need to obtain a suitable ARM64 Stage 3 tar ball from a Gentoo mirror.

Andrius has some handy hints for compiling your kernel and when you have finished you should have your kernel image file and a device tree file called rk3399-rockpro64.dtb. Copy both of these to your second partition which will become /boot.

Unpack the Stage 3 tar-ball to the larger partition and configure it via chroot as per the Gentoo handbook. Obviously one would expect to follow the ARM64 version but unfortunately, for some loony reason there isn’t (at the time of writing in 2021) one of these for ARM64 or even ARM32 despite the existence of handbooks for such ‘popular’ architectures as MIPS and Alpha! The best plan is to just follow the AMD64 one and adapt as necessary.

If like me you wish to usethe serial console then you will need to edit /etc/inittab. Comment out the lines under ‘TERMINALS’ that start ‘c1’ etc and the last line which starts ‘f0’. Then, uncomment the first of the serial console lines and edit it as follows (note that we need to use ttyS2):

s0:12345:respawn:/sbin/agetty -L 1500000 ttyS2 vt100

Finally (for this stage of the operations) we need a u-boot boot script which should look like the following.

ext4load mmc 1:2 $kernel_addr_r vmlinux
ext4load mmc 1:2 $fdt_addr_r rk3399-rockpro64.dtb
setenv bootargs rw panic=10 init=/sbin/init root=/dev/mmcblk0p3 rootfstype=ext4
booti ${kernel_addr_r} - ${fdt_addr_r}

The first two lines tell u-boot to load the kernel image and dtb files into memory at pre-defined addresses from the second partition of the first disk (1:2). We then set an environment variable containing the kernel command line and finally call the booti command to actually boot the kernel image.

In order for u-boot to find and execute your boot script two things are needed. The first is that it be called boot.scr and located in the top level of the partition we identified earlier with the boot flag. The second is that it be converted to u-boot format and for this we need the handy mkimage command which came with u-boot. You can also choose to install it on Gentoo as part of the u-boot-tools package. The conversion command is as follows:

mkimage -A arm -T script -d boot.txt boot.scr

Note that there are a lot of instructions online telling you to specially prepare your kernel image and use the bootm command. These were clearly written before u-boot acquired the ability to directly boot kernel images. The only slight variation you might need is to use the bootz command if your kernel image is compressed.

Re-boot, and in the unlikely event that everything is working correctly you will arrive at the Linux command line.

Before proceeding, it is worth making sure that the fan runs as there is much compiling to be done. Eventually, we should be able to get proper fan speed control working, but for the time being a simple fixed fan speed will suffice. I added an executable file called fan.start to /etc/local.d as follows. Adjust the 100 value to get the fan speed that you desire (max 255).

#!/bin/bash
echo disabled > /sys/devices/virtual/thermal/thermal_zone0/mode
echo 100 > /sys/devices/platform/pwm-fan/hwmon/hwmon0/pwm1

Stage 3 – Installing and switching to Xen

Firstly, we need the Xen tools including the essential xl command which we get by emerging xen-tools. Then we also need to emerge the xen ebuild to obtain Xen itself. Also, make sure that your Kernel contains the various options required to run properly under Xen. The Gentoo Xen Wiki page has useful tips on this.

Installing the Xen ebuild puts the Xen executable into /boot. Unfortunately we can’t use this directly as we need to convert it to a uboot executable first. Once again, the mkimage command comes to our aid as follows:

mkimage -A arm64 -T kernel -a 0x02000000 -e 0x02000000 -C none -d xen xen.uImage

Note that we have specified various memory locations and produced a new file called xen.uImage which we will now be booting from.

We now need a suitable boot script which will both boot the Xen kernel and give it sufficient information to allow it to then load the Linux kernel and boot dom0. A suitable example is provided below:

ext4load mmc 1:2 $kernel_addr_r xen.uImage
ext4load mmc 1:2 $fdt_addr_r rk3399-rockpro64.dtb
ext4load mmc 1:2 $ramdisk_addr_r vmlinux-xen
setenv kernel_size 0x${filesize}
fdt addr $fdt_addr_r
fdt resize

fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>

fdt mknod /chosen module@0
fdt set /chosen/module@0 compatible "multiboot,kernel" "multiboot,module"
fdt set /chosen/module@0 reg <${ramdisk_addr_r} ${kernel_size}>
fdt set /chosen/module@0 bootargs "console=hvc0 earlyprintk=xen clk_ignore_unused nomodeset rw panic=10 init=/sbin/init root=/dev/mmcblk0p3 rootfstype=ext4"
fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial2"

bootm ${kernel_addr_r} - ${fdt_addr_r}

Clearly, this is quite a bit more complex than the previous script (and note that as before it needs to be converted with mkimage).

The first lines load the Xen and Linux kernels and the device tree into pre-defined memory locations as before. The fourth line sets a variable containing the size of the Linux kernel using the ‘filesize‘ variable which contains the size of the last file loaded.

Parameters are passed to the Xen kernel using additional parameters created in the device tree and most of the rest of the script is involved in setting these up and some of these parameters are specific to the use of a serial console.

The ‘xen-bootargs’ line tells the Xen kernel to use a serial console and then specifies the relevant port as ‘serial2’ which is the same as the ‘ttyS2’ which we set earlier.

The ‘bootargs’ line is of course the Linux kernel command line. Here we set the Linux console to be /dev/hvc0 which is the connection to the Xen console which is in our case the serial console. The earlyprintk and clk_ignore_unused options are involved in ensuring that the kernel boot messages appear correctly.

Finally, the bootm command causes the Xen kernel previously loaded into memory to be executed.

There is one final change which you may need to make before re-booting. You will need to change the serial console line in /etc/inittab which we changed earlier to now read hvc0 rather than ttyS2 so that a login prompt will appear.

Stage 4 – Kernel

With the basic operating system installed, it is time to build your kernel. Here are a few notes on useful configuration options to select. Clearly, many choices will be driven by your intended application, for example – whether you intend to use the GPU.

  • Under Kernel Features -> ARM errata… there are numerous options to fix possible ARM bugs. For the processor on my board the only two which seem to be required are 845719 and 843419.
  • Since this kernel will eventually run on your DOM0 VM, you need to select Kernel Features -> Xen guest support…
  • The Ethernet device you need is the STMicroelectronics one. Under PHY Device support, be sure to select the “Driver for Rockchip Ethernet PHYs”.