On this page I am going to document the steps I went through to install the Xen hypervisor on my Pine ROCKPro64 single board computer.

A quick reminder of the key features of the ROCKPro64 which are relevant to me:

  • Hex core ARM processor – The ARM port of Xen is generally considered more secure than the x86 version as the code base is considerably smaller. Six cores leaves me lots of room to isolate VMs on their own cores for increased security.
  • 4GB RAM – A relatively large amount of RAM for a SBC which is required to support multiple VMs.
  • Gb Ethernet
  • A real PCIe 4x slot – Will allow me to fit a SATA controller card to efficiently drive a small RAID array.
  • Small and relatively inexpensive.
  • The RK3399 SOC has already been used to run Xen on some other boards.

There is also a powerful GPU, but I won’t be using that.

A few key points about my installation which will be relevant to anyone trying to follow along:

  • I will be using Gentoo for dom0 for no other reason than that I am used to it.
  • All control will be via the serial console. I won’t be switching on the GPU at all.
  • I will be booting from an SD card which I can mount and manipulate by plugging it into a Linux laptop.

Stage 1 – Install Ubuntu image

The first step is to get the board working with one of the default Linux installations. This not only checks that the hardware is working OK, but also installs a bootloader which you can then use in subsequent stages.

I used the Ubuntu Bionic minimal release from here, but there may well be a new version available by the time you read this. The release consists of a single .img.xz file which you need to write to your SD card. I put the SD card into my laptop (where it appears as /dev/sdc) and extracted the image with the following command:

xzcat bionic-minimal-rockpro64-0.9.14-1159-arm64.img.xz > /dev/sdc

The process will take a while, but when it finishes you should find that your SD card now has seven partitions (the purposes of most of which remain a mystery to me). Put the SD card in your ROCKPro64 and switch it on. It should boot into a minimal Ubuntu for which the default username and password are both ‘rock64’.

Stage 2 – Switching to Gentoo ‘Stage 3’

The next step is to switch over to running the Gentoo Stage 3 install image. We will do this by retaining the Ayufan kernel and boot loader, but replacing the rest of the operating system.

You need to obtain a suitable ARM64 Stage 3 tar ball from a Gentoo mirror. Mine was called stage3-arm64-20200609.tar.bz2, but use whatever is the most recent one.

The next step is to remove Ubuntu from the SD card. Put it back into the laptop and mount the seventh partition somewhere suitable (remembering that your SD card might not end up as sdc.)

mount /dev/sdc7 /mnt/sdcard

Next, delete everything except boot, lib/modules and lost+found. Then untar your stage 3 tar ball over the top of what is left. Finally, two small edits are required. Firstly, Gentoo doesn’t have a default root password, so we need to have an empty root password on first boot. Delete the ‘x’ from the second field of the first line of /etc/passwd to achieve this.

The final edit is related to the use of the serial console. Edit /etc/inittab and comment out the lines under ‘TERMINALS’ that start ‘c1’ etc and the last line which starts ‘f0’. Then, uncomment the first of the serial console lines and edit it as follows (note that we need to use ttyS2):

s0:12345:respawn:/sbin/agetty -L 1500000 ttyS2 vt100

If you return the card to the ROCKPro64 and boot you should find yourself in a Gentoo Stage 3 install but still running the Ayufan kernel. There will be no root password, so you should set this via the passwd command as soon as you log in.

Stage 3 – Setting up your Gentoo environment

Most of the next steps are the normal ones to install Gentoo. Ideally, one would follow the instructions in the Gentoo ARM64 handbook. Unfortunately, for some loony reason there isn’t one of these for ARM64 or even ARM32 despite the existence of handbooks for such ‘popular’ architectures as MIPS and Alpha! The best plan is to just follow the AMD64 one and adapt as necessary.

Before proceeding, it is worth making sure that the fan runs as there is much compiling to be done. Eventually, we should be able to get proper fan speed control working, but for the time being a simple fixed fan speed will suffice. I added an executable file called fan.start to /etc/local.d as follows. Adjust the 100 value to get the fan speed that you desire (max 255).

echo disabled > /sys/devices/virtual/thermal/thermal_zone0/mode
echo 100 > /sys/devices/platform/pwm-fan/hwmon/hwmon0/pwm1

For added complexity, the handbooks are written around the idea that one will chroot into the Stage 3 from a functioning machine which, for example, already has networking functional. Since that isn’t the case here, we need to set up networking to start with, but that is tricky because dhcpcd isn’t installed and without networking we can’t do the emerge-webrsync stage needed to get started with installing.

The solution I adopted was to start by manually setting the IP address and route using ifconfig and then making dhcp one of my first installs. I also had to put my local DNS details in /etc/resolv.conf and remember to set the date and time manually until I could get ntp installed.

The root partition has another difficulty in a shortage of inodes. This causes problems once you start to do ’emerge –sync’ as the huge number of files in /var/db/repos/gentoo uses them all up. I created an additional partition in the empty space at the end of the SD card and mounted it over the offending folder. I did the same trick with /usr/src. Eventually these can be replaced with logical volumes for more flexibility.

Stage 4 – Kernel

With the basic operating system installed, it is time to build your kernel. Here are a few notes on useful configuration options to select. Clearly, many choices will be driven by your intended application, for example – whether you intend to use the GPU.

  • Under Kernel Features -> ARM errata… there are numerous options to fix possible ARM bugs. For the processor on my board the only two which seem to be required are 845719 and 843419.
  • Since this kernel will eventually run on your DOM0 VM, you need to select Kernel Features -> Xen guest support…
  • The Ethernet device you need is the STMicroelectronics one. Under PHY Device support, be sure to select the “Driver for Rockchip Ethernet PHYs”.