Ombro

This is the super-repository for Ombro, a port of Xen to the SVA virtual instruction set. For details, see our USENIX ATC 2022 paper.

This repository contains tools to make a from-scratch build of Ombro, including building LLVM with SVA patches and using it to build our Xen port and link the SVA runtime library.

Building

Prerequisites

You will need to have make, cmake, and a recent C++ compiler (e.g. clang or GCC) installed to build the SVA compiler.

We recommend you install Xen 4.12.0 according to your distribution's instructions before attempting to build or install Ombro. Ombro does not build or install its own Xen toolstack, a component which is necessary for running guest domains. Additionally, the SVA runtime library currently needs the Xen headers to be installed in order to locate them during the build. These headers are usually installed with the Xen toolstack. Note that headers or toolstacks from other versions of Xen are extremely unlikely to work.

Clone the sources for the SVA compiler, the SVA runtime library, and our port of Xen:

$ git submodule update --init --recursive

Build Ombro:

$ make

The top-level makefile will build the SVA compiler and use it to build the SVA runtime library and our port of Xen.

Configuring your system for Ombro

Unfortunately, the grub-mkconfig scripts for configuring Xen don't allow you to specify certain necessary details of how Ombro is booted. You will need to edit the GRUB configuration file (yes, the one that says "DO NOT EDIT THIS FILE" at the top). Alternatively, you can attempt to make the necessary changes by editing the grub-mkconfig scripts, usually located in /etc/grub.d/.

Xen command-line options

Add no-xpti no-pcid spec-ctrl=no,eager-fpu to Xen's command line.

Boot with Multiboot 1

Xen 4.12.0 built with clang crashes on boot when booted with Multiboot 2. Multiboot 1 works. This bug was not introduced by Ombro.

First, enable the Compatibility Support Module (CSM) in your system firmware, which is needed for Xen to boot with Multiboot 1. If you are using legacy BIOS boot, this step isn't necessary.

To boot Xen with Multiboot 1, find the Xen menuentry in the GRUB configuration and change this line

multiboot2 /path/to/xen ...

to read

multiboot /path/to/xen ...

(without the 2). Additionally, change any lines starting with module2 to instead start with module. If the line already starts with multiboot (no 2), then you are already using Multiboot 1 and don't need to change this line.

Don't use ZSTD compression for your kernel

Xen 4.12.0 is too old to support loading kernels compressed with ZSTD. If your distribution uses ZSTD to compress their kernels, we recommend you unpack the kernel bzImage using the extract-vmlinux script from the kernel source tree. From the kernel source root:

$ ./scripts/extract-vmlinux /path/to/bzImage > kernel-unpacked
$ sudo cp kernel-unpacked /boot/kernel-unpacked

Then, make sure your bootloader is configured to load the unpacked kernel as Xen's first module. For example, if Xen's menuentry contains a line referencing a compressed kernel image (bzImage) like

module /boot/vmlinuz-linux ...

Change it to

module /boot/kernel-unpacked ...

Note that this limitation applies both to the Xen hypervisor itself and the toolstack. This means you will need to extract (and make the necessary configuration changes for) both dom0 and domU kernels. In a domU's GRUB configuration (which is loaded by Xen's pygrub tool), change

linux /boot/vmlinuz-linux

(or any path to a compressed kernel image) to

linux /boot/kernel-unpacked

Disable Intel VMD

Xen 4.12.0 does not support Intel Volume Management Device (VMD). If your system has this feature, it will need to be disabled in your firmware settings.

Disable MMIO above 4GB

Some systems allocate MMIO space at the high end of the physical address space, outside the bounds of Ombro's frame metadata. The SVA runtime library therefore rejects attempts to map this memory, which can cause Ombro to crash. The current workaround is to disable MMIO above 4GB in your firmware settings. TODO: modify the frame metadata to allow for a discontiguous physical address space (it should be sufficient to allow just one hole).

Known bugs

Ombro will sometimes crash during boot while doing IO port emulation for the dom0 kernel. This crash is rare and nondeterministic and the cause is currently unknown.

The SVA runtime library does not yet auto-detect which XSave features are available. It currently tries to enable a hard-coded list in SVA/SVA/lib/init.c. If any of these features are not available, Ombro will crash from a general protection fault. The current workaround is to modify this line to include only features that are actually available on the hardware you intend to run Ombro on. TODO: automatically detect which XSave features are available and attempt to enable only those.