Mirage OS Logo
Home Blog Docs API Community

Building a "Xenstore stub domain" with MirageOS

By Dave Scott - 2012-09-12

[ Due to continuing development, some of the details in this blog post are now out-of-date. It is archived here. ]

On all hosts running Xen, there is a critical service called xenstore. Xenstore is used to allow untrusted user VMs to communicate with trusted system VMs, so that

  • virtual disk and network connections can be established
  • performance statistics and OS version information can be shared
  • VMs can be remotely power-cycled, suspended, resumed, snapshotted and migrated.

If the xenstore service fails then at best the host cannot be controlled (i.e. no VM start or shutdown) and at worst VM isolation is compromised since an untrusted VM will be able to gain unauthorised access to disks or networks. This blog post examines how to disaggregate xenstore from the monolithic domain 0, and run it as an independent stub domain.

Recently in the Xen community, Daniel De Graaf and Alex Zeffertt have added support for xenstore stub domains where the xenstore service is run directly as an OS kernel in its own isolated VM. In the world of Xen, a running VM is a "domain" and a "stub" implies a single-purpose OS image rather than a general-purpose machine. Previously if something bad happened in "domain 0" (the privileged general-purpose OS where xenstore traditionally runs) such as an out-of-memory event or a performance problem, then the critical xenstore process might become unusable or fail altogether. Instead if xenstore is run as a "stub domain" then it is immune to such problems in domain 0. In fact, it will even allow us to reboot domain 0 in future (along with all other privileged domains) without incurring any VM downtime during the reset!

The new code in xen-unstable.hg lays the necessary groundwork (Xen and domain 0 kernel changes) and ports the original C xenstored to run as a stub domain.

Meanwhile, thanks to Vincent Hanquez and Thomas Gazagnaire, we also have an OCaml implementation of xenstore which, as well as the offering memory-safety, also supports a high-performance transaction engine, necessary for surviving a stressful "VM bootstorm" event on a large server in the cloud. Vincent and Thomas' code is Linux/POSIX only.

Ideally we would have the best of both worlds:

  • a fast, memory-safe xenstored written in OCaml,
  • running directly as a Xen stub domain i.e. as a specialised kernel image without Linux or POSIX

We can now do both, using Mirage! If you're saying, "that sounds great! How do I do that?" then read on...

Step 1: remove dependency on POSIX/Linux

If you read through the existing OCaml xenstored code, it becomes obvious that the main uses of POSIX APIs are for communication with clients, both Unix sockets and for a special Xen inter-domain shared memory interface. It was a fairly painless process to extract the required socket-like IO signature and turn the bulk of the server into a functor. The IO signature ended up looking approximately like:

    type t
    val read: t -> string -> int -> int -> int Lwt.t
    val write: t -> string -> int -> int -> unit Lwt.t
    val destroy: t -> unit Lwt.t

For now the dependency on Lwt is explicit but in future I'll probably make it more abstract so we can use Core Async too.

Step 2: add a Mirage Xen IO implementation

In a stub-domain all communication with other domains is via shared memory pages and "event channels". Mirage already contains extensive support for using these primitives, and uses them to create fast network and block virtual device drivers. To extend the code to cover the Xenstore stub domain case, only a few tweaks were needed to add the "server" side of a xenstore ring communication, in addition to the "client" side which was already present.

In Xen, domains share memory by a system of explicit "grants", where a client (called "frontend") tells the hypervisor to allow a server (called "backend") access to specific memory pages. Mirage already had code to create such grants, all that was missing was a few simple functions to receive grants from other domains.

These changes are all in the current mirage-platform tree.

Step 3: add a Mirage Xen "main" module and Makefile

The Mirage "main" module necessary for a stub domain looks pretty similar to the normal Unix userspace case except that it:

  • arranges to log messages via the VM console (rather than a file or the network, since a disk or network device cannot be created without a working xenstore, and it's important not to introduce a bootstrap problem here)
  • instantiates the server functor with the shared memory inter-domain IO module.

The Makefile looks like a regular Makefile, invoking ocamlbuild. The whole lot is built with OASIS with a small extension added by Anil to set a few options required for building Xen kernels rather than regular binaries.

... and it all works!

The code is in two separate repositories:

  • ocaml-xenstore: contains all the generic stuff
  • ocaml-xenstore-xen: contains the unix userspace and xen stub domain IO modules and "main" functions
  • (optional) To regenerate the OASIS file, grab the add-xen branch from this OASIS fork.

Example build instructions

If you want to try building it yourself, try the following on a modern 64-bit OS. I've tested these instructions on a fresh install of Debian Wheezy.

First install OCaml and the usual build tools:

    apt-get install ocaml build-essential git curl rsync

Then install the OCamlPro opam package manager to simplify the installation of extra packages

    git clone https://github.com/OCamlPro/opam.git
    cd opam
    make
    make install
    cd ..

Initialise OPAM with the default packages:

    opam --yes init
    eval `opam config -env`

Add the "mirage" development package source (this step will not be needed once the package definitions are upstreamed)

    opam remote -add dev https://github.com/mirage/opam-repo-dev

Switch to the special "mirage" version of the OCaml compiler

    opam --yes switch -install 3.12.1+mirage-xen
    opam --yes switch 3.12.1+mirage-xen
    eval `opam config -env`

Install the generic Xenstore protocol libraries

    opam --yes install xenstore

Install the Mirage development libraries

    opam --yes install mirage

If this fails with "+ runtime/dietlibc/lib/atof.c:1: sorry, unimplemented: 64-bit mode not compiled in" it means you need a 64-bit build environment. Next, clone the xen stubdom tree

    git clone https://github.com/djs55/ocaml-xenstore-xen

Build the Xen stubdom

    cd ocaml-xenstore-xen
    make

The binary now lives in xen/_build/src/server_xen.xen

Deploying on a Xen system

Running a stub Xenstored is a little tricky because it depends on the latest and greatest Xen and Linux PVops kernel. In the future it'll become much easier (and probably the default) but for now you need the following:

  • xen-4.2 with XSM (Xen Security Modules) turned on
  • A XSM/FLASK policy which allows the stubdom to call the "domctl getdomaininfo". For the moment it's safe to skip this step with the caveat that xenstored will leak connections when domains die.
  • a Xen-4.2-compatible toolstack (either the bundled xl/libxl or xapi with some patches)
  • Linux-3.5 PVops domain 0 kernel
  • the domain builder binary init-xenstore-domain from xen-4.2/tools/xenstore.

To turn the stub xenstored on, you need to edit whichever init.d script is currently starting xenstore and modify it to call

    init-xenstore-domain /path/to/server_xen.xen 256 flask_label