MirageOS Spring 2019 hack retreat roundup

Early March 2019, 31 MirageOS hackers gathered again in Marrakesh for our bi-annual hack retreat. We'd like to thank our amazing hosts, and everyone who participated on-site or remotely, and especially those who wrote up their experiences.

On this retreat, we ate our own dogfood, and used our MirageOS DHCP, recursive DNS resolver, and CalDAV unikernels as isolated virtual machines running on a PC Engines APU with FreeBSD as host system. The CalDAV server persisted its data in a git repository on the host system, using the raw git protocol for communication, the smart HTTP protocol could have been used as well. Lynxis wrote a detailed blog post about our uplink situation.

Lots of interesting discussions took place, code was developed, knowledge was exchanged, and issues were solved while we enjoyed the sun and the Moroccan food. The following list is not exhaustive, but gives an overview what was pushed forward.

Imagelib

Imagelib is a library that can parse several image formats, and eye-of-mirage uses it to display those images in a framebuffer.

During the retreat, imagelib was extended with support for the BMP format, it's build system was revised to split off Unix-dependent functionality, and preliminary support for the GIF format was implemented.

ActivityPub

ActivityPub is an open, decentralized social networking protocol, as used by mastodon. It provides a client/server API for creating, updating, and deleting content, and a federated server-to-server API for notifications and content delivery. During the retreat, an initial prototype of a protocol implementation was drafted.

opam

Opam, the OCaml package manager, was extended in several directions:

External (OS package system) dependency integration
Interleaving download with build/install actions
Generalisation of the job scheduler
JSON serialisation, including crowbar round-trip tests
Plugin evaluating (binary) reproducibility of opam packages
some smaller cleanup PRs (return values, locking code)

marracheck

Work was started on a new utility to install as many opam packages as possible on a machine (there just wasn't enough choice with opam-builder, opamcheck and opam-check-all). It uses opam-lib and Z3 to accomplish this.

Conex

Conex is used for signing community repositories, esp. the opam-repository. Any opam package author can cryptographically sign their package releases, and users can verify that the downloaded tarball and build instructions are identical to what the author intended.

Conex has been developed since 2015, but is not yet widely deployed. We extended opam-publish to invoke the conex_targets utility and sign before opening a pull request on the opam-repository.

SMTP

The simple mail transfer protocol is an Internet standard for sending and receiving eMail. Our OCaml implementation has been improved, and it is possible to send eMails from OCaml code now.

HTTP2

The hypertext transfer protocol is an Internet standard widely used for browsing the world wide web. HTTP 1.1 is a line-based protocol which was specified 20 years ago. HTTP2 is an attempt to fix various shortcomings, and uses a binary protocol with multiplexing, priorities, etc. An OCaml implementation of HTTP2 has been actively worked on in Marrakesh.

Irmin

Irmin is a distributed database that follows the same design principles as git. Soon, Irmin 2.0 will be released, which includes GraphQL, HTTP, chunk support, and can use the git protocol for interoperability. Irmin provides a key-value interface for MirageOS.

OCaml compiler

Some hints on type errors for int literals and int operators were developed and merged to the OCaml compiler.

# 1.5 +. 2;;
         ^
Error: This expression has type int but an expression was expected of type
         float
       Hint: Did you mean `2.'?

# 1.5 + 2.;;
  ^^^ ^
Error: This expression has type float but an expression was expected of type
         int
Line 1, characters 4-5:
  Hint: Did you mean to use `+.'?

Also, the whole program dead code elimination PR was rebased onto trunk.

BGP / lazy trie

The mrt-format library which can parse multi-threaded routing toolkit traces, has been adapted to the modern OCaml ecosystem. The border gateway protocol (BGP) library was slightly updated, one of its dependencies, lazy-trie was adapted to the modern ecosystem as well.

Xen PVH

Xen provides several modes for virtualization. MirageOS's first non-Unix target was the para-virtualized (PV) mode for Xen, which does not require hardware support from the hypervisor's host operating system but has a weak security profile (static mapping of addresses, large attack surface). However, PV mode provides an attractive target for unikernels because it provides a simple software-based abstraction for dealing with drivers and a simple memory model; this is in contrast to hardware-virtualization mode, which provides greater security but requires more work from the guest OS.

A more modern virtualization mode combining the virtues of both approaches is PVH (formerly referred to as HVMLite), which is not yet supported by MirageOS. Marek Marczykowski-Górecki from the QubesOS project visited to help us bring PVH support to the unikraft project, a common platform for building unikernels which we hope to use for MirageOS's Xen support in the future.

During the retreat, lots of bugs porting MirageOS to PVH were solved. It boots and crashes now!

Learn OCaml as a unikernel

The platform learn OCaml embeds an editor, top-level, and exercises into a HTTP server, and allows students to learn OCaml, and submit solutions via the web interface, where an automated grader runs unit tests etc. to evaluate the submitted solutions. Teachers can assign mandatory exercises, and have an overview how the students are doing. Learn OCaml used to be executable only on a Unix host, but is now beeing ported into a MirageOS unikernel, executable as a standalone virtual machine.

Network device driver (ixy)

The ixy network driver supports Intel 82599 network interface cards, and is implemented in OCaml. Its performance has been improved, including several failing attempts which degraded its performance. Also, it has been integrated into the mirage tool and is usable as a mirage-net implementation.

DNS client API

Our proposed API is described here. Unix, Lwt, and MirageOS implementations are already available.

mirage-http unified HTTP API

Since we now have two HTTP servers, cohttp and httpaf, in OCaml and MirageOS available, the new interface mirage-http provides a unified interface, and also supports connection upgrades to websockets.

cstruct capabilities

We use cstruct, a wrapper around OCaml's Bigarray, quite a lot in MirageOS. Until now, cstruct is a readable and writable byte array. We used phantom types to add capabilities to the interface to distinct read-only and write-only buffers.

patch

An OCaml implementation to apply unified diffs. This code has been extracted from conex (since we found some issues in it), and still needs to be fixed.

Statistical memory profiler

Since 2016, Jacques-Henri Jourdan has been working on a statistical memory profiler for OCaml (read the OCaml 2016 paper). An Emacs user interface is available since some years. We integrated statmemprof into MirageOS unikernels using the statmemprof-mirage library, marshal the data via TCP, and provide a proxy that communicates with Emacs over a Unix domain socket, and the unikernel.

P2Pcollab

P2Pcollab is a collection of composable libraries implementing protocols for P2P collaboration. So far various P2P gossip protocols has been implemented. At this retreat the focus was on a gossip-based publish-subscribe dissemination protocol. Future plans include building P2P unikernels and adding P2P pub/sub sync functionality to Irmin.