Our first Cambridge-based MirageOS hackathon took place yesterday - and what a fantastic day it was! The torrential rain may have halted our punting plans, but it didn't stop progress in the Old Library! Darwin College was a fantastic venue, complete with private islands linked by picturesque wooden bridges and an unwavering wifi connection.
People naturally formed groups to work on similar projects, and we had a handful of brand new users keen to get started with OCaml and Mirage. The major tasks that emerged were:
- new hypervisor target: the integration of the Solo5 KVM-based hypervisor backend, bringing the number of officially supported targets up to 3 (Xen, Unix and KVM)
- build system template: establishing a new topkg template for MirageOS libraries, to prepare us for building a unified API documentation bundle that works across all the entire project.
- CPU portability: improving ARM support via a better base OS image.
- libraries breadth: hacking on all the things to fill in the blanks, such as btree support for bare-metal Irmin, or a peer-to-peer layer for the DataKit.
We'll write about all of this in more detail, but for now here are the hackathon notes hot off the press...
Solo5/MirageOS integration (KVM-based backend)
Progress on the Solo5 project has been steaming ahead since January, and this was the perfect opportunity to get everyone together to plan its integration with MirageOS. Dan Williams from IBM Research flew over to join us for the week, and Martin Lucina headed to Cambridge to prepare for the upstreaming of the recent Solo5 work. This included deciding on naming and ownership of the repositories, detailing the relationships between repositories and getting ready to publish the mirage-solo5 packages to OPAM. Mindy Preston, our MirageOS 3.0 release manager, and Anil Madhavapeddy and Thomas Gazagnaire (OPAM minions) were on hand to help plan this smoothly.
See their updates from the day on Canopy and related blog posts:
- Introducing Solo 5
- Unikernel Monitors HotCloud 2016 paper and slides
- upstreaming GitHub issue and FreeBSD support tracking issue from Hannes Mehnert.
Onboarding new MirageOS/OCaml users
Our tutorials and onboarding guides really needed a facelift and an update, so Gemma Gordon spent the morning with some of our new users to observe their installation process and tried to pinpoint blockers and areas of misunderstanding. Providing the simple, concise instructions needed in a guide together with alternatives for every possible system and version requirement is a tricky combination to get right, but we made some changes to the installation guide that we hope will help. The next task is to do the same for our other popular tutorials, reconfigure the layout for easy reading and centralise the information as much as possible between the OPAM, MirageOS and OCaml guides. Thank you to Marwan Aljubeh for his insight into this process.
Other industrial users are also steaming ahead with their own MirageOS deployments. Amir Chaudhry spent the hackathon blogging about NFV Platforms with MirageOS unikernels, which details how Ericsson Silicon Valley has been using MirageOS to build lightweight routing kernels.
Thomas Gazagnaire was frenetically converting
mirage-console to use topkg, and the feedback prompted fixes and a new release from Daniel Buenzli.
ARM and Cubieboards
Ian Campbell implemented a (slightly hacky) way to get Alpine Linux onto some Cubieboard2 boxes and provided notes on his process, including how to tailor the base for KVM and Xen respectively.
MirageOS 3.0 API changes
Our MirageOS release manager, Mindy Preston, was on hand to talk with everyone about their PRs in preparation for the 3.0 release along with some patches for deprecating out of date code. There has been a lot of discussion on the development list. One focus was to address time handling properly in the interfaces: Matthew Gray came up from London to finish up his extensive revision of the CLOCK interface, and Hannes developed a new duration library to handle time unit conversions safely and get rid of the need for floating point handling. We are aiming to minimise the dependency on floating point handling in external interfaces to simplify compilation to very embedded hardware that only has soft floats (particularly for something as ubiquitous as time handling).
Thomas Leonard continued with the work he started in Marrakech by updating the error reporting patches (also here) to work with the latest version of MirageOS (which has a different logging system based on Daniel Buenzlis Logs). See the original post for more details.
Ctypes 0.7.0 release
Jeremy released the foreign function interface library Ctypes 0.7.0 which, along with bug fixes, adds the following features:
- Support for bytecode-only architectures (#410)
- A new
sinttype corresponding to a full-range C integer and updated errno support for its use (#411)
See the full changelog online.
P2P key-value store over DataKit
KC Sivaramakrishnan and Philip Dexter took on the challenge of grabbing the Docker DataKit release and started building a distributed key-value store that features flexible JSON synching and merging. Their raw notes are in a Gist -- get in touch with them if you want to help hack on the sync system backed by Git.
Developer experience improvements
The OCaml Labs undergraduate interns are spending their summers working on user improvements and CI logs with MirageOS, and used the time at the hackathon to focus on these issues.
Ciaran Lawlor is working on an editor implementation, specifically getting the IOcaml kernel working with the Hydrogen plugin for the Atom editor. This will allow developers to run OCaml code directly in Atom, and eventually interactively build unikernels!
Joel Jakubovic used Angstrom (a fast parser combinator library developed by Spiros Eliopoulos) to ANSI escape codes, usually displayed as colours and styles into HTML for use in viewing CI logs.
Most of the Mirage libraries already work on Windows thanks to lots of work in the wider OCaml community, but other features don't have full support yet.
Dave Scott from Docker worked on ocaml-wpcap: a ctypes binding to the Windows winpcap.dll which lets OCaml programs send and receive ethernet frames on Windows. The ocaml-wpcap library will hopefully let us run the Mirage TCP/IP stack and all the networking applications too.
David Allsopp continued his OPAM-Windows support by fine-tuning the 80 native Windows OCaml versions - these will hopefully form part of OPAM 2.0. As it turns out, he's not the only person still interested in being able to run OCaml 3.07...if you are, get in touch!
General Libraries and utilities
Olivier Nicole is working on an implementation of macros in OCaml and started working on the
HTML and XML templates using this system. The objective is to have the same
behaviour as the
Pa_tyxml syntax extension, but in a type-safe and more
maintainable way without requiring PPX extensions. This project could be
contributed to the development of Ocsigen once implemented.
Nick Betteridge teamed up with Dave Scott to look at using ocaml-btree as a backend for Irmin/xen and spent the day looking at different approaches.
Anil Madhavapeddy built a Docker wrapper for the CI system and spun up a big cluster to run OPAM bulk builds. Several small utilities like jsontee and an immutable log collection server and bulk build scripts will be released in the next few weeks once the builds are running stably, and be re-usable by other OPAM-based projects to use for their own tests.
Christophe Troestler is spending a month at OCaml Labs in Cambridge this summer, and spent the hack day working on implementing a library to allow seamless application switching from HTTP to FastCGI. Christophe has initiated work on a client and server for this protocol using CoHTTP so that it is unikernel-friendly.