The MirageOS Blogon building functional operating systems

Feb
23
2017
17
0

Announcing MirageOS 3.0.0

By Mindy Preston

We're excited to announce MirageOS 3.0! MirageOS is a modern, modular library operating system that allows the creation of small, secure, legacy-free services. MirageOS applications can be compiled to run as self-contained virtual machines (a few MB in size) on Xen or KVM hosts, FreeBSD's bhyve, or even as regular Unix processes (allowing access to regular debugging tools). The system libraries themselves can be reused in traditional applications, just like any other software library.

Full release notes are available on GitHub. If you're interested in getting started with MirageOS 3 right away, you might be interested in the revamped guide to getting started, a small collection of example unikernels, or the porting guide for updating Mirage 2.x unikernels to Mirage 3.

Here's a summary of the things in MirageOS 3 that we're most excited about:

Solo5

MirageOS 3.0 is the first release that integrates the solo5 targets, virtio and ukvm, fully with the mirage front-end tool. Now you can mirage configure -t ukvm, build a unikernel, and run directly with the generated ukvm-bin! We've updated the "hello world" tutorial to reflect our excitement about ukvm -- the ukvm target is considerably easier to interface with and configure than xen was, and for a lot of users this will be a clearer path toward operational deployment of unikernels.

For a lot more information on the Solo5 targets, see the earlier blog post announcing solo5, Unikernel Monitors: Extending Minimalism Outside of the Box, and the very readable solo5 repository README. You can also read how to run solo5 unikernels on FreeBSD via bhyve.

Playing More Nicely with OPAM

MirageOS 3 has a much richer interface for dealing with the package manager and external library dependencies. A user can now specify a version or range of versions for a package dependency, and the mirage front-end tool will construct a custom opam file including both those package dependencies and the ones automatically generated from mirage configure. mirage will also consider version constraints for its own packages -- from now on, opam should notice that releases of mirage are incompatible with your unikernel.

For more information on dealing with packages and dependencies, the documentation for the Functoria.package function will likely be of use. The PRNG device-usage example in mirage-skeleton demonstrates some useful invocations of package.

Amazing Docs

Thanks to a lot of hard work, a fully interlinked set of module documentation is now automatically generated by odig and available for your reading pleasure at the MirageOS central documentation repository. While documentation was previously available for most modules, it was scattershot and often required having several disconnected pages open simultaneously. We hope you'll find the new organization more convenient. The documentation generation system is still in beta, so please report issues upstream if you run across rendering issues or have other feedback.

Result-y Errors

The module types provided by MirageOS 3 replace the previous error paradigm (a combination of exceptions and directly returning polymorphic variants) with one that uses the Result module included in OCaml 4.03 and up. A notable exception is when problems occur during the unikernel's initialization (i.e., in connect functions), where unikernels will now fail hard as soon as they can. The goal of these changes is to surface errors when the application cares about them, and to not present any uninitialized or unstable state to an application at start time.

The MirageOS 3 module types define a core set of likely errors for each module type (see the mirage-flow module type for an example), which can be extended by any given implementation. Module types now specify that each implementation must include a pretty-printer that can handle all emitted error types. Functions that return a success type when they run as expected return a (success, error) Result.t, which the caller can print with pp_error if the value is an Error.

For more background on the result type, see the Rresult library which defines further useful operations on Result.t and is used widely in MirageOS libraries. A more in-depth explanation of errors in Mirage 3 is also available.

Logs Where You Want Them

MirageOS version 2.9.0 included automatic support for logging via the Logs and Mirage_logs library, but by default logs were always printed to the console and changing the log reporter was cumbersome. In MirageOS 3, you can send logs to a consumer of syslog messages with syslog_udp, syslog_tcp, or with the full authentication and encryption provided by ocaml-tls using syslog_tls. For more information, see the excellent writeup at hannes.nqsb.io.

Disaggregated Module Type Definitions

Breaking all of the MirageOS 3.0 APIs showed us that keeping them all in the same place made updates really difficult. There's now an additional set of packages which contain the definitions for each set of module types (e.g. mirage-fs for the FS module type, mirage-block for the BLOCK module type, etc). A few module types had some additional useful code that was nicely functorized over the module type in question, so we've bundled that code in the module type packages as well. Documentation for all of the module type packages is available at the Mirage documentation hub.

We hope that this change combined with the opam workflow changes above will result in much less painful API changes in the future, as it will be possible for unikernel authors to target specific versions more readily.

Clockier Clocks, Timelier Time

In older MirageOS versions, we noticed that we were often having to deduce a span of time from having taken two wall-clock samples of the current time. In MirageOS 3, you have your choice of two types of clock - MCLOCK, which provides a monotonically increasing clock reflecting the time elapsed since the clock started, and PCLOCK, which provides a traditional POSIX wall-clock time. Most previous users of CLOCK were able to migrate to the more-honest, less-complicated MCLOCK. For an example of both clocks, see the speaking clock. You may also be interested in an example of converting existing code from CLOCK to MCLOCK.

MCLOCK provides a nice interface for dealing with time at a nanosecond granularity. The TIME module type has been updated to expect an int64 number of nanoseconds, rather than a float, as an argument to its function sleep. For those of us who don't think in nanoseconds, the Duration library provides convenient functions for translating from and to more familiar units like seconds.

Build System Shift

Mirage 3.0 has many, many more packages than before, and so we turned to OCaml Labs to help us to scale up our package management. In many but not all MirageOS packages, we've replaced oasis with topkg, the "transitory OCaml software packager". topkg is a lighter layer over the underlying ocamlbuild. Using topkg has allowed us to remove several thousand lines of autogenerated code across the MirageOS package universe, and let our release manager automate a significant amount of the MirageOS 3 release process. We hope to continue benefitting from the ease of using topkg and topkg-care.

Not all packages are using topkg yet -- if you see one that isn't, feel free to submit a pull request!

Less Code, Better Behavior

There's more in MirageOS 3 than we can fit in one blog post without our eyes glazing over. The release notes for mirage version 3.0.0 are a nice summary, but you might also be interested in the full accounting of changes for every package released as a part of the MirageOS 3 effort; links for each library are available at the end of this post.

Across the package universe, a net several thousand lines of code were removed as part of MirageOS 3. Many were autogenerated build-time support files removed in the transition from oasis to topkg. Others were small support modules like Result, which had previously been replicated in many places and were replaced by a reference to a common implementation. Some large implementations (like the DHCP client code in mirage-tcpip) were replaced by smaller, better implementations in common libraries (like charrua-core).

For example, ocaml-fat had 1,280 additions and 10,265 deletions for a net of -8,985 lines of code; version 0.12.0 jettisoned a custom in-memory block device in favor of using the in-memory block device provided by Mirage_block_lwt.Mem, removed several thousand lines of autogenerated OASIS code, removed several custom error-case polymorphic variants, and lost a custom result module. The mirage repository itself netted -8,490 lines of code while adding all of the features above!

A number of improvements were made to mirage to limit the number of unnecessary build artifacts and reduce the amount of unnecessary code linked into unikernels. Modules you're unlikely to use like Str are no longer included in the OCaml runtime. MirageOS 3 is also the first to drop support for OCaml 4.02.3, meaning that all supported compilers support the flambda compiler extension and a number of related optimization opportunities.

Very many people were involved in making the MirageOS package universe smaller and better than it was before. We'd like to thank, in a particular alphabetical order, the following people who contributed code, suggestions, bug reports, comments, mailing lists questions and answers, and other miscellaneous help:

  • Aaron Cornelius
  • Amir Chaudhry
  • Andrew Stuart
  • Anil Madhavapeddy
  • Ashish Agarwal
  • Balraj Singh
  • Cedric Cellier
  • Christiano Haesbaert
  • Daniel Bünzli
  • Dan Williams
  • Dave Scott
  • David Kaloper
  • David Sheets
  • Enguerrand Decorne
  • Eugene Bagdasaryan
  • Federico Gimenez
  • Gabriel de Perthuis
  • Gabriel Jaldon
  • Gabriel Radanne
  • Gemma Gordon
  • Hannes Mehnert
  • Ian Campbell
  • Jochen Bartl
  • John P. McDermott
  • Jon Ludlam
  • Kia
  • Leo White
  • Leonid Rozenberg
  • Liang Wang
  • Madhuri Yechuri
  • Magnus Skjegstad
  • Martin Lucina
  • Matt Gray
  • Mindy Preston
  • Nick Betteridge
  • Nicolas Ojeda Bar
  • Nik Sultana
  • Pablo Polvorin
  • Petter A. Urkedal
  • Qi LI
  • Ramana Venkata
  • Ricardo Koller
  • Richard Mortier
  • Rudi Grinberg
  • Sean Grove
  • Takayuki Imada
  • Thomas Gazagnaire
  • Thomas Leonard
  • Vincent Bernardoff
  • Vittorio Cozzolino
  • GitHub user waldyrious
  • Wassim Haddad
  • Jeremy Yallop

Please let us know if you notice someone (including yourself) is missing so we can add them and apologize! We're happy to remove or change your listed name if you'd prefer as well. Names were taken from metadata on commit messages and e-mail headers.