This post is the story of how we got from 285 MB down to 4 MB and made the resulting bundle compose cleanly with the in-browser toplevel, so the lecture’s Await_capsule form works end-to-end in the cell at the bottom of this post. Most of the work happened on a branch of ocsigen/js_of_ocaml, with a smaller piece in art-w/x-ocaml, the WebComponent that powers the cells.

Why bundle size matters

I teach two OCaml-heavy courses at IIT Madras: CS3100, the undergraduate functional programming course, and CS6868, the more recent graduate course on language abstractions for parallelism. The lecture notes, examples and homework for both would be much more useful as interactive books that a student can read, edit and run entirely client-side, with no local installation. The same shape would help us when we run hands-on OCaml and OxCaml workshops, where the first session routinely gets eaten by the installation hump: getting opam, the compiler and the required libraries working on every attendee’s machine over patchy conference WiFi, before the teaching can begin.

The broader effort to make installation painless is the OCaml Platform roadmap, which we have been working on at Tarides as a “zero to OCaml in one click” experience. That roadmap targets a developer who wants a real local toolchain, with the full editor, debugger and project-management story, and a generous latency budget since this is a one-time setup. A workshop attendee has a much narrower target: just enough OCaml to complete the exercises in front of them. The client-side x-ocaml toplevel fits that target naturally, because everything ships as static assets and there is no installation step. The bundle, in this setting, is the latency budget: 285 MB makes the in-browser path unshippable, 4 MB makes it a realistic alternative to a local toolchain for a 90-minute session.

Why 285 MB?

The recipe x-ocaml already had for “load extra libraries into a running in-browser toplevel” goes like this. For each cma you want to ship, run

$ js_of_ocaml --toplevel <library>.cma -o lib.js

then concatenate the per-cma outputs into a single bundle and load it via <script src-load=...>. Each per-cma output is kind=cma: it registers the cma’s modules into the existing toplevel without clobbering anything, the modules light up, and you can open them from a cell. This works, and it is what the previous two OxCaml posts already use.

The trouble is that dead code elimination runs one cma at a time. If you ship base, you ship all of base, because the per-cma DCE pass never gets to look at the linked-together program and notice that the await library you actually want only touches a small slice of sexplib0. So the bundle for the closure await + capsule + basement ends up being the union of every cma in the transitive dependency, fully linked, which comes out to roughly 285 MB after the OxCaml compiler’s normal optimisations and before any of the JavaScript-side cleverness has had a chance to run.

In other words, the await-based blessed API is unshippable not because the bundling tooling is broken, but because per-cma DCE is the wrong granularity for this problem.

The other recipe, and why it doesn’t compose

It turns out js_of_ocaml already has a second recipe that does perform cross-cma DCE, which I learned about when Ricky Vetter pointed me at the --export route on X. The recipe has two steps:

1. Build a single bytecode that links every library you want, with -linkall so nothing gets pruned at the bytecode level.
2. Hand that single bytecode to js_of_ocaml --toplevel --export units.txt. The export list names the compilation units that should remain visible to the toplevel; everything else is fair game for DCE on the unified intermediate representation.

For the same await + capsule + basement set, this recipe produces a 4 MB bundle, almost two orders of magnitude smaller than the per-cma concatenation. The link-step DCE works at function granularity over the whole linked program, so the unused parts of sexplib, base, base_quickcheck and the rest of the dependency closure get pruned away cleanly.

So why aren’t we already using this recipe?

Because the artifact that comes out the other end is kind=exe, that is, a self-contained executable rather than a library. When you load such a bundle into a Web Worker that already has a running toplevel, its initialisation runs in caml_main style and starts by overwriting the host’s globals:

caml_global_data.symbols    = <bundle's symbol table>
caml_global_data.sections   = <bundle's bytecode sections>
caml_global_data.prim_count = <bundle's primitive count>
caml_global_data.aliases    = <bundle's alias table>

Those four assignments overwrite the host toplevel’s tables. After the bundle loads, caml_get_global_data().symbols is the bundle’s symbols, not the worker’s, and anything in the host toplevel that does name-based symbol resolution (Toploop, hover-types lookup, open Stdlib) now consults a table that does not know about the modules the worker had already linked. The toplevel survives, but its typing environment is wrong, and cells stop being able to open anything. We hit this dead end in the capsules post; the bundle was correct and the sizes were great, but the integration step that connects an exe-shaped bundle to an existing toplevel was the missing piece.

The patch

The fix is a flag, --toplevel-extend, that I added on a branch of ocsigen/js_of_ocaml. When it is set, js_of_ocaml --toplevel --export … emits exactly the same DCE output as before, with the same bytes for the registered modules and the same .cmi files embedded under /static/cmis/, but with three small changes:

• packed as ~standalone:false, so there is no globalThis polyfill IIFE wrapping the output,
• with the four caml_js_set writes to caml_global_data from above skipped, and
• tagged as kind=cma in the buildInfo header.

The bundle’s modules still register themselves on load via the ordinary caml_register_global(n, v, name), which the runtime correctly merges via symidx into the host’s existing tables. The result is additive, not destructive: the host toplevel’s symbol table, sections, primitives and aliases all survive intact, and its typing environment continues to resolve Stdlib, Toploop and everything else that was already linked. The new modules from the bundle simply show up as new symbols on top.

The initial diff is small: parse_bytecode.ml gates the caml_js_set block on the new flag, and cmd_arg.ml/compile.ml thread it through. That gets the bundle composable. The actual debugging is what came afterwards, when we tried to make the composed bundle behave the way the host expected, and that is what the rest of this post is about.

Wiring it through x-ocaml

On the x-ocaml side, the --dce flag drives the single-bytecode + --export build, invoking js_of_ocaml --toplevel-extend --export units.txt for the bundles we ship. The oxcaml branch of my x-ocaml fork carries the change; the patched js_of_ocaml needs to be on PATH when the build runs.

Numbers

For the full closure of libraries the lecture’s gensym example uses, the two paths come out very differently:

The basement + capsule0 row is essentially a wash, because the bundle size at that scale is dominated by the .cmi files and there is very little JavaScript code for DCE to chew on. Once await and the curated capsule API enter the picture the per-cma path balloons, because it has to ship every cma in the transitive closure of base, sexplib0, bin_prot, base_quickcheck and the ppx_* runtime libraries, while --dce keeps only the functions that are actually reachable from the export list, plus the .cmi files the toplevel needs to elaborate the signatures the user types into a cell.

A few more snags

Getting the bundle to be kind=cma was the easy part. Composing it with an already-running toplevel turned out to surface a small zoo of follow-on issues, each of which had a short fix once we understood it. They all live on the same kc-toplevel-extend branch; the commit messages have the gory details.

• Predefined-exception identity drifts across bundles. The bundle’s re-allocated Not_found, Sys_error and friends are physically distinct from the host’s copies, so a try ... with Not_found -> in stdlib code (the first place we hit this was Hashtbl.randomized_default reading OCAMLRUNPARAM) fails to catch the host’s Not_found raised by the runtime. The fix is to bind each predef-exn variable in the bundle to a runtime caml_get_global_data lookup so the bundle reuses the host’s instances.

• The pseudo-filesystem raises on duplicate .cmi registrations. The bundle wants to re-emit /static/cmis/stdlib.cmi, which the host has already registered at boot, and MlFakeDevice.register refuses to overwrite. Making register idempotent removes the conflict without losing anything: the two copies of stdlib.cmi agree, since they come from the same opam switch.

• Stdlib re-registration overwrites host modules. Without a guard, caml_register_global was cheerfully replacing the host’s caml_global_data["Format"] (and every other stdlib module the bundle’s bytecode happens to link in) with the bundle’s freshly initialised copy. Adding an early return when the name is already known fixes this without changing the behaviour for any name the host does not yet have.

• Domain.DLS slot collisions silently broke hover types. The bundle re-runs stdlib’s module init when it loads, and Stdlib__Domain.DLS’s let key_counter = Atomic.make 0 re-allocates the counter and starts it from zero. The bundle’s Format.stdbuf_key then ends up at a low DLS index the host had already assigned to its Format.stdbuf_key, and DLS.set overwrites the host’s entry in the shared caml_domain_dls array. The host’s Format.flush_str_formatter then reads from the bundle’s empty buffer, merlin’s type-enclosing printer (which flushes through Format.str_formatter) returns "" for every query, and hover-on-identifier tooltips come up blank. The fix is at the bundle-load boundary in build_portable_js_extend.sh: snapshot caml_domain_dls_get () before the bundle’s IIFE runs, and restore the host-owned slots afterwards. The bundle’s new high-index slots are left alone; only the host’s previously populated slots get restored. I spent a while convinced this was a Format DCE bug before tracing through the OCaml 5+ Domain.DLS init path.

• Bundle build: the curated capsule and await.capsule APIs both open! Base at the top of their files, so their interfaces mention Base.unit rather than Stdlib.unit. To elaborate those signatures the host toplevel needs a small chain of base and sexplib0 .cmi files at /static/cmis/. We ship three base cmis (base.cmi, base__.cmi, base__Unit.cmi) and two sexplib0 cmis via js_of_ocaml’s --file=<src>:/static/cmis/ flag, which embeds the file directly without putting base on the bytecode-link line (which would drag the whole of base back in).

The cell

The cell below uses the lecture’s gensym_capsule.ml shape directly: Await_capsule.Mutex.with_lock taking an Await_kernel.Await.t witness, with Capsule_expert.Data.create and Capsule_expert.Data.unwrap for the brand-locked counter. No [@@@alert "-deprecated"], no shim. Press Run.

We cannot actually demonstrate parallelism in the browser worker (it is single-domain), so we hand the body the trivial blocker Await_blocking.await Terminator.never. The mode-checking is the same brand/local/once dance as in the capsules post; what is new is that the same dance now type-checks against the blessed Await_capsule API directly, without the Capsule_blocking_sync shim.

What next?

The fork is small enough that --toplevel-extend is plausibly upstreamable into ocsigen/js_of_ocaml. The DLS-snapshot dance would also be cleaner as a proper runtime primitive than as a wrapper around the bundle’s IIFE. The branch is open for review.

For x-ocaml itself, the obvious next step is to broaden the set of extension bundles. The current portable.js covers basement + capsule0 + capsule + await + portable, which is just enough for the capsule and parallelism material in CS6868; the CS3100 material would want a different slice, and an async/Eio-flavoured bundle would let the same cells host concurrency examples. Once a small library of these bundles exists, turning a lecture set or a workshop tutorial into an interactive book becomes mostly a matter of picking the right bundle, which is the workshop-scale “zero to OxCaml” story I want to get to.

x-ocaml is one of Arthur Wendling’s hacking expeditions, and it remains a pleasure to build on. Thanks to Ricky Vetter for the --export pointer that got the whole thing started, and to the OxCaml team for the libraries.

Written together with Claude Opus 4.7 (1M context). The js_of_ocaml diff against the +ox base and the x-ocaml integration diff are both on GitHub.

The example here is the capsule version of gensym from my CS6868 lecture (handout). Capsules don’t introduce a new mode axis; they’re a small library that composes the axes we’ve already met: contention and portability, uniqueness, and linearity. The cells below run in the same in-browser OxCaml toplevel as the previous post; the A note on the API section at the end explains why we use a slightly older flavour of the capsule API (it’s a bundle-size thing, not pedagogical).

Why atomics aren’t enough

For a single integer counter, atomics are perfect. Drop the size constraint and they’re not. Suppose gensym had to consult a hash table of already-issued names, or update two counters in lockstep, or do a multi-step modify-then-record. None of that fits an atomic word. The standard OCaml answer is Mutex.t:

let mutex = Mutex.create ()
let table = Hashtbl.create 16

let safe_insert k v =
  Mutex.lock mutex;
  Hashtbl.add table k v;
  Mutex.unlock mutex

(* Nothing stops you from forgetting to lock: *)
let unsafe_insert k v =
  Hashtbl.add table k v   (* races, but compiles. *)

Two things are wrong here. First, the mutex and the data are separate values; nothing at the type level connects mutex to table. Second, “always lock before access” is a programmer convention. The compiler can’t tell what mutex you meant for what state, and an unlocked Hashtbl.add is a runtime data race that nothing catches.

Yes, OxCaml’s contention rule from the previous post would reject direct mutation of a @ contended table in a parallel context. But that only helps if the type system can see that table is contended in the first place. Once you have a top-level shared mutable, you need a structural reason for the type checker to believe that “you are holding the lock right now.” That’s exactly what a capsule provides.

Gensym in a capsule

The capsule pattern packs the counter inside a brand-locked container, making the lock the only handle to the state:

Run it; you get two distinct ids, and the toplevel binds val next : unit -> int and val gensym : string -> string. No handle to the inner ref is in scope outside Mtx.with_lock. Let’s read what’s making that true.

The brand 'k: an existential tied to a fresh capsule

Cap.create () returns a key wrapped in an existential constructor Cap.Key.P:

type packed = P : 'k Key.t -> packed [@@unboxed]
val create : unit -> packed

The let Cap.Key.P key = Cap.create () pattern unwraps it and introduces a fresh type variable, call it $k, in the surrounding scope. key is then bound at type $k Cap.Key.t. A second Cap.create () would introduce $k2, distinct and unrelated. Each P pattern brings a new type into the world.

That brand is the static glue between capsule, data, and mutex. When we call Cap.Data.create (fun () -> ref 0) next, the compiler unifies the data’s type parameter with the brand of the key it’ll eventually be unwrapped under, so counter : (int ref, $k) Cap.Data.t. The mutex created from key (consuming it as @ unique) is also branded $k. Henceforth this data only opens under this mutex’s lock.

(Why is the existential pattern wrapped inside make_counter () rather than at the top level? OCaml refuses top-level let-bindings that introduce existentials, so we hide the unwrap inside a function. The closure returned by make_counter carries the $k-branded mutex and counter in its environment.)

The bare ref is unreachable from outside the capsule

Look at where the ref 0 lives:

let counter = Cap.Data.create (fun () -> ref 0)

That ref has no binding outside the closure handed to Cap.Data.create. The only handle out is the Cap.Data.t value branded $k. There is no aliased reference to smuggle out: the ref’s reachability is single-pathed. This is exactly the configuration the uniqueness post was about, a unique reference with no aliases to it, but here we get it for free from how the API is shaped, without writing @ unique anywhere ourselves. The capsule API is exploiting uniqueness as a construction principle.

Cap.Data.t mode-crosses contention and portability

Just like Portable.Atomic.t from the previous post, the type ('a, 'k) Cap.Data.t carries the kind annotation value mod contended portable. A closure that captures a capsule value stays @ portable, and any domain may hold the capsule. So the closure returned by make_counter (which captures both mutex and counter) type-checks at portable mode and is safe to send to another domain.

This is the same mode-crossing move we saw with Portable.Atomic, just applied to a more general container.

with_lock produces an access token, briefly

Mtx.with_lock is the only way into the critical section. Its signature:

val with_lock
  : 'k t
  -> f:('k Cap.Password.t @ local -> 'a @ once unique) @ local once
  -> 'a @ once unique

Three different things in this signature are doing real work.

• Brand 'k: the password’s brand matches the mutex’s brand. A password from a different mutex (different $k1) cannot unwrap our counter. The type checker refuses to unify the two existentials and the program does not compile.
• @ local: the password is local to the body’s region. It cannot escape f by being returned, stored in a global, or stuffed into a closure that outlives the call. When with_lock returns the lock is released, and there’s no password left in scope to attempt to touch the data with. (Locality is the stack-allocation / no-escape axis we covered; same mechanism, different use.)
• @ once on the body: f can be called at most once. This is the linearity guarantee. The body cannot be re-entered with the same password, and the runtime cannot smuggle it out for later. Each with_lock use is a single shot.

Inside f, Cap.access ~password ~f:(fun access -> ...) converts the password into a brand-matching 'k Access.t for the inner body. Cap.Data.unwrap ~access counter then accepts the access and returns the underlying int ref at @ uncontended. We hold the lock; no other domain can be touching the data. The next two lines, incr c; !c, are ordinary OCaml mutation; the contention rule is satisfied by construction because unwrap promised uncontended access.

Outside the lock body, neither password nor access exists in scope, so there is no path from the outer program to incr c that doesn’t go through with_lock.

Brand mismatch is a type error

The cell below tries the genuinely unsound case: protecting one Cap.Data.t with two distinct mutexes. If this compiled, two threads holding distinct mutexes could enter critical sections on the same data simultaneously. The type checker refuses, and reports the two existentials don’t match.

The error names the two existentials and points out they cannot unify. The handout’s capsule_abuse.ml walks through two more attempted escape hatches:

• Leak the inner ref through a top-level Portable.Atomic. The store compiles, but anything inside a portable atomic is @ contended, and the contention rule from the previous post forbids reading or writing a mutable field of a contended value. So you can park an alias, but you can’t dereference it.
• One mutex protecting two Cap.Data.t values. Allowed, and correctly so. Both data values are branded with the same $k, and a single critical section can update both. (Equivalent to one mutex guarding two fields of a struct.)

The first closes the obvious leak path (via the same uniqueness-of-paths property that protects the bare ref); brand mismatch closes the aliased-mutex case; the one-mutex-two-data case shows the rule isn’t gratuitously restrictive.

What each mode is doing

Pulling the threads together, with one line per axis:

• Contention ensures shared mutable state can’t be read or written from outside a critical section. Inside with_lock we get an @ uncontended view via unwrap; outside, the contents aren’t in scope at all.
• Portability lets the closure cross domain boundaries. Cap.Data.t mode-crosses, so the closure capturing it stays @ portable.
• Locality confines the password (and the access derived from it) to the body of with_lock. No way to keep a token past lock release.
• Linearity (@ once on the body) makes the critical section single-shot. The body cannot be re-entered with the same password.
• Uniqueness isn’t an annotation we wrote; it’s the property that the inner ref has exactly one path of access, through the capsule. The API shape forces it.

A note on the API

Two API choices in the cells above are worth flagging for anyone transcribing this to a real .ml file.

Curated vs expert capsule API. The full capsule opam library wraps the brand-based primitives in a curated layer (Capsule.Isolated, Capsule.Guard, Capsule.Shared, Capsule.Data) that hides most of the Key.P/password/access plumbing for common patterns. For most application code you reach for those first. We use Capsule_expert directly here because the curated wrapper pulls in base, sexplib0, and friends that would balloon the in-browser bundle by ~270 MB. The lecture’s Capsule.Data calls are syntactically identical to ours.

Mutex flavour. The lecture uses Await_capsule.Mutex.with_lock, the recommended (non-deprecated) primitive that hands the body an Access.t directly. Bundling the await library into the in-browser toplevel pulls in roughly 280 MB of transitive deps (sexplib, stdio, ppx_*, base.shadow_stdlib, …), so we use the deprecated-but-equivalent Capsule_blocking_sync.Mutex with the deprecation alert silenced. It hands the body a 'k Password.t @ local, which we convert to a 'k Access.t via Capsule_expert.access. One extra line of plumbing; same modes are doing the work. In a real .ml file with await available, swap Capsule_blocking_sync.Mutex for Await_capsule.Mutex and drop the Capsule_expert.access step; the lecture’s verbatim form in the handout is what you want.

What’s left

The capsule pattern is enough to put a hash table or a more elaborate shared structure under a single mutex with a compile-time guarantee that nothing touches it without the lock. To actually run gensym from two domains in parallel we need the parallel scheduler: Parallel.fork_join2 and the @ portable once story for closures crossing the fork boundary. That’s the next post.

The point of stopping here is that the hard part of safe shared mutable state in OxCaml isn’t the parallel primitive, it’s the lock discipline. And the lock discipline turns out to be a small composition of the mode axes we already had to learn anyway. The new library is small; the framework was already in place.

The examples below are adapted from the OxCaml team’s excellent Intro to Parallelism (Part 1) tutorial, by way of my recent CS6868 lecture on OxCaml (handout). The tutorial is the canonical reference and is worth reading in full; this post tries to capture the essence in a more bite-sized form, focused on the two new mode axes that together rule out data races at compile time — and on one subtlety about portability that’s easy to misread.

A quick aside: data races in OCaml are less scary

Before we dive in, it’s worth pausing on why we want to rule out data races at all in OCaml. In C, C++, or unsafe Rust, a data race is catastrophic: the standard licenses the compiler to do anything, including silent memory corruption, on the basis that your program had no defined meaning to begin with. OCaml is considerably gentler. OCaml’s memory model guarantees that even programs with races preserve type safety and memory safety — a racy program may observe weakly-consistent values across domains, but it will not crash, won’t read uninitialised memory, and won’t violate the type system’s invariants.

So a race in OCaml is a logic bug, not a runtime footgun. Why bother catching them statically, then? Because once a program is data-race free, you get sequential consistency: any observed behaviour can be explained as some interleaving of the operations of the different domains, with each domain’s own operations executed in program order. That’s still concurrent reasoning — there are many possible interleavings — but it’s the simplest model concurrent code can have, and equational reasoning, induction, and the usual program-logic tools all transfer over to each interleaving. Drop race freedom and you lose this: observed behaviours can only be justified by allowing intra-thread reorderings as well, where a single domain’s operations appear to execute out of program order to other domains. Sequential consistency is the real prize. Race freedom is what buys it.

Hello, OxCaml

A quick sanity check that your browser really is running an OxCaml toplevel. The @ local annotation is OxCaml-only syntax; on a stock OCaml parser it wouldn’t even parse.

A racy gensym

Here’s the example we’ll keep coming back to: a symbol generator that hands out distinct ids by incrementing a captured counter. Sequentially it works; the cell prints two ids. The last line tries to ship gensym to another domain, and that’s where the type checker stops us.

Run it from two domains in parallel and you’d tick every box on the four-ingredient recipe for a data race: two domains, a shared count, both writing, and count is a plain ref — not atomic. On stock OCaml 5, the compiler would happily let you ship this closure to another domain. OxCaml refuses, before anything runs. The error names two things — nonportable and portable — that aren’t in vanilla OCaml’s vocabulary. What do they mean?

Two new modes for data race freedom

OxCaml extends OCaml’s type system with several modes — annotations that describe how a value can be used, orthogonal to its type. I’ve already written about two of them on this blog: uniqueness, which tracks whether a value has at most one reference, and linearity, which tracks how often a value may be used. Today’s post is about a different pair — the two modes that, together, rule out data races at compile time:

• contention — uncontended / contended: tracks whether multiple domains can simultaneously access a value. A contended value might be in another domain’s hands right now, so the type system limits what you can do with it.
• portability — portable / nonportable: tracks whether a value can safely cross a domain boundary at all. Closures that are sent to other domains must be portable.

The pair is enough to catch the gensym race. Let’s look at each restriction in isolation, then come back to the closure.

Contention rejects mutable writes

A contended value might be mutated by another domain right now. So OxCaml refuses to let you read or write its mutable fields:

The error on the last line names exactly the rule: “This value is contended but is expected to be uncontended because its mutable field mood is being written.” Even reading a mutable field on a contended value is rejected — another domain might be mid-write at the same instant.

Portability rejects captured refs

Portability is about closures. A @ portable closure is one the compiler has verified is safe to ship to another domain, with one critical catch: every value the closure captures from its enclosing scope is treated as contended inside the closure body. A pure function that doesn’t mutate anything is trivially portable — there’s nothing the contention rule can object to:

Capturing a ref is fine on its own; mutating one that’s been captured is what falls afoul of the rule:

Read the error carefully — it tells you exactly why counter isn’t portable. The closure is @ portable, so the captured r is treated as contended inside the body. But incr r is a mutation, and writing through a ref requires it to be uncontended. The two rules collide. And now the original gensym rejection makes sense: it does exactly the same thing — mutates a captured count.

The trap, and the actual rule

Read those last two cells together and it’s tempting to draw the wrong moral: that to make a function safe to ship to another domain, you have to give up side effects. That would be far too restrictive.

The actual rule is narrower, and the difference matters.

Portability constrains what a closure captures from its enclosing scope. Captured values become contended — that’s the rule, and it’s why gensym got rejected. But a closure’s parameters are not captures. They’re handed in fresh at each call, so they can be passed at any mode the type spells out — including @ uncontended. The obligation to provide an uncontended argument shifts to whoever is doing the calling. That’s a much weaker requirement than “no side effects.”

Captured vs parameter, in code

Here’s the example that makes the distinction concrete. loop is @ portable, and its body mutates an int ref. That works because the ref is a parameter annotated @ uncontended, not something captured:

Two @ portable annotations are doing work here. The inner one says loop is shippable to another domain — that’s the interesting one, and it works because a is loop’s parameter, not a capture: when loop is eventually called from somewhere parallel, that somewhere has to prove the a it passes in is uncontended. Portability didn’t ban mutation; it pushed the proof to the call site. The outer annotation says the whole factorial_portable is portable too — it allocates a fresh a on each call and captures nothing from outside, so we can ship the whole function to another domain. The compiler verifies both annotations as part of accepting the program.

A formal aside: defaults and submoding

We’ve been writing @ contended and @ portable as if they were the interesting modes and their absence was nothing in particular. There’s actually a small lattice on each axis, with a default and a direction of “how strong” the guarantee is. The handout summarises it like this:

The relation A ⊑ B is the submoding order: a value at mode A may be used wherever mode B is expected, because A carries the stronger guarantee and B is the looser expectation. An uncontended value satisfies a @ contended parameter (we just promised the callee might let other domains touch it; if no other domain ever does, that promise is trivially compatible). A portable closure satisfies a @ nonportable slot. But not the other way around — submoding goes one way only.

The defaults are what you get when you write plain OCaml. Every value starts out uncontended — no other domain has it, so reads and writes are unrestricted. Every closure starts out nonportable — we make no claim about shipping it elsewhere. That’s why ordinary OCaml code keeps type-checking under OxCaml: defaults are the most permissive end of each axis, and you only meet the new restrictions when you (or a parallel API) ask for a stronger guarantee. The post uses only the two endpoints of the contention chain — uncontended and contended — and skips shared, which permits read-only access across domains and isn’t needed for the gensym story.

Back to gensym: the fix

The captured-vs-parameter distinction is enough to fix simple loops, but gensym is a different shape: there’s a single counter that must be shared across calls. We need a counter type that’s itself portable — something whose values mode-cross both contention and portability, so a closure capturing it stays portable. That’s exactly Portable.Atomic.t, from OxCaml’s portable library: a 'a Portable.Atomic.t is always portable and always uncontended, regardless of where it lives.

Wrap the counter and gensym in a module so the toplevel can see the whole bundle as portable, then actually do what got us in trouble at the start: ship gensym to a freshly-spawned domain. The compiler accepts it (it verified Gen is portable), and the program runs:

The toplevel reports module Gen : sig ... end @@ portable — mode inference noticed every value inside is portable and made the whole module portable, so Gen.gensym survives extraction at portable mode. You might wonder why we don’t just write let (gensym @ portable) prefix = ... at the top level, the same shape we used for factorial_portable above. That’s a quirk of the toplevel: a bare let lands in the implicit toplevel module, which itself sits at the default nonportable mode, so when Domain.Safe.spawn later reads gensym back out, it sees a nonportable binding and refuses — even though the closure was verified portable at its binding site. The factorial_portable cell got away with the simpler form only because nothing else tried to extract the binding at portable mode. Wrapping in module Gen gives the closure its own portable home, which is what lets it actually cross a domain boundary. In a real .ml file the whole compilation unit is a module that mode inference can mark portable wholesale, so this dance isn’t necessary.

(A note on the Portable.Atomic you see here: in a real program you’d get it by open Portable from the portable opam library. To keep the page weight reasonable, this notebook ships only basement — the small library that provides the actual atomic operations — and a hidden setup cell at the top of the page wraps it with the kind annotation : value mod contended portable. That kind is what tells the compiler this type mode-crosses both axes — a closure capturing one stays portable, and any domain may touch it.) The handout notes that for state more elaborate than a single atomic counter, the right answer is capsules — a structural way to bundle mutable state with its access protocol so the whole package is portable.

What’s left

The race-freedom guarantee here is independent of any test run: every rejection above happened at the same compile step an OxCaml binary on disk would go through, before any code executed. The final spawn is a demonstration, not a proof — it’s the compiler accepting the program in the first place that tells us no race is possible. Two mode axes, a kind annotation, and a clear rule about what “portable” means were enough to make data-race freedom a property the type checker enforces.

The lecture this is drawn from goes further into capsules (for shared state more elaborate than an atomic) and Parallel.fork_join2 (for structured parallelism). Material for another post.

A stronger answer, replication-aware linearizability from Wang et al. (PLDI 2019), asks for full functional equivalence with a sequential specification: the merged state should behave as if the operations everyone did had run in some sequential interleaving. RA-linearizability is what our verification work has aimed at for the past few years, and for a class of useful data types it still falls short: the ones where the state is a grow-only bag and the interesting semantics live in the read function. This post is the longer-form version of the keynote argument, plus a digest of recent work on Sal.

PaPoC 2026 was co-located with EuroSys in Edinburgh, and was a wonderful workshop. Thanks to the program chairs, Lindsey Kuper and José Orlando Pereira, for running it. A permanent entry for the keynote is on my talks page.

Convergence is not enough

The canonical example is the OR-Set. The state is two sets, an adds set and a tombstones set. Adding an element tags it with a fresh id and stores the (element, id) pair in adds. Removing an element does not delete anything; it adds a tombstone entry for each currently observed (element, id) pair to tombstones. Reading filters adds against tombstones, and merging unions both components componentwise. The data type converges, but tombstones accumulate without bound. The fix is a bit cleverer than just tagging: each replica tracks causality explicitly, by maintaining a context of operations it has observed, and the merge uses that context to recognise when an element was removed at another replica without having to keep a tombstone for it. The result is the “efficient” OR-Set.

In practice this is harder than it sounds. Riak’s riak_dt, a production CRDT library, shipped an optimisation to its OR-Set Map that broke monotonicity (issue #79). Christopher Meiklejohn, one of the authors of the Riak DT Map paper, later wrote about how easy it is to get the inflationary, deterministic, least-upper-bound conditions wrong, and noted that the team “have even got this wrong a few times” themselves. Martin Kleppmann’s 2019 PaPoC paper on collaborative-text anomalies had errata of the same flavour, appended to the publication page a few years after the fact: the non-interleaving condition proposed in §2.1 “cannot be guaranteed by any algorithm,” and the algorithm in §3.1 does not guarantee convergence.

A well-defined join and a join that does what the user expects are not the same thing, and the gap between the two is where most of these bugs live.

A short history of how we tried to close that gap

My group has been chipping at this for a few years. In Peepul (PLDI 2022) we tried to capture intent axiomatically: write the spec as a fold over a partially ordered event graph, then prove the implementation simulates it. The proofs closed, but the cost was telling. The queue MRDT case study took 1,123 lines of proof spread over 75 lemmas against a 32-line implementation, with each F* verification run taking on the order of 4,753 seconds. F*’s SMT-aided proofs were brittle (z3 upgrades broke them, and the solver only ever returned yes / no / timeout with no counterexample), and the methodology did not really scale past a handful of carefully-chosen examples.

Neem (OOPSLA 2025) replaced the axiomatic spec with RA-linearizability, with one twist: instead of taking a separate sequential specification as input, Neem treats the operational definition of do_ itself as the spec. The merge is correct if the resulting state behaves like some sequential interleaving of the updates under that definition. Neem also gave us a fixed schedule of 24 verification conditions (six on do_, twelve on merge, six on conflict resolution) that, if all closed, certify RA-linearizability. That was a real upgrade. But it was still in F*, still SMT-fragile, and there was a class of useful RDTs for which closing the 24 VCs left the correctness statement itself near-vacuous.

The pattern that exposes that vacuity is recognisable. When porting an op-based CRDT to a state-based one, a common move is to dump every operation into a grow-only set and leave the read function to reconstruct the result. The merge is then set union, any sequence of operations is trivially its own linearisation, and the 24 RA-linearizability VCs close without much work. The data type can still be wrong, because nothing in that proof has said anything about intent: which characters are bold, which key sits at the head of the queue. Peepul did try to capture intent, but it expressed the spec over a partially ordered graph of events with no further structure, and that turned out to be both an awkward medium to write specs in and a brittle one to prove against.

We have started calling these vacuous-convergence cases Tier-C RDTs, in a three-tier taxonomy that we settled on this past month:

• Tier A: state is the semantic content. LWW-Register, PN-Counter, Bounded-Counter, MAX-Map. The RA-linearizability VCs reduce to lattice arithmetic and cover what one wants to know.
• Tier B: merge does non-trivial computation. Multi-Valued-Register, LWW-Element-Set. Still substantive.
• Tier C: state is a grow-only bag, and the semantics live in the read. OR-Set, RGA, Add-Win Priority Queue, and Peritext (which the rest of the post comes back to).

Roughly half the Sal suite, and most of the RDTs people actually want to use, sit in Tier C. For Tier C, “verified” without read-side theorems is an overclaim.

What Sal actually is

Sal is the work I presented at PaPoC, joint with Pranav Ramesh and Vimala Soundarapandian. The wider line of work this builds on also involves Adarsh Kamath, Aseem Rastogi, and Kartik Nagar. It ports Neem to Lean 4 and adds a multi-modal proof tactic. The repo is at https://github.com/fplaunchpad/sal.

The Lean choice is not original to us. Ilya Sergey is the one who convinced me to take Lean seriously, and Sal’s multi-modal tactic stack is directly inspired by Loom, the multi-modal proof-orchestration layer his group has been building. Velvet, the Lean library it sits inside, verifies imperative programs; Sal verifies RDTs. The toolchain is shared.

When you write by sal in front of a verification condition, three things happen in sequence. First, dsimp + grind: pure rewriting plus Lean’s bit-level automation. If this closes the goal, the proof term is checked by the Lean kernel and the trusted base is just Lean. If it doesn’t, Sal hands the goal to Z3 via lean-blaster. Fast, but Z3 is now in the trusted base, and Sal records this with a Blaster_admit annotation so the borrowed trust is visible. If that does not close either, an AI agent (Claude Code or Aristotle) tries to write a proof term, which Lean then kernel-checks. The TCB shrinks back to just Lean.

For the LWW-Register all 24 VCs close at stage 1. The whole specification fits on a screen:

abbrev concrete_st := ℕ × ℕ
def init_st : concrete_st := (0, 0)

def lex_max (a b : ℕ × ℕ) : ℕ × ℕ :=
  if a.1 > b.1 then a
  else if b.1 > a.1 then b
  else if a.2 ≥ b.2 then a else b

def do_ (s : concrete_st) : op_t → concrete_st
  | (ts, _, .Write v) => lex_max s (ts, v)

def merge (a b : concrete_st) : concrete_st := lex_max a b

theorem merge_comm (a b : concrete_st) : eq (merge a b) (merge b a) := by sal
theorem merge_idem (s : concrete_st) : eq (merge s s) s            := by sal
-- ... 22 more, all closed by `by sal`

Across the current Sal suite (28 RDTs, 17 CRDTs and 11 MRDTs, 648 verification conditions) the breakdown from the paper was roughly 69% closed at stage 1, 28% via Z3, and 3% via AI-completed ITP. That is the “push-button” claim, and for Tier A and B it holds up well.

Two practical notes. First, Aristotle has been the workhorse at stage 3: LWW-Map, Shopping-Cart, and MAX-Map each had VCs that neither grind nor Z3 closed, and Aristotle produced kernel-checked intermediate lemmas that did. Second, on the brittleness-of-tooling worry, the suite was bumped from Lean 4.26 to 4.28 between the paper draft and PaPoC. One Shopping-Cart obligation drifted under the new grind and needed a tweak; everything else carried over. That is a different experience from the F* / z3-upgrade story above.

Tier C is where things get more involved.

The Peritext story

Peritext is a CRDT for collaborative rich-text editing, by Geoffrey Litt, Sarah Lim, Martin Kleppmann, and Peter van Hardenberg (CSCW 2022). The design has four state components (characters, parent pointers, tombstones, formatting marks). What makes the paper unusually pleasant to mechanise is the care its authors put into specifying intent. §3 walks through eight worked examples of intent preservation, each spelled out as a small concrete scenario, and appendix §A.2 catalogues them as a single table for reference.

In one, Alice bolds the entire sentence “The fox jumped” while Bob inserts the word “brown” in the middle, and the result should be The brown fox jumped. In another, working with the same sentence, Alice bolds “The fox” while Bob bolds “fox jumped”, and the overlapping word “fox” ends up bold under both. A third concerns link spans: a link has a hard right edge, so a concurrent insert at the end of the link should not expand it. And so on for the rest of the eight. The paper argues informally that the design handles each example correctly, and ships a TypeScript reference plus property-based tests, but no machine-checked proofs.

The 24 RA-linearizability VCs are trivial here. Peritext’s state is four grow-only components and componentwise union commutes; by sal closes them in seconds. That work lives in Peritext_CRDT.lean (571 lines, mostly state plumbing), and at the level of RA-linearizability the data type is verified.

The interesting questions, however, are about the read: the function that takes the state and renders it as formatted text. That function lives in Peritext_ReadSide.lean, currently 1,316 lines. It answers questions like: do the start- and end-side anchor bits matter? do overlapping bolds compose? does tombstoning a character preserve the formatting of everything else? do insertions at a span boundary fall inside or outside the span under the paper’s expand-vs-contract rules? None of these are anywhere in the 24 VCs.

The eight worked examples in §3 turn out to be a much better starting point than the VCs for actually mechanising the data type. Each example is small enough to write down as a concrete state and a concrete claim about its read. Nik Swamy and Shuvendu Lahiri have a recent post calling this kind of artifact a Small Proof-Oriented Test, or SPOT: “a small, concrete, verified test case, proving that the test always succeeds.” By writing §3 the way they did, the Peritext authors had effectively done the specification-engineering half of that job for us; all that was left was to translate each example into Lean and prove it against the implementation. So we did.

Take Example 1 from the paper. Alice bolds the entire sentence “The fox jumped” while Bob concurrently inserts the word “brown” in the middle. Convergence is not the question: both replicas will agree on a state once they merge. The question is what the rendered text looks like. Peritext’s expand-on-the-bold-side rule says the inserted text should fall inside Alice’s bold span, so the merged result reads “The brown fox jumped”, with “brown” formatted bold along with everything else. In Sal we simplify Bob’s insert to a single character ‘b’ (enough to exercise the rule, and the proof stays tractable), and the SPOT translates almost directly:

-- Alice (rid 0) types "The fox jumped" and bolds the whole thing.
def ex1_pre : Scenario :=
  the_fox_jumped.bold 0
    ['T','h','e',' ','f','o','x',' ','j','u','m','p','e','d']

-- Bob (rid 1) concurrently inserts 'b' between "The " and "fox".
def ex1_post : Scenario := ex1_pre.insertCharAfter 1 (4, 0) 'b'

-- Alice's bold mark, opId (15, 0), spans the original sentence.
def ex1_mark : MarkOp := Mark.bold (15, 0) (1, 0) (14, 0)

-- The intent claim: Bob's 'b' ends up inside Alice's bold span.
example : in_span_visible ex1_post.state ex1_mark (16, 1) := by ...

The other seven worked examples take the same shape: an English description in the paper, a few lines of DSL builder to set up the concrete scenario, and a one-line theorem statement that captures the paper’s intent claim. Peritext_SPOT.lean is 323 lines for all eight. The proofs themselves are unremarkable; the work was in setting up the DSL so the scenario reads like the paper.

The read-side theorems in Peritext_ReadSide.lean are universally quantified versions of the same statements: instead of “Bob’s ‘b’ is in Alice’s bold span at this concrete state,” the theorem insert_within_span_in_span_visible says “for any state, any mark, any character inserted strictly inside the span at a fresh opId, the inserted character is in the span.” Each SPOT then closes via that universal theorem applied to its concrete state. Generalisation of the SPOT is the read-side proof, which I find a useful way to think about the relationship.

We have since written SPOTs for all 28 RDTs in the suite, and the generalised read-side theorems for the Tier-C ones. The Tier-A SPOTs are mostly there as documentation that a reader can confirm the read function does what the lattice arithmetic suggests it does.

It is worth saying how an early draft of the Peritext read-side went wrong, since the failure is a sharp version of the talk’s title. I had Claude draft a predicate called in_span_boundary from a careful reading of paper §3.3, with four cases (start of span, end of span, after-of-start, after-of-end) and a boolean side bit. The proofs went through. About 400 lines of theorems closed across both the CRDT and the MRDT versions, and I was about to commit. Re-reading §3.3 with the predicate in front of me, the predicate turned out to be backward. The after_of c endId → endSide clause encoded the opposite of the link-contract case: it included post-endId inserts in the span where the paper says they should be excluded. The proofs validated the proof, not the spec. Tests pass, code is wrong. The bug was caught only when I wrote an alternative predicate (in_span_visible) and the two disagreed on Example 8.

The fix was a four-rule inductive characterising the RGA visible order (parent-child, sibling via opid_max, left-descendant-of-older-sibling, transitive) plus a separate bold_expand_reach predicate for the bold-expand semantics in Example 7. About 800 lines of the buggy parallel track were deleted in the process. Eight _visible theorems now map one-to-one to the paper’s eight worked examples; there is a crosswalk for anyone who wants the full table.

The piece of methodology I would recommend to someone starting out: the SPOTs catch this kind of failure if the SPOT itself is faithful to the paper’s example. Had I started by formalising the worked examples as SPOTs and only then generalised, the disagreement on Example 8 would have shown up immediately rather than after 400 lines of proofs against the wrong predicate. John Regehr’s zero-degree-of-freedom LLM coding post argues a similar point in a different domain: pin the agent with fast, deterministic, executable oracles, and the agent has nowhere to drift. The Peritext SPOTs are an executable oracle for the read-side spec.

Beyond Peritext

Peritext is one example of a broader pattern. PaPoC 2026 had several other talks where the hard work was not the merge but in writing down what the data type was supposed to mean in the presence of concurrent edits. Three I want to flag, because they suggest the intent-formalisation problem has finally become the bottleneck people are willing to talk about openly.

AegisSheet (Florian Pfeil, David Scandurra, and Julian Haas, TU Darmstadt) studies collaborative spreadsheets. Their starting move is honest empirical work: they tabulate how Google Sheets and Notion behave under all combinations of concurrent EditCell / InsertR/C / RemoveR/C / MoveR/C and classify each outcome as desirable, suboptimal, or destructive. Most of the cells in those tables are red. They then design a compositional spreadsheet CRDT that turns the red cells green. The CRDT is the easy half; the table of intended semantics is the hard half, and they did it.

ERA (Kegan Dougal, Element Creations) addresses the duelling admins problem in group-management CRDTs in Matrix and Keyhive: two equally-permissioned admins concurrently revoke each other’s permissions. Whose revocation wins? Kleppmann’s PaPoC 2025 keynote suggested seniority ranking, but a Byzantine admin can backdate events to roll back their own revocation. ERA proposes epoch events: an external arbiter batches concurrent events into epochs and imposes a bounded total order, which gives finality without sacrificing availability. Like the spreadsheet case, the contribution is mostly about specifying what correct looks like under adversarial concurrency, with the data structure designed to match.

The closing lightning talk by Florian Jacob, Johanna Stuber, and Hannes Hartenstein (KIT) sketches a path to formal verification of local-first access control. Same shape: write the intent down precisely, then prove an implementation matches.

None of these are RDT-specific. Each is a different domain (rich-text, spreadsheets, group membership, access control) where the merges are easy and the intent is hard. SPOTs, oracles, and the multi-modal Lean stack are all reactions to the same shift in where the difficulty lives.

The past three weeks

Since the Sal paper was finalised, the repo has had something like 200 commits, overwhelmingly mine, with Pranav having landed the architectural heavy lifting earlier on. The headline change is in coverage: 13 new CRDTs and 2 new MRDTs since the paper, taking the suite from 13 RDTs to 28. Most of the additions are smaller scalar, set, and map types (MAX-Register, MIN-Register, LWW-Register, LWW-Map, MAX-Map, Grow-Only-Set, Grow-Only-Multiset, LWW-Element-Set, Shopping-Cart, and a few more) that fill out Tier-A and Tier-B coverage and serve as documentation that the obvious read functions do the obvious thing. The three additions that took real work, and that the rest of this section is about, are the paper-only designs that finally got machine-checked Lean ports.

Peritext is the largest of the three: 1,316 lines of read-side, 571 of CRDT, 291 of a small DSL for the §3 worked examples, and 323 of SPOTs. Eight intent-preservation theorems matching the paper’s worked examples one-to-one, plus a wf_afters acyclicity invariant and a bold_expand_reach predicate for Example 7. Roughly two days of agent-paired work, mostly spent on the spec rather than on the proofs.

The Add-Wins Priority Queue of Zhang et al. (Internetware 2023) is a 391-line CRDT plus a 325-line read-side. The translation is interesting in its own right. The paper has a per-record count field that ties commutativity in knots; in Sal we flatten it into a separate I component for increment records and use a snapshot-in-op-payload trick (the Rmv op carries the observed-set as a parameter), which gives us rc := Either everywhere and lets the RA-linearizability VCs drop out without any SMT.

The Bounded Counter of Balegas et al. (SRDS 2015), in the state-based formulation that Sypytkowski wrote up in 2019, is a 465-line CRDT structurally a PN-Counter plus a transfer matrix. The 24 VCs are trivial. The bound itself is not part of the verified model: it is enforced operationally at the client boundary (a replica refuses to emit a Dec that would push its quota negative), and that part is currently unverified. The headline number alone might suggest more than is there, so worth calling out.

A separate cleanup is worth noting. About 85 Blaster-admits across the suite closed via a single pattern, per-component decomposition: instead of one monolithic SMT call on the full state, split the state into components, prove each component independently, then combine. The OR-Set MRDT alone went from 20 VCs trusting Z3 down to 3 (closing 17 in a single commit). The pattern was human-found; the agent applied it consistently across files.

The qualifier from the Peritext section applies to all three of these ports as well: the RA-linearizability VCs are easy, and the work that took time was the spec.

For readers who want to play with the data types directly, fplaunchpad.org/sal hosts an interactive playground for every RDT in the suite. The CRDT pages let you drive two replicas in parallel and merge one into the other; toggling “show concrete state” exposes the lattice that the convergence proofs are about. The MRDT pages render the operation history as a git-style commit DAG and let you do three-way merges over the lowest common ancestor. None of this is load-bearing for the verification story, but it is a more direct way to see what the data types actually do than reading the proofs.

Closing

Agents have made proof-engineering noticeably cheaper. Spec engineering still sits with the human author, and SPOTs, executable oracles, and the multi-modal Lean stack each approach that from a different direction. None of them is sufficient on its own; in combination they look like a workable methodology.

The slides are here, the repo is at https://github.com/fplaunchpad/sal, and the question I closed the talk with was: what will you prove next?

There are many fundamental systems skills that go into working on a language like OCaml that only come with soaking in systems programming. By systems programming, I mean the ability to use tools like the command-line, editors, version control, build systems, compilers, debuggers, bash scripting, and so on. This is often something that one takes for granted when working on such projects, but is often inscrutable for new contributors, who may not have had the opportunity to develop these skills.

I struggle with this in my own research group. Students approach me to work on the OCaml compiler because they have studied OS, Compilers and Computer Architecture in class. But once they understand that working on OCaml involves actually hacking on systems, they are often lost. How do you build the compiler from source? How do you manage your changes? Do I have to build the entire compiler if I make a small change in the runtime system? The compiler crashes with a segfault – how do I debug it? Worse, the students do not even know what questions to ask, and come back with “This is all new to me, I don’t know where to begin. ChatGPT doesn’t help.”

The CS education in India often lacks a focus on these practical systems skills, which can make it challenging for new contributors to get involved in systems programming. Looking at my own past, my undergraduate CS education, like many others in India (and potentially elsewhere), had mandatory OS and Compiler Construction courses. But neither had a dedicated lab component. It is natural that these theoretical courses do not prepare the students for the practical aspects of systems programming.

I was privileged to have a computer at my school, an IBM PC AT Model 5170 and later an IBM PC 340, and surprisingly, had an education where I got to do programming from a very young age. There was lots of BASIC programming but also just tinkering with the system, learning how to use DOS, and later Windows 3.1, 95, and of course playing games (Doom and Prince of Persia, mostly). This early exposure to computers and systems programming gave me a head start. Many students, especially those from less privileged backgrounds, do not have this early exposure. They may have learned some programming, but not had the time to tinker with systems for extended periods of time.

This challenge of bridging the gap between theoretical CS education and practical systems programming skills is a common one faced by professors working in the broad systems area. The problem is compounded by the fact that these skills are difficult to teach in a traditional classroom setting—they require hands-on experience, experimentation, and often many hours of frustration and debugging. These are skills that come from doing, not from reading or watching lectures. I would be curious to hear from others about their experiences and how they have addressed this challenge.

That said, there are resources available online that can help new contributors acquire these skills. This list is biased to the areas of the compiler that I work on. I mainly work on the backend and the runtime system. The only reason I usually touch the frontend is to lower the features that I care about to the backend. Here are some I have found useful for working on the OCaml compiler:

• Systems programming
  • Course: MIT Missing Semester: This is a fantastic resource that covers a wide range of topics related to systems programming, including command-line tools, version control, editors, and more. The course is available online for free and includes video lectures, notes, and exercises. I encourage you to read the motivation for this course.
  • Course: Stanford CS45: CS45 is an extended version of the MIT course, and delves into the topics in more detail.
  • Video: CppCon 2015: Greg Law “Give me 15 minutes & I’ll change your view of GDB”: The talk explores GDB’s less-known features and sheds light on some advanced debugging techniques.
  • Tool: rr - Lightweight Recording and Deterministic Debugging: rr is a powerful tool for recording and replaying program execution, which can be invaluable for debugging complex issues in systems programming. I’ve stopped using gdb directly for anything non-trivial and have switched to rr.
• OCaml
  • Course: CS3100 Paradigms of Programming: The course covers a significant chunk of the OCaml language. You should be able to self-study the course to get a good understanding of the language. That said, the course deliberately does not cover the build system (dune), package manager (opam), command-line tools for the compiler (ocamlc, ocamlopt), editor integration (merlin, ocaml-lsp, ocamlformat), etc.
  • Book: Real World OCaml: The book has a section on the compiler and the runtime system, which gives a great overview of the memory representation, garbage collection, and other aspects of the runtime system.
• Diving deeper
  • Book: Systems Performance: Enterprise and the Cloud, 2nd Edition: This book provides an in-depth look at systems performance, covering topics such as CPU architecture, memory hierarchy, storage systems, and networking. It is a valuable resource for understanding the underlying principles of systems programming and performance optimization.
  • Book: The Garbage Collection Handbook: This book offers a comprehensive overview of garbage collection techniques, algorithms, and implementations. It is an essential resource for understanding memory management in programming languages like OCaml.
  • Book: The Art of Multiprocessor Programming: This book provides a deep dive into concurrent programming and synchronization techniques, which are crucial for understanding multi-threaded runtime systems like OCaml 5’s multicore runtime and the programming model.

I will probably keep editing this post as I find more resources. If you have suggestions for other useful resources or experiences to share, please feel free to reach out to me.

The answer to all of these turns out to be a resounding yes thanks for x-ocaml. This post is my experiment playing with x-ocaml and integrating that into this blog.

The most wonderful thing about programming is that it lets you experiment freely. You can try out an idea, get instant feedback, and learn by doing—much like playing with Lego bricks or sketching on a canvas. The two main courses that I teach at IITM, CS3100 and CS6225, both involve me live-coding during every lecture. However, blogging about OCaml where the code is static and non-interactive always felt a bit unsatisfying.

Enter x-ocaml, which allows for a way to embed OCaml notebooks into any webpage thanks to WebComponents. All you need to do is to load some JavaScript in your webpage and you can start embedding code cells using <x-ocaml> tag. The snippet below:

<x-ocaml>
print_endline "Hello, world"
</x-ocaml>

renders to:

The code is interpreted in the browser thanks to the OCaml interpreter compiled to JavaScript through the Js_of_ocaml compiler. There is also support for Merlin and OCamlformat in the code editor. Try hovering over the functions and writing some code. You should see inferred types and auto-completion suggestions. It turns out that this solution integrates well with Jekyll, which is what I use for this blog.

Reverse-mode AD using Effects

Js_of_ocaml also supports effect handlers. Here’s the implementation of reverse-mode algorithmic differentiation, implemented using effect handlers running in the browser.

Here are some tests.

Using other libraries

x-ocaml also supports loading any js_of_ocaml compatible library into the webpage. Let’s use digestif.

For any library that you want to export, install the library using opam. x-ocaml provide a command-line utility to export the library.

$ x-ocaml --effects digestif.ocaml -o digestif.js

This produces the JavaScript artifact that can be used in the webpage. It may be instructive to look at the source of this post to see how the compiler and the libraries are integrated into this blog post. There is a little script at the top of the file:

<script async
  src="{{ '/assets/x-ocaml.js' | absolute_url }}"
  src-worker="{{ '/assets/x-ocaml.worker+effects.js' | absolute_url }}"
  src-load="{{ '/assets/digestif.js' | absolute_url }}"
></script>

What next?

There is a number of rough edges to x-ocaml. This is expected since this project appears to be one of Arthur’s hacking expeditions (which, as usual, is pushing the state of the art forward).

It would be fun to use this for teaching CS3100 and also other OCaml tutorials. Perhaps even have an interactive version of Real World OCaml book.

Not all OCaml libraries can be compiled to JavaScript. The common reason being that they may depend on features not available on JavaScript. In writing this post, I unsuccessfully tried for a long time to get mirage-cypto working. mirage-crypto has a large C dependency, which does not work with Js_of_ocaml. Js_of_ocaml promises to take any opam library installed on your opam switch and compiles that to JavaScript. However, at that point, we’re really cross compiling the opam packages installed on your switch to JavaScript since the installed package may make some assumptions about the platform that it is supposed to run on. Hence, JavaScript compilation of arbitrary OCaml packages is unlikely to work in the general case. Unfortunately, the error was difficult to debug since the failure was at runtime, and was not apparent in the error messages (at least for me, who has little JavaScript experience). It would be nice to have the opam packages explicitly say whether they are JavaScript compatible, and have build tooling that reports errors like these early.

Capturing unique values

Let’s start with an example. Recall the signature of the unique reference module.

module type Unique_ref = sig
  type 'a t
  val alloc : 'a -> 'a t @ unique
  val free : 'a t @ unique -> unit
  val get : 'a t @ unique -> 'a Modes.Aliased.t * 'a t @ unique
  val set : 'a t @ unique -> 'a -> 'a t @ unique
end

Assume that we also have an implementation of the module:

module Unique_ref : Unique_ref

Consider the following example, which works fine:

let works () =
  let t = alloc 42 in (* Allocate a unique reference *)
  free t (* free it *)

Now consider this modified example:

let wat () =
  let t = alloc 42 in (* Allocate a unique reference *)
  let f () = free t in (* capture free in a closure *)
  f (); (* free it *)
  f () (* free it again??? *)

Observe that f has captured t in the closure, and when called frees t. It should be clear that calling f more than once is bad – leads to a double-free issue! What property do we want of f? Uniqueness is insufficient; we have a unique reference to f in this program, with which we call f twice.

What we want to enforce is that f can be called at most once. The compiler has a linearity mode which captures the idea of how many times a value can be used. We have two modes in the linearity axis – once, which stands for “at most once” and many (the default one for all values), which allows values to be used arbitrary number of times.

Whenever a unique value is captured by a closure, the closure gets a once mode, which allows the closure to be called at most once. This program rightly gets rejected by the compiler.

File "./unique_ref.ml", line 32, characters 2-3:
32 |   f () (* free it again??? *)
       ^
Error: This value is used here,
       but it is defined as once and has already been used:
File "./unique_ref.ml", line 31, characters 2-3:
31 |   f (); (* free it *)
       ^

A linear ref

Now, one might wonder whether the unique reference that we’ve implemented may be implemented with the linear mode. The answer is yes.

module type Linear_ref = sig
  type 'a t
  val alloc : 'a -> 'a t @ once
  val free : 'a t @ once -> unit
  val get : 'a t @ once -> 'a * 'a t @ once
  val set : 'a t @ once -> 'a -> 'a t @ once
end

module Linear_ref : Linear_ref = struct
  type 'a t = { mutable value : 'a }
  let alloc x = { value = x }
  let free t = ()
  let get t =
    t.value, t
  let set t x =
    t.value <- x;
    t
end

This works as expected:

open Linear_ref

let works () =
  let r = alloc 42 in
  let v,r = get r in
  let r = set r (v + 1) in
  let v,r = get r in
  print_int v;
  free r;
  ()

let fails () =
  let r = alloc 42 in
  free r;
  get r (* fails here *)

with the error message:

File "./linear_ref.ml", line 34, characters 6-7:
34 |   get r (* fails here *)
           ^
Error: This value is used here,
       but it is defined as once and has already been used:
File "./linear_ref.ml", line 33, characters 7-8:
33 |   free r;

Why both linearity and uniqueness?

Given this example, you might be wondering, if the safe reference may be implemented equivalently using both uniqueness and linearity, why do we need both? Obviously, there’s something interesting going on where unique values captured in a closure needs linearity. Does that mean linearity is sufficient?

It turns out that only recently was the relationship between the two formally studied in the same type system. While linear types and uniqueness types have a long history of being studied independently, Marshall et al. in their paper, “Linearity and Uniqueness: An Entente Cordiale”, present the ideas in the same type system. They provide some key insights.

The first insight is that

in a setting where all values must be linear, we can also guarantee that every value is unique, and vice versa! Intuitively, if it is never possible to duplicate a value, then it will never be possible for said value to have multiple references.

In our Unique_ref and Linear_ref every operation that operates on the ref requires uniqueness or linearity, respectively. Hence, they seem almost equivalent in expressive power.

It is when we also have the ability for unrestricted use (non-linear/non-unique) that differences between linearity and uniqueness begin to arise, as we will soon see.

In our language, we do have the ability for unrestricted use. That is, in the linearity axis, many is the default mode attributed to all the values not tagged or inferred as once. Similarly, aliased is the default mode attributed to all the values not tagged or inferred as unique.

The type system has submoding: values may move freely to greater modes (which typically restrict what can be done with those values) but not to lesser modes. For example, a many value may be safely use in a context where a once value is expected.

let works () =
  let set_to_20 (r @ once) =
    r := 20
  in
  let r @ many = ref 10 in
  set_to_20 r (* [r @ many] is passed to a function that expects [int ref @ once] *)

Similarly, you can use a unique value in a context where an aliased value is expected.

let dup r = r,r

let works () =
  let r = Unique_ref.alloc 42 in
  let a,b = dup r in
  a,b

The type of the works function is val works : unit -> int Unique_ref.t * int Unique_ref.t, which crucially lacks the fact that the references are at unique mode. We can’t call any functions from the Unique_ref module with these references, all of which expect a reference with unique mode.

Uniqueness is more appropriate for safe refs

In our running example of implementing a safe ref, it turns out that uniqueness is more appropriate. Consider the type signature of free in Unique_ref:

val free : 'a t @ unique -> unit

The type signature says that there are no other aliases to this reference. Hence, its memory may be safely deallocated. However, consider the Linear_ref.free signature:

val free : 'a t @ once -> unit

The signature says that this reference must be used at most once. In particular, just by looking at the signature, we cannot conclude that there are no other aliases to this reference. But we know that the API is safe since the only way to create a safe reference is through the alloc function, which returns a once-usable reference, and every other operation also expects and returns a once-usable reference.

The correctness of the linear version depends on reasoning over the whole API, whereas the unique version can be concluded to be safe just by looking at the signature of the free function. This modular reasoning makes uniqueness more appropriate for our safe reference API.

Past and the future

In a sense, uniqueness and linearity are duals of each other. Uniqueness talks about the past – whether a value may be aliased in the past. It is okay to alias a unique value in the future and lose the uniqueness mode. Linearity talks about the future – whether a value may be used more than once in the future. You can take any value and ascribe a linear mode to it, restricting its use in the future. However, there may be other aliases to this value in the past.

Conclusions

The code examples are available here. Section 2.1 of Marshall et al.’s paper is quite readable and explains the distinction between linearity and uniqueness with some historical context. I highly recommend it.

Acknowledgements

Thanks to Richard Eisenberg for the discussions which spurred this post.

My intention in this post is not to explain how the different modes work. There are a number of blog posts and academic papers written about modes. I recommend the interested reader to have look at them. The following table summarizes the main properties tracked by modes, the corresponding mode names, and resources for further reading:

The OCaml compiler extended with modes is developed in the open, and is used in production at Jane Street. The repo also has some documentation of the extensions.

Be warned that the compiler and the language features are fast evolving. The code examples presented in the blog and the paper referenced above are likely not to work. I expect the same for the code examples in this post in the near future, but that’s what one should expect with these bleeding-edge features.

Behavioural types and runtime overhead

A couple of years ago, I wrote a post on behavioural types where the types capture the sequence of operations that may be performed on the values with those types. The correctness of the system depended on the linear use of the resources. Since OCaml does not provide support for enforcing linearity statically, the implementation uses a dynamic check, using a fresh ref cell that gets consumed every time the type state changes. If we are guaranteed that the resource is not aliased statically, then there’s no need for the dynamic check. This is where uniqueness helps.

Uniqueness mode allows the OCaml compiler to statically guarantee that certain values are not aliased. This enables optimizations and eliminates the need for some runtime checks, which is particularly valuable in systems programming for ensuring memory safety and efficient resource management.

Setting up OCaml with modes

An opam repository with the modes extensions and packages supporting modes is available here. Here’s how you can set up the new compiler:

# this will take time
opam switch create 5.2.0+flambda2 --repos with-extensions=git+https://github.com/janestreet/opam-repository.git#with-extensions,default
eval $(opam env --switch 5.2.0+flambda2)

An explicitly memory-managed reference

Suppose you want to implement a mutable reference whose memory is explicitly managed (not managed by the GC), you may go for the following interface:

module type S = sig
  type 'a t
  val alloc : 'a -> 'a t
  val free : 'a t -> unit (* unsafe *)
  val get : 'a t -> 'a
  val set : 'a t -> 'a -> unit
end

This interface provides an explicit free, which releases the memory associated with this reference. This opens up the possibility of memory safety bugs such as use-after-free and double-free. We can use uniqueness modality to get a safe API. Here’s the interface:

module type S = sig
  type 'a t
  val alloc : 'a -> 'a t @ unique
  val free : 'a t @ unique -> unit
  val get : 'a t @ unique -> 'a Modes.Aliased.t * 'a t @ unique
  val set : 'a t @ unique -> 'a -> 'a t @ unique
end

The unique annotation states that the value is not aliased. The operations on the reference expect that this reference is not aliased. Observe that get and set take in the unique reference and also return them unlike the original interface. You can use this like so:

# let okay r =
    let v, r = get r in
    let r = set r 20 in
    free r;;
val okay : int M.t @ unique -> unit = <fun>

The key bit is that free consumes the unique reference; you can no longer produce a unique handle to the same reference and hence, you cannot call free, get or set on this reference which has been freed.

# let wont_work r =
    free r;
    get r
  ;;
Error: This value is used here, but it has already been used as unique:
Line 2, characters 7-8:

Modes.Aliased.t

Uniqueness applies deeply. If a value is marked as unique, then the transitive closure of the reachable parts of the object is also expected to be unique. The return value of get is a pair, which is marked as unique¹. Hence, both the components of the pair are expected to be unique. However, we don’t want to impose uniqueness of the value stored in the reference. The language allows parts of the value to be marked as aliased. Modes.Aliased.t is defined as:

module Aliased : sig
  type 'a t = { aliased : 'a @@ aliased } [@@unboxed]
end

The language allows record fields to be annotated as aliased, while the record itself may be uniquely referenced.

Implementation

Here’s an implementation of that satisfies the signature.

module M : S = struct
  type 'a t = { mutable value : 'a }
  let alloc x = { value = x }
  let free t = ()
  let get t =
    let a = Modes.Aliased.{aliased = t.value } in
    a, t
  let set t x =
    t.value <- x;
    t
end

There’s nothing surprising about this implementation. Note that the compiler is doing a lot of work behind the scenes to ensure that the functions do in fact satisfy the uniqueness requirements. For example, if you change the implementation of set to do something innocuous where the compiler cannot prove that the value is not aliased, the program no longer compiles:

# let set t x =
    t.value <- x;
    let t' = Fun.id t in (* compiler cannot prove [t'] is not aliased *)
    t'
Error: <snip>
Values do not match:
 val set : 'a t -> 'a -> 'a t
is not included in
 val set : 'a t @ unique -> 'a -> 'a t @ unique
The type 'a t -> 'a -> 'a t is not compatible with the type
 'a t @ unique -> 'a -> 'a t @ unique

Refs that explain their work

The earlier blog post used polymorphic variants to encode the protocol of operations that are permitted on a ref cell. The implementation is reproduced below:

module type Ref =
sig
  type ('a, 'b) ref constraint 'b = [>]

  val ref   : 'a -> ('a, 'b) ref
  val read  : ('a, [`Read of 'b]) ref
              -> 'a * ('a, 'b) ref
  val write : ('a, [`Write of 'b]) ref
              -> 'a
              -> ('a, 'b) ref
end
module Ref : Ref =
struct

  type ('a, 'b) ref =
    {contents     : 'a;
     mutable live : bool} (* For linearity *)
     constraint 'b = [>]

  let ref v = {contents = v; live = true}

  let check r =
    if not r.live then raise LinearityViolation;
    r.live <- false

  let fresh r = {r with live = true}

  let read r =
    check r;
    (r.contents, fresh r)

  let write r v =
    check r;
    { contents = v; live = true }

  let branch r _ = check r; fresh r
end

Observe that we use a dynamic check to enforce linearity. It requires a fresh ref cell for each operation performed on this reference. With uniqueness, we can enforce this statically, avoiding the dynamic check and the fresh ref cell requirement.

module type Ref =
sig
  type ('a, 'b) ref constraint 'b = [>]
  (* 'b is the behavioural type variable *)

  val ref   : 'a -> ('a, 'b) ref @ unique
  val read  : ('a, [`Read of 'b]) ref @ unique
              -> 'a Modes.Aliased.t * ('a, 'b) ref @ unique
  val write : ('a, [`Write of 'b]) ref @ unique
              -> 'a
              -> ('a, 'b) ref @ unique
  val branch : ('a, [>] as 'b) ref @ unique
               -> (('a, [>] as 'c) ref @ unique -> 'b)
               -> ('a, 'c) ref @ unique
end

module Ref : Ref =
struct
  type ('a, 'b) ref = {mutable contents : 'a} constraint 'b = [>]

  let ref v = {contents = v}

  let read r =
    let c = Modes.Aliased.{aliased = r.contents} in
    c, Obj.magic_at_unique r

  let write r v =
    r.contents <- v;
    Obj.magic_at_unique r

  let branch r _ = Obj.magic_at_unique r
end

The only changes necessary in the signature were a number of uniqueness and aliasing annotations. Notice that the implementation no longer needs the dynamic check! Obj.magic_at_unique has the type 'a @ unique -> 'b @ unique, and is the version of Obj.magic with uniqueness annotation. We use it to advance the protocol type state.

Where next

The rest of the examples in the original post should also benefit from uniqueness annotations to remove the runtime overheads.

The complete code examples are available here. You can also play with the code examples directly in the browser thanks to Patrick Ferris’ OCaml with extensions js_of_ocaml top-level.

Since the modes features are constantly evolving, there are no stability guarantees yet. However, I’m excited about the possibility of modes improving how we do safe systems programming in OCaml.

Addendum

Looks like there’s a part 2 of this post.

Footnotes

At IIT Madras, my research group develops programming language abstractions to solve systems problems. The group is composed of research associates (fixed-term project staff), PhD, MS and MTech students, undergraduate research students (who are typically BTech students from IIT Madars) and interns. I made the following post a few weeks ago, for which I received a lots of enquiries, and I have been busy writing similar responses to many of them, which I summarise below.

Internship positions

Internship enquiries are the most frequent ones that I receive. Here’s how you can make it work. Please do go through my web page to look at what areas I work on. Write to me about what interests you and what your goals are.

My group works on systems. To make the internship work well, we require that you have demonstrable systems building experience. Do build projects that go beyond your coursework. Make sure that the projects are developed publicly on GitHub or other similar platforms so that one can take a look at what you’ve built. Even better is contributions to other open-source projects.

The group solves systems problems with functional programming. If you have prior experience with functional programming, such as building small projects with OCaml, Haskell, Scala, Scheme or other languages, it is easier for me to assess your interest. That said, if you are great at any programming language, having built non-trivial projects in any language, then you have the right skills for internships in my group. Generally, I expect the interns to have done course work on OS, compilers and computer architecture. Significant projects in any of those areas is a huge plus.

I should clarify that my recommendation letters for graduate programs will reflect my honest assessment of the internship. I will decline writing a recommendation letter if I think I may not be able to provide a strong one.

I do not work on projects that are primarily AI/ML or Web Development. If you write to me looking for projects in those areas, it is very likely that you won’t hear from me. Please don’t bulk email faculty CCing or BCCing everyone in the department. It is likely that no one will read such an email.

PhD/MS/MTech positions

For academic positions, please have a look at https://research.iitm.ac.in/. There are alternative ways to enter MS and PhD positions by being a reserach associate and completing some coursework at IITM. For more information, see here.

Contributing to the OCaml community

A significant chunk of the enquiries were from folks who hold full-time positions looking to be involved in the research group. Unfortunately, making part-time positions work is a challenge for both sides. I would encourage contributions to the wider OCaml community.

There are several great ways to get involved with the community. Here’s what I usually recommend.

• Learn the basics.
  • Go through the OCaml part of my CS3100 course. The course has a YouTube playlist and programming assignments. Complete the programming assignments.
  • Read the Real World OCaml book.
  • There are lots of other resources at OCaml.org, the official website of the OCaml community and the ecosystem.
• Join the community.
  • OCaml discord and discuss are great places to hang out with other OCaml folks and ask questions.
  • Discord is better for quick clarifications and discuss for longer form discussions.
• Look for “good first issues” in the OCaml projects and work on them
  • Check out the core platform tools under the OCaml github org. See OCaml compiler, dune build system, opam package manager, ocaml.org, etc.
  • Across the wider ecosystem – SemGrep, OpenGrep, Rocq, etc.
• Work on self-directed projects. Here is my list of ideas.

OCaml community also participates in Outreachy internships. Outreachy internships are paid internships for underrepresented groups. It is a great way to contribute to the community while being mentored by folks from the OCaml community. Here’s a nice intro (in Tamil) to the impact that Outreachy program had on an Outreachy intern. Look out for announcements about Outreachy internships in the OCaml discuss forum.

Research Associate positions

This is for folks who want to contribute to the core research programme but do not see themselves joining academic programs. The expectation here is that you are an experiened systems engineer, who should see themselves easily qualifying for the internship positions in the group.

One useful way to look at this position is similar to a research software development engineer who helps build out the systems used for research or translate research to practice. In the past, research associates have helped upstream multicore OCaml. The easiest way to get into this role would be to do an internship, see whether you like this area, do well in the internship and then choose to apply to research associate position.

Another variant is a post-bacc or a pre-doc position aimed at highly motivated recent graduates, who are looking to build research experience. The expectation here is that we get papers into top venues in PL and Systems. For such students, I recommend going through my CS6225 Programs and Proofs course, watch the video lectures and complete the assignments. The course is not an easy one, but will expose you to the broad area of PL and specifically to deductive program verification. At the very least, you will come out with an understanding of what it is to think rigorously about program correctness.

Research associate positions are fixed-term positions. In order to make this work, the tenure should be at least 18 months to make it work.

Summary

While I may not be hiring actively all the time, do reach out to me if you are interested in any of hte above. Please follow me on LinkedIn, X or Bluesky, where I am likely to announce any open positions.

Installation

Install the tools from https://github.com/iovisor/bcc/.

Enabling frame pointers

The off-CPU stack trace collection, offcputime-bpfcc, requires the programs to be compiled with frame pointers for full backtraces.

OCaml

For OCaml, you’ll need a compiler variant with frame pointers enabled. If you are installing a released compiler using opam, you can create one the following switch command opam switch create 5.2.0+fp 5.2.0 ocaml-option-fp. Change out 5.2.0 for your preferred OCaml version.

Instead, if you are building the OCaml compiler from source, configure the compiler with --enable-frame-pointers option:

$ ./configure --enable-frame-pointers

Lastly, there is an option to create an opam switch with the development branch of the compiler. The instructions are in ocaml/HACKING.adoc. In order to create an opam switch from the current working directory, do:

$ opam switch create . 'ocaml-option-fp' --working-dir

glibc

The libc is not compiled with frame pointers by default. This will lead to many truncated stack traces. On Ubuntu, I did the following to get a glibc with frame pointers enabled:

1. Install glibc with frame pointers

   $ sudo apt install libc6-prof
   

2. LD_PRELOAD the glibc with frame pointers

   $ LD_PRELOAD=/lib/libc6-prof/x86_64-linux-gnu/libc.so.6 ./myapp.exe
   

Running

On one terminal run the program that you want to analyze:

$ LD_PRELOAD=/lib/libc6-prof/x86_64-linux-gnu/libc.so.6 ./ocamlfoo.exe

On another terminal run offcputime-bpfcc tool:

$ sudo offcputime-bpfcc --stack-storage-size 2097152 -p $(pgrep -f ocamlfoo.exe) 10 > offcputime.out

The command instruments the watches for 10s and the writes out the stack traces corresponding to blocking calls in offcputime.out. We use a large stack storage size argument so as to not lose stack traces. Otherwise, you will see many [Missing User Stack] errors in the back traces.

Caveats

offcputime-bpfcc must run longer than the program being instrumented by a few seconds so that the function symbols are resolved. Otherwise you may see [unknown] in the backtrace for function names.

Oddities

I still see an order of magnitude difference between the maximum pauses observed using offcputime-bpfcc and olly trace. Something is off.

Other links

• https://www.pingcap.com/blog/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance/