Spec for GNU make jobserver support

Overview

opam's jobs defaults to one less than the number of CPUs and can be overridden with the --jobs argument on the command line. It is supposed to control the number of packages built simultaneously. In practice, the %{jobs}% is frequently passed as the parallelisation parameter for build systems (e.g. to make and dune). However, each of these packages receives the same jobs value, so in practice this means that opam permits jobs² concurrent processes.

GNU make has a similar problem when make is invoked recursively. It solves this with its jobserver protocol (see the GNU make manual) and this protocol is intended to be adoptable by other systems.

The GNU jobserver protocol has a seriously shortcoming: tokens must be returned by child processes, but the jobserver does not know which processes have tokens at any given point. Tokens (and parallelism) can therefore be lost to a malfunctioning client or - worse - if a sub-process is killed. The benefit of the approach used, however, is its simplicity both in terms of implementation and resources.

This spec proposes adding support for opam to be a client of an existing jobserver (for example, if opam is called within make) and for setting up its own jobserver if it is not passed one, a mechanism for indicating that commands wish to partake in the jobserver, and also an extended protocol for the jobserver which aims to deal with the GNU make shortcoming.

See also ocaml/dune#2647 and ocaml/dune#4331.

Implementation

jobserver

• ??? opam can use the jobserver to at the OpamProcess level, governing all sub-processes, or it could be used at the OpamParallel level.
• ??? opam can choose whether to include utility processes, such as download jobs, in the count
• ??? opam could choose a hybrid of these approaches - for example, if called from within make (where opam is a jobserver client) it may choose to govern all process creation with the jobserver (respecting GNU make's intention) but be more liberal when opam itself has created the jobserver

The jobserver itself is a named semaphore on Windows and a pair of pipes on Unix. The Windows implementation requires the addition of wrappers for Win32 Semaphores but is otherwise extremely simple, as opam simply creates the semaphore and then passes its name to any processes which wish to take part. The Unix implementation is most easily with a separate thread which simply selects for returned tokens and immediately writes them to the other pipe to be available for other processes. When opam is a client of another jobserver, care is required to ensure that opam returns all tokens it obtains (even on error).

jobserver variable

GNU make automatically sets up recursive calls to make (the rules are complicated, but in essence commands which being $(MAKE) will usually be included) to have access to its jobserver, but doesn't include this by default for all processes. This can be opted into for any command by prefixing it with +.

A similar scheme is proposed for opam: a command beginning make will be assumed to take part in the jobserver (since make is intended to be GNU make and is gmake on platforms where this is necessary).

• ??? We could also provide a global list of commands which are allowed to take part in the jobserver (e.g. "dune").

Commands which wish to opt-in should indicate this by including {jobserver} in the filter for the command, e.g.

["dune" "build" "-p" name "@install"] {jobserver}

Note that the call dune does not specify "-j" jobs - it is expected that Dune would pick up the jobserver from the MAKEFLAGS variable. jobserver is a meta-variable (similar to the status functions in GitHub Actions). It is treated as true, but the jobserver is only inferred if the variable was part of the evaluation. For example, {os = "win32" || jobserver} should disable the jobserver on Windows.

Extended jobserver

It's not totally clear (without digging through the history) why GNU make uses named semaphores for its Windows jobserver, as it's perfectly possible to spawn processes which inherit HANDLEs on Windows (indeed, this works with pipes created with Unix.pipe in OCaml). Our proposed extension to the jobserver protocol to deal with killed processes or malfunctioning clients can therefore be the same on Unix and Windows.

For compatibility, MAKEFLAGS should be set as normal, but for a command tagged {jobserver}, opam will also communicate via OPAMJOBSERVER the fds (or HANDLEs, on Windows) of another pair of pipes. These pipes are specific to the process. When the child process wants a token, it writes a single + to the write fd and attempts to read a character from the read fd. The jobserver responds to this by acquiring a token from the main job server (via either WaitForSingleObject or by reading the jobserver read fd) and then writes a dummy token back to the client. When the client wishes to return the token, it writes a single - to the write fd which the jobserver uses as a signal to return one of the tokens it is holding on behalf of that client to the main jobserver.

This approach increases the complexity of the jobserver in opam, since a pair of fds is required for each process. There is a risk of fd exhaustion, but only on extremely high core count machines - we can hope that the maximum number of fds will increase as these systems become more commonplace, rather than having to increase the complexity of the jobserver further to have sub-processes managing the token pools!