Dev Meeting 2020.10.23 (opam2nix)

opam-dev meeting 23-Oct-2020

Agenda: discuss opam2nix

Present: @rjbou, @timbertson, @patricoferris, @talex5, @Julow, @dra27, @AltGr, @avsm, @emillon, @samoht

@avsm: Present state of PL package managers elsewhere (and to a lesser with us with opam) are fighting OS package managers. CI work has lead us to obuilder as a replacement for Docker. Why? Managing containers in Docker provides more than we need for opam building, and the shift to want to support BSDs, Windows, macOS better suggests that we can have a more reliable infrastructure by using isolation and file-system layers directly. On the opam side, as part of this, we also want to be moving towards better integration between the opam universe and OS package management systems. Idea for this meeting is to identify lowest hanging fruit for supporting another package manager well.

Of those present, @timbertson, @Julow, @patricoferris and @talex5 have actually used Nix.

Tim presented an overview of Nix and opam2nix slides

Various notes:

• Process runs in two phases - an impure first phase generates, essentially, a lock file (which must be done on a developer's machine). The second pure phase abstracts all of the opam state away (so that everything now refers to paths, not switches), and handles the actual build and installation of the package.
• nix hashes related to opam's build hashes - difference is that they extend to everything (e.g. m4, etc.)
• historically, problems with opam packages not explicitly depending on things (like ocamlfind) which fundamentally fails in nix, since if you haven't depended on it, it won't be there. Dune has made this much better.
• Packages cannot themselves use the opam CLI directly

@avsm - compared with, say, esy which merges opam, workflow and integration with another external package manager, aim with opam is to get to the stage that the opam file is the source of truth (a principle of least churn). The opam2nix part looks exciting since it is an example of splitting the opam CLI stage, processing the metadata and then handing over to nix cleanly.

What at present would block using opam2nix as an alternative front-end for consuming opam packages? @timbertson - at the moment, it's more of a personal project, but it feels that it should be entirely possible.

@samoht what kind of digest nix is using. @timbertson digest of nar files instead of tar files so need to re-packages everything locally.

@avsm - at the moment, for opam-repository, the key thing would be ensuring that depexts are correct? @timbertson - at present opam2nix is very generous with depexts, if it's there it can be used. @dra27 - presumably this could be used from a CI perspective on opam-repository to try to harden depexts? @samoht - feels that we could very quickly be adding required metadata to opam-repository packages for improved nix integration.

@samoht - what can be done for binary caches? @timbertson - there's a service, Cachix (paid for). These have to be trusted. @avsm - but we could therefore provide a binary cache.

@emillon - how does opam2nix deal with external dependencies (C flags, include directories, etc.). @timbertson - nix has a bunch of things built into its upstream compiler packages. So far, it's just worked!

@avsm - might a first use-case from our side for opam2nix be to help with docs.ocaml.org and the multiple package universes required?

@avsm - noticing more users starting to use nix for workflows. Have you seen others? @timbertson - not really hearing from users much (5 contributors or so, so far).

@avsm Concluding (given dreadful audio lag on Whereby today!), moving towards improved support is a good idea. Core developers trying it out will potentially double the number of known users! @dra27 - agreed: a useful caveat is that there's a one-to-one correspondence between opam-repository packages and the resulting package in Nix, as opposed to RHEL/Debian where legacy makes the picture much more complicated. @avsm - indeed, and the version story here is much easier than the Epoch-bumping requirements.