Spec for opam lock integration

This feature aims at easier replication of working development environments for applications (as opposed to libraries).

It is very common in other package managers like npm, so we use the same term "lock files" here.

Status

The feature consists of two parts:

• Using a lockfile involves creating a separate foo.opam.locked file, which is an alternate opam file with more constrained dependencies. The opam install --locked option activates the alternative file, and the suffix can be overridden via --lock-suffix=SUFFIX.
  • Using lock-files like this is implemented since opam 2.0.0.
  • The user can have multiple alternate opam lock files, and select across them by varying the suffix.
  • Lock files are just normal opam format files, so this feature is (at its simplest) the ability to select alternatively named opam files.
• Generating lockfiles has been released as the opam-lock plugin in 2.0.0. Doing it now in opam allows finer control and led to small improvements.

The interface and user-experience are really similar in 2.0 and 2.1, unless we decide to no longer need --locked to be always specified (see below). It seems the feature is still little-known and under-used. Check for ① discoverability concerns, and ② usability issues.

• @avsm : learn-ocaml, irmin-unix, janestreet releases core

Usecases

Lock-file generation (was in opam-lock plugin)

The basic usage is to reproduce an existing, known-good compilation setup (like e.g. npm does). From such a setup, run:

opam lock

from the package source directory. This will create "lock-files", i.e. in our case a copy of existing package definitions with:

• a .locked suffix appended
• dependencies made strict (e.g. depends: "foo" {>= "2"} will become depends: "foo" {= "2.7"} if foo.2.7 is installed in the current switch) ; more details in Implementation below

It is expected for applications to commit these files to version-control. It makes much less sense for libraries.

@avsm: what about multi repo: any way to lock just one package? @avsm: can we mark a binary only package?

Lock-file usage (already in opam 2.0)

There are mainly two use-cases:

1. For onboarding developpers: so they can get started from the known-good setup (or for developpers who got a broken dependency configuration and want to quickly get back to a working one)
2. For CI:

• when generating artifacts/deployments, or production in general, you want to use the known-good dependency configuration
• when testing, it can make sense also, but testing against the latest versions of the dependencies (without the lock-file) also makes sense, esp. if you publish on opam-repository

Usage is basically:

• opam install/upgrade/pin --locked (ongoing discussion: forbid this usage and force users to delete/recreate their switch)
• opam switch create DIR --locked (see considerations below about making it the default in this case)

Opam already does discovery of package definitions in the sources of pinned packages. This alters this behaviour by preferring the file with suffix .locked if it exists. Since this file contains the constrained dependencies, it will force the same setup to be reproduced.

CI

On the CI, it's expected to use the lock-file to check against the versions of the dependencies we care about, and for better reproducibility. Just adding --locked to the switch create command should be enough.

@avsm: need to modify ci scripts to test both locked (default) and unlocked (allowed_failure)

@avsm: lockfiles not used by default currently. Users create a switch for the application in a fresh switch. This needs to be tested as well.

Out of scope

Work in an installed switch

Dev meeting: for 2.2, out of scope for 2.1. But need to find a user.

Caveats

1. Opam files of dependencies are not registered: sensitive to changes in package definitions / differing repositories setups. opam switch export|import might be more suited for cases where this is a problem.
2. Lock files are generated on a given arch/os setup, and may not be applicable on a different one. This is only a limitation of lock-file generation, since opam files have the required expressivity.
   • Workaround: fine-tune the locked file manually. --best-effort might help using the incorrect lock-file.
   • Possible improvement (2.2): make the lock-file generation more clever (preserve arch/os related filters), and allow incremental generation from different arch/os combinations

Implementation

Format for lock files

The opam format already has everything, so locked files are plain opam files. The cost of this flexibility is that some info gets duplicated between .opam and .opam.locked files and can get out of sync ; that shouldn't be a problem as long as .opam.locked files are generated, and a warning was added when that happens.

Generation of the files

Running opam lock will generate *.opam.locked from *.opam with the following changes:

• for any dependency, change the version constraint to be strict {= "current-version"} ; transitive dependencies can be skipped with --direct-only.
• filters are removed (except for dev, with-test, with-doc)
• optional dependencies (depopts or alternatives) are:
  • turned into strict dependencies if installed
  • turned into conflicts if absent
• dependencies that are pinned are stored as pinned-depends. Opam will detect the remote of local pins and point to that.

Changes from 2.1 alpha to beta

• Making --locked the default this seems obvious for newcomers used to NPM and I agree... In some cases:

  • opam switch create . [--deps-only]: here it's clear we probably want it
  • opam install . [--deps-only]: less so. You probably only want it if you're in a local switch
  • opam upgrade: we probably don't
  • upgrade, then install again: should we downgrade to the lockfile ?
  • opam pin .: ?

  I'd say making it the default for local switches and using switch create or install seems like a good middle-ground; but we should be extremely careful with moving defaults. And have a [NOTE] advertising the auto-lock.

• Consider: having lock-files as overlay to avoid duplication (the locked file would only contain e.g. depends, conflicts...)

• Bypassing the solver: it was discussed, for consumers of lock-files and from-scratch switch creation, to bypass the solver

  • if the lock-file includes the complete dependency tree, we can disable the call to the proper solver and only use the opam/dose libs (almost: it seems at the moment lock-files stop before the compiler by default ?). this change would have no impact whatsoever on the user, if all pre-conditions are met.
  • it could be interesting to add a check on switch creation, ensuring that what is getting installed is exactly what is specified in the lock-file and no more.

• lockfiles as default for 2.1? Not for 2.1 until we see it evolve more at scale. Document workflow, let ecosystem adapt and then consider default change.