On breaking compat for the CLI for 2.1

[Drup]

Hi,

I recently migrated to the last version of opam-git, and noticed that opam now breaks compat with the CLI. In particular opam config var is not available anymore, and opam var must be used instead.

This is just a head's up that it break at the very least the following tools (and probably more):

• merlin
• ocp-browser
• tuareg
• dune

thanksfully this can be rolledback using the OPAMCLI variable, but I'm not sure the change is worth it.

---

# opam config report
# opam-version         2.1.0~beta4 (8ce5e420f8007fd8d7ad56fb6293af6f5469a432) 
# self-upgrade         no
# system               arch=x86_64 os=linux os-distribution=arch os-version=rolling
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[hidden-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[hidden-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 3
# repositories         2 (http), 1 (local), 4 (version-controlled) (default repo at 1aa37053)
# pinned               4 (git)
# current-switch       main
# read-state           Not_found

[hannesm]

Thanks for opening this issue, indeed in opam-repository there are several occasions of opam config var (and these are as well present in some "cross-compilation" settings for MirageOS: mirage-solo5, mirage-crypto, bigstringaf, ... are invoking opam config var prefix as well). I'd as well be in favour to keep the opam config var (maybe it could emit a deprecation warning on stderr?)

[dra27]

@hannesm - opam-repository should not be affected, since opam itself sets OPAMCLI=2.0 for package builds (see #4992). Scripts which intend to use 2.1 features/CLI syntax should specify --cli=2.1

[hannesm]

@dra27 I'm not sure what the story is. Do you mean that with opam 2.1, the following will be run with OPAMCLI=2.0: opam install mirage-crypto, while opam pin add mirage-crypto --dev will use the 2.1 cli version? OR is it the case that opam will always behave as 2.0, but only if --cli=2.1 is passed it'll unlock the new behaviour?

Makes me slightly puzzled and afraid, versioning in command line behaviour...

[dra27]

Thanks for the list, which I've added to the release checks for 2.1 (we need to update that during our next dev meeting, so we'll consider if there are more tools affected then).

The whole purpose of this feature is to allow the CLI to evolve cleanly without the mess of mixed ways of specifying commands
(cf. OPAM 1.2.2 opam config env vs opam 2.0 opam env). CLI versioning creates a pain point once (this time) where one must either decide to set OPAMCLI=2.0 or start using --cli=2.1 and then programs and code will never see deprecation warnings again.

We considered having warnings appear in opam 2.1 and then turn it in to an actual error in 2.2, but experience of watching OCaml deprecate like this (and, indeed, opam itself) shows that all that happens is that people ignore the messages and we have the same discussion at the next version when it becomes an error. opam's CLI is already complicated enough without having to actively support multiple syntaxes at the same time!

[dra27]

@hannesm - if you run opam then you get the latest version of the CLI (so 2.1). If you're writing a script, a blog post, etc. then you should use --cli. Package build instructions (so pinning, installing, removing, etc.) will all be run with OPAMCLI=2.0 - it's set in the same way as variables like OPAM_PACKAGE_NAME, OPAM_PACKAGE_VERSION and anything else specified in build-env.

One may of course choose to set OPAMCLI=2.0 permanently in your shell.

[hannesm]

@dra27 I'm still not sure whether I understand the new semantics. What I'm concerned about are:

• packages in opam-repository that call out to opam config env during build-time (gmp-freestanding, mirage-crypto, ocaml-freestanding) -- IIUC they'll be fine since something will set OPAMCLI=2.0 when installing packages via opam!?
• packages that call during normal operation out to opam config var prefix (such as mirage) in order to locate pkg-config files that contain some CFLAGS. Invocation is done during the mirage build step.

With a user hat on, I'd expect an executable within the same major version to not drop support for command line arguments. I'm aware that openssl went this path, but I don't think it is wise. Obviously, bumping / removing arguments in the next major version is fine with me. But I have not followed the discussion at all.

Writing shell scripts that need certain arguments / behaviour could parse the output of opam --version and branch (or error out) depending on the result thereof.

[dra27]

That's correct for the first point. For the second, either mirage build should be updated to set OPAMCLI=2.0 before doing what it's doing with the packages or the packages should use opam var instead (or potentially both). The change would be one-time and never need reverting - as soon as a feature which requires opam 2.1, 2.2, 3.0, 99.0 is done, --cli= should be specified on every call.

For the user hat, and with a lot of reverence to semver and so forth, opam 2.0 was released coming up on 2.5 years ago and I put slightly more weight on that, then the version number. Put another way, whether the version is 2.1 or 3.0, you'll still have to change the mirage tool, or force users to uninstall the package manager!

For shell scripts (and, indeed, tools like mirage), the idea is that you will not switch behaviour based on the client. The point with CLI versioning is that the new features are only available in the version in which they were added (so with OPAMCLI=2.0, you cannot access switch invariants, for example). The idea therefore is that if your script needs opam 2.1, it will not have a behaviour for opam 2.0 because it must be using something which is missing. However, if you declare OPAMCLI=2.0 (or, from now, pass --cli=2.1) then all future releases in the 2.x series promise to support CLI as you expect.

Running opam without OPAMCLI set and without --cli is the user version in the shell - you get the version you know is installed without having to specify it. It should gradually become unhygienic to invoke opam within another program without specifying the CLI version.

There is further on this in the specification in opam's wiki. I'm also planning on getting a Discuss post up to announce beta4 rather more formally and to introduce this change in more detail. I appreciate it's happened in the wrong order, but I keep being interrupted this week either by children or requests to hack OCaml's build system 🙂

We'll be discussing this in tomorrow's developer meeting - I am expecting that we will conclude that we would like tools to update (in many of these cases, not to set OPAMCLI but to use opam var instead of opam config var). For opam config var specifically, that may get reintroduced but because it's in editor configs (because of merlin advertising that), not because it might break tools.

[c-cube]

Please do not break compatibility in a minor release, whatever the "cleanliness" arguments are. The --cli option is not present in 2.0 which means a user cannot write code compatible with 2.0 and 2.1 without a magic environment variable. Maybe 2.1 should be 3.0?

[dra27]

@c-cube: there is no such thing as "write code compatible with 2.0 and 2.1" - your code is either for 2.0's CLI or for 2.1's. There's nothing magic about the environment variable - every single opam command line option is, and always has been, exposed this way.

[avsm]

More specifically, I'd like our 2.1 beta users to report on CLI changes that are proving problematic aside from opam config var. Are there any other scripts breaking for other CLI changes?

[dra27]

This has been reworked in #4575 and so should be fixed with the release candidate.

By default you'll now see:

dra@thor:~/opam$ ./opam config var bin
[WARNING] var was deprecated in version 2.1 of the opam CLI. Use opam var instead or set OPAMCLI environment variable to 2.0.
/home/dra/.opam/dev-4.11/bin

the deprecation can be silenced by setting OPAMCLI=2.0 (as advised):

dra@thor:~/opam-2$ OPAMCLI=2.0 ./opam config var bin
/home/dra/.opam/dev-4.11/bin

and the syntax is now an error only with --cli specifically set to 2.1+:

dra@thor:~/opam$ ./opam --cli=2.1 config var bin
opam: var was removed in version 2.1 of the opam CLI, but version 2.1 has been requested. Use opam var instead or set OPAMCLI environment variable to 2.0.