[Feature Request] Provide a way to upgrade & install (build dependencies of a pinned project) at the same time

[RalfJung]

First, some context: For a CI build, I want to build whatever is in the git commit that is tested. There is an opam file at the root of this repository, documenting the build dependencies. The opam root is cached between CI builds so that build dependencies are re-built only when necessary.

As part of this, I am trying to accomplish roughly the following: I want to say "Install the build dependencies given in that opam file, and at the same time upgrade everything that's already installed to make sure we got the latest version".
Similarly, I'd like to provide a make target "build-dep" to the user that will install and upgrade build-dependencies in their opam.
(The reasons upgrade may be necessary is that the exact version of the build dependency is expressed via pins handled separately. Depending on how #2762 is handled, this may no longer be necessary.)

Currently, the best way to achieve this I can think of is:

	opam upgrade
	opam pin add coq-lambda-rust "$$(pwd)#HEAD" -k git -n -y
	opam install coq-lambda-rust --deps-only
	opam pin remove coq-lambda-rust

This has some problems.

One minor problem is that I have to pin, install build-dependencies, and un-pin again. If the user had their own pin of this project, that would be lost on the way.

And finally, what is more annoying is that it may compile packages twice: Since upgrading and installing build dependencies is done in two transactions, it could happen that the opam install has to re-compile a package that opam upgrade also had to re-compile.

One way to fix this would involve two orthogonal changes -- but maybe they make little sense and the goal should better be achieved in a different way, which is why I reported this as one feature request.

• opam install could gain a flag --upgrade to also perform an upgrade (or vice versa, opam upgrade could gain a flag to also install [the build dependencies of] something).
• opam install could take a path to operate on, instead of a package, so that one could write opam install <local dir> or opam install <URL>#<HASH> --deps-only.

[AltGr]

Thanks once again for the detailed report and suggestions.

There are several points to consider here:

1. opam install --upgrade: It'll start to sound repetitive, but opam 2 adds this option :). What it does is upgrade packages listed on the command-line if not already installed, though, so not quite what you want. However, install, upgrade, etc. have been made more flexible to allow for different situations, e.g. opam upgrade will prompt and install a package if not already there ; it's not compatible with --deps-only, though, and adding it would make the CLI semantics confusing IMHO.
2. it's not yet there, but handling the dependencies directly from an opam file appears in a few common workflows, so that should make it into 2.0. Something like opam install --dependencies-from-file <opam file>. opam show has already been extended to allow querying a file, so you can do opam show --file <opam-file> -f depends:, but that is difficult to automatically interpret and feed back into opam.

So, until 2. is implemented, you are stuck with having to pin the package temporarily — you could ease the issue you mention by pinning under a temporary name, but that'd be an ugly workaround.

However, there is a trick to resolve the main issue, that works with opam 1.2: opam install by default tries to install your package while minimising changes, but you can actually customise the solver criteria used.

If you use opam install coq-lambda-rust --deps-only --criteria="-removed,-notuptodate", opam will perform the installation while using upgrade-like criteria, which will give a result equivalent to an upgrade. This, of course, requires that opam is installed with an external solver, but that is very highly recommended anyway.

[RalfJung]

It'll start to sound repetitive, but opam 2 adds this option :)

That's great :D

What it does is upgrade packages listed on the command-line if not already installed, though, so not quite what you want.

Well, I don't actually need all packages to be upgraded, just the entire recursive dependency chain from the things that are listed. So if I could do opam install --upgrade-recusrive --deps-only package, that would be perfect.

it's not yet there, but handling the dependencies directly from an opam file appears in a few common workflows, so that should make it into 2.0. Something like opam install --dependencies-from-file . opam show has already been extended to allow querying a file, so you can do opam show --file -f depends:, but that is difficult to automatically interpret and feed back into opam.

Why should this be restricted to installing dependencies from a file? Is it because installing the entire thing from a file is already covered by opam pin add $DIR? IMHO it would make sense for opam install $DIR to be essentially equivalent except that the pin is immediately removed again, and then opam install --deps-only $DIR would work as expected.

If you use opam install coq-lambda-rust --deps-only --criteria="-removed,-notuptodate", opam will perform the installation while using upgrade-like criteria, which will give a result equivalent to an upgrade. This, of course, requires that opam is installed with an external solver, but that is very highly recommended anyway.

Wait, so setting -notuptodate will instruct opam to also make everything up-to-date? Wow, I did not expect the criteria to be so flexible. I will try this (and also try using a random obscure name for the pin), thanks!

[RalfJung]

If you use opam install coq-lambda-rust --deps-only --criteria="-removed,-notuptodate", opam will perform the installation while using upgrade-like criteria, which will give a result equivalent to an upgrade. This, of course, requires that opam is installed with an external solver, but that is very highly recommended anyway.

Wait, so setting -notuptodate will instruct opam to also make everything up-to-date? Wow, I did not expect the criteria to be so flexible. I will try this (and also try using a random obscure name for the pin), thanks!

This did not work, unfortunately. opam upgrade will also downgrade packages to satisfy a pin, which --criteria="-removed,-notuptodate" does not do. (The reason I want downgrades is that the versions are of the form dev.<git commit SHA1>, so as the development progresses, sometimes opam sees a smaller version number, and sometimes a larger one.)

[AltGr]

Why should this be restricted to installing dependencies from a file? Is it because installing the entire thing from a file is already covered by opam pin add $DIR? IMHO it would make sense for opam install $DIR to be essentially equivalent except that the pin is immediately removed again, and then opam install --deps-only $DIR would work as expected.

The issue is that, by implicitely removing the pin just afterwards (assuming -n), you would get in a non-reproducible state, i.e. the package has been installed but opam hasn't kept track of its source. We try to avoid getting into this situation implicitely, because it means that, for example, an upgrade that touches any of the dependencies would lead to the package being removed (it would normally be recompiled, which is not possible). Keeping the pin seems to be a more elegant solution in this case. Of course, with --deps-only, the problem disappears.

This did not work, unfortunately. opam upgrade will also downgrade packages to satisfy a pin, which --criteria="-removed,-notuptodate" does not do. (The reason I want downgrades is that the versions are of the form dev.<git commit SHA1>, so as the development progresses, sometimes opam sees a smaller version number, and sometimes a larger one.)

That is strange, the solution proposed by opam should first guarantee consistency (respect your strict dependencies), then only optimise according to the criteria. If you can reproduce, could you try running the opam install command with --cudf foobar and send me the generated foobar-* files so I can check what happens ?

NB: "-removed,-notuptodate,-changed" are the default upgrade criteria, that also minimise changes, all other things equal. Or, for the more advanced version (aspcud 1.9 only) that opam uses internally: "-count(down),-count(removed),-sum(solution,version-lag),-count(new)"

[RalfJung]

The issue is that, by implicitely removing the pin just afterwards (assuming -n), you would get in a non-reproducible state, i.e. the package has been installed but opam hasn't kept track of its source. We try to avoid getting into this situation implicitely, because it means that, for example, an upgrade that touches any of the dependencies would lead to the package being removed (it would normally be recompiled, which is not possible). Keeping the pin seems to be a more elegant solution in this case. Of course, with --deps-only, the problem disappears.

Ah, so opam doesn't independently just remember "this was the opam file I used to install that software", it instead rederives that information from the installed version and pins where necessary?

That is strange, the solution proposed by opam should first guarantee consistency (respect your strict dependencies), then only optimise according to the criteria. If you can reproduce, could you try running the opam install command with --cudf foobar and send me the generated foobar-* files so I can check what happens ?

Yes, it is easily reproducible. You got mail.

[AltGr]

Sorry for the delay in getting back to you. And thanks for the data!

Ah, so opam doesn't independently just remember "this was the opam file I used to install that software", it instead rederives that information from the installed version and pins where necessary?

Opam 2 does, actually; but at the moment we try to avoid reinstalling packages that no longer have an upstream, because there might be a reason for that (package removed from the repository because its upstream died, it has security issues, etc. ; also, if you unpinned a package that existed in the repository, you would expect to get back to the repository version upon reinstall...)

Yes, it is easily reproducible. You got mail.

Ah, I understand: I thought you had a specific dependency from coq-lambda-rust to the specific git version of coq-iris, but that is not the case, so the returned state is still "consistent" in opam terms. Here you end up with two versions of the coq-iris package, the installed one and the pinned one: opam will remember to switch to the pinned one at the first opportunity, but that's actually outside the scope of the solver.

"At the first opportunity" may not be so soon, though: we put effort in not having opam install perform pending changes unrelated to your request, because that was generally annoying; and in this case the opam install coq-lambda-rust is considered unrelated (the coq-iris package not being in the cone of package that depend on what you requested).

The problem is the same with the current opam 2 opam install --upgrade foo: it will install foo, or upgrade it if not installed¹, but not run a general upgrade (opam upgrade foo is quite different from just opam upgrade).
I myself have hit a similar case (in a similar batch-install setup) where I wanted to do a general upgrade, while ensuring that some packages would be installed, so I think there should be an option for that (there are no special difficulties implementation-wise).
To sum up, now we have:

1. opam install --upgrade foo bar: installs (or upgrades) foo and bar
2. opam upgrade: upgrades all
3. opam upgrade foo: upgrades (or installs) foo and bar

Where 1. and 3. are mostly equivalent; Another option should be added for "upgrade all AND install foo and bar"
That option could be:

• opam upgrade --all foo bar: like opam upgrade, but ensure a global upgrade is done rather than limit it to foo and bar.
• a replacement of 1. above, opam install --upgrade foo bar, since that command is duplicate. We could still change the semantics since that option hasn't been released yet. (3., on the other hand, shouldn't be changed now)
• or, if we want to keep 1., something like opam install --upgrade-all foo bar.

¹ I actually found a bug, that I just fixed, that made opam install foo --upgrade not change to the pinned-to version if that was a downgrade. opam upgrade foo worked correctly there.

[AltGr]

Note: a bit unrelated, but you may want to try versionning your git package with the output of git describe (it looks like 2.0-alpha5-147-g4a83e9f): assuming your tags are ordered, and you are following a branch, that would allow you to get well-ordered opam versions while keeping all the git details.

[AltGr]

#2797 adds opam upgrade foo --all, that will run a global upgrade and install foo (or make sure it's not removed). It also removes opam install --upgrade. The bug mentionned above is fixed by #2795

[AltGr]

Thanks a lot for reports, use-case and suggestions.
The "install deps of opam file" is not there yet, but will be soon. Closing.

[RalfJung]

Ah, I understand: I thought you had a specific dependency from coq-lambda-rust to the specific git version of coq-iris, but that is not the case, so the returned state is still "consistent" in opam terms. Here you end up with two versions of the coq-iris package, the installed one and the pinned one: opam will remember to switch to the pinned one at the first opportunity, but that's actually outside the scope of the solver.

Right, we can't have coq-lambda-rust depend on a specific commit of coq-iris as we can only depend on a version in the opam file and we can't have a new version for every commit. That's why we have the opam.pins file to express the dependency on a particular commit.

#2797 adds opam upgrade foo --all, that will run a global upgrade and install foo (or make sure it's not removed). It also removes opam install --upgrade. The bug mentionned above is fixed by #2795

Great, thanks :)
However, I guess with this being added to "upgrade", we can't say "upgrade all and install build-deps of X"?

[RalfJung]

I think this issue should be reopened: it is not actually possible with opam 2 to do what I asked here. Namely, opam upgrade -a foo will not install foo if that requires downgrading other packages -- see #3737.