Next, Feature request: opam switch pin

[Drup]

See #2531 for the previous part.

In order to avoid doing

% opam switch install 4.03.0+trunk
% opam pin add ocaml source/ocaml

which compiles the compiler twice, I would like to be able to say

% opam switch pin ocaml.4.03.0+patate source/ocaml

[dbuenzli]

What configures what is installed in a switch in next ?

Before overloading the command set in a manner that will certainly not scale (you may want to pin more than one package), I think this should also be designed in conjunction with local switches @seliopou has been working on.

For those some kind of .opam_config file will be needed that defines what gets installed in the locally created switch. If you have a pin for a package in that config file it should override the switch creation definitions, and will solve this problem. So for a global switch you'd simply like to be able to specify such an .opam_config file via a command line argument.

[Drup]

@dbuenzli it's in the opam file (the one that configures packages, and now compilers). There is also a file ocaml.config.in that specifies a bunch of variables for compilers. I think your worry is a non issue. It's about pinning what defines the compiler, not about pinning the new packages you are going to install next.

The few things I understood from local switches sounds like it has nothing to do with the current topic.

[dbuenzli]

The few things I understood from local switches sounds like it has nothing to do with the current topic.

Well it has, if compilers become packages, creating a switch should result in an empty shell in which you install packages (some of which may be compilers). Now you certainly want to be able to predefine a certain number of packages to be installed in a switch.

[Drup]

The ability to create a completely empty switch would please me equally.

[Drup]

Actually, I got a doubt: I can do it, there is a new --empty argument !
@AltGr already anticipated my requests. :O

[dbuenzli]

@AltGr do you have user-level documentation about the way switches and their command set work in next ? I'd like to have a look at it.

I have the feeling that it would be worthwhile to introduce an opam switch create NAME [TEMPLATE] command that simply creates a switch named NAME and takes definitions of what to install first in it from a given TEMPLATE if provided and is empty otherwise.

A TEMPLATE would simply be an opam managed, named, opam switch export file (i.e. a set of pin/package constraints) that defines what needs to be installed on switch creation. Beyond this it would be nice if opam could provide commands to manage user-defined templates so that we can easily apply/remove sets of constraints to switches.

Then local switches could be created by specifying an additional --local param to create (or we could default to local and have a --global switch) and the current install SWITCH would simply be deprecated and aliased to create SWITCH SWITCH.

[AltGr]

Sorry, I am lagging behind on documentation, but I'll be glad to explain. The best you can find yet is --help (I should have the manpages published somewhere; I need to find the time for that too.)

Metadata is now below /.opam-switch, and contains:

directories

• backup (backup export files, as before)
• build (where packages are built, unchanged)
• config (config files containing variables set by installed packages as before, plus some optional fields like system files to check for changes, and the global-config.config file)
• install (cached package install files, as before)
• overlay (local definitions for pinned packages)
• packages (cache of metadata of all installed packages)
• packages.dev (mirror of archives and VCs of all installed packages)

files

• environment file containing cached environment variables to be set when in this switch
• reinstall (the list of dev packages that have been changed and should be reinstalled. Recompilation-triggering changes on non-dev packages are detected by comparing their metadata to what's in packages)
• switch-state (single file defining what is currently installed in this switch, what forms the compiler, as well as pins

@Drup: glad you found it!

@dbuenzli I think you can do what you want already with something in the lines of:

opam switch NAME --empty
opam switch import FILE

The import/export format has been much extended: it includes what packages are set as compiler, pins, etc. And it also contains the local definitions of pinned packages (there is even an option to include all packages, but it's not bound in the CLI yet) -- limited by the fact that they must not use a files/ overlay.
The idea is for opam export | opam import to be closest as possible to reproducing the exact same installation.

I don't understand, however, what kind of management of the export files you would want from opam, and to what purpose exactly.

The opam switch command is mostly backwards-compatible: opam switch [install] SWITCH will attempt to create a switch named SWITCH with SWITCH (and its dependencies) as its "compiler" packages. The selection is done among packages that have the compiler package-flag set. You can also use --alias-of or --packages to be explicit about what package(s) should be the base, or even --empty.

To Daniel's delight, I also added a nice retro-compatibility hack so that if the above doesn't work, specifying just a version is also possible (e.g. opam switch 4.02.3)

[dbuenzli]

@dbuenzli I think you can do what you want already with something in the lines of:

Good, but I think the UI and command set needs rework and I wouldn't care too much about backward compatiblity. First opam switch NAME should only be a synonym for opam switch set. I'd introduce a create command rather than have that --empty option.

The opam switch command is mostly backwards-compatible: opam switch [install] SWITCH will attempt to create a switch named SWITCH with SWITCH (and its dependencies) as its "compiler" packages. The selection is done among packages that have the compiler package-flag set. You can also use --alias-of or --packages to be explicit about what package(s) should be the base, or even --empty.

I don't think this is a good model, why would a SWITCH need a compiler package ? You may have an interesting universe that doesn't have such a package. Besides this way of operating still gives a special status to compiler packages something it is better to minimize in general.

It would be much better if SWITCH was simply a reference to an export file SWITCH present in the OPAM repository. This simply provides a named convenience to initialize an empty switch with the content of that export file --- files that I would rename to something like switch (init|status) files.

So

# create an empty switch named NAME
opam switch create NAME 

# create an empty switch NAME and populate it with the defs of SWITCH (export file in 
# the OPAM repo). If SWITCH is a path then read from that file
opam switch create NAME SWITCH

# Backward compat, install is deprecated and behaves as follows.
opam switch install SWITCH = opam switch create SWITCH SWITCH

I don't understand, however, what kind of management of the export files you would want from opam, and to what purpose exactly.

This was discussed somewhere else and in the context of being able to scale the management of local (or "per project") switches. One of the problem of local switches is that there are a lot of things you'd like in all your switches but it's a pain if you have to manage them manually in all of them (e.g. merlin, ocp-indent, etc.). Another use case is when you are working on a set of pins (e.g. uunf, uuseg, uucp, ucdb) that you need to develop/test/release simultaneously. Basically the user need a mecanism to be able to specify, apply and update sets of constraints easily in a set of given switches.

So the rough idea (many operational details would need to be specified) is to add a notion that I'll call, by lack of better name, a mixin. A mixin is simply the ability to specify packages to install with possible constraints (e.g. pin or version number) under a named entity. Adding a mixin (by its name) to a switch applies the constraints of the mixin to the switch and keeps the reference to the mixin in the switch (it's not added if the constraint fail). Updating the constraints in the mixin, tries to update them in all the switches that have a reference to the mixin (failing switches should drop the reference to the mixin, or maybe the whole operation should be aborted). Removing the mixin from a switch simply drops the reference to the mixin, possibly removing constraints aswell (e.g. would be useful to remove pins).

This allows the user e.g. to have a dev-tools mixin that has what she needs for development. The user can then easily add that dev-tools mixin to the switches she creates to setup her working environment. If suddenly she needs a new tool, she can simply add a new package to dev-tools and the switches to which dev-tools were added will update accordingly.

[AltGr]

On the UI:

First opam switch NAME should only be a synonym for opam switch set.

It never stopped being the case. switch set will install the switch if it is not present already.

I'd introduce a create command rather than have that --empty option.

I have nothing against a create command. The basic usage that we want to support for most, and in particular beginners without getting too confusing is to create a switch with some compiler inside. Wanting to create an empty switch kinds of implies that you know what you are doing, so that was my reason to add an option for this case rather than make it the default.

I don't think this is a good model, why would a SWITCH need a compiler package ? You may have an interesting universe that doesn't have such a package. Besides this way of operating still gives a special status to compiler packages something it is better to minimize in general.

It shouldn't impact too much on design, but first, note that the current repository doesn't have dependencies towards ocaml, so this allows to have those implicitely. Of course, they would be easy to add through an automated rewrite, but I pondered this and (i) it would be a bit heavy and redundant and (ii) I am sure we can expect packagers to forget to add these dependencies more often than not, esp. since they weren't needed before.

OTOH, it's a bit awkward to have the consistency of the repository depend on the packages that are set as compilers; and if we consider transitive dependencies through ocamlfind, and constraints on ocaml-version turned into dependencies (#2537), it could be not that heavy.

On switch-templates

I see, so what you wish for is replacing the older mechanism of separate .comp files with preset selections of packages. This is possible manually through the export/import mechanism, but these would be actually provided in the package repository for everyone to use. While I can see some advantages, I am not fully convinced: first, it will annoy me for putting in place repository signing :p.
But also, and more generally, it brings back (some) complexity that part of my goal was to avoid by removing .comp files, and re-adds multiple formats to the repository. For example, this is an issue for proper linting and checking of PRs.

Note that simple bundles have been mentionned already, in the form of virtual packages. These don't provide all the functionality, and in the current state switching to them forces them to be set as compiler, but we could work on that.

On mixins

Interesting ideas, and that seems like it could be useful. However, it seems a bit overkill to me, and adds a whole lot of mechanisms that could be difficult to track (think about conflict messages).
For now, a switch state is made of three sets of packages (ignoring pinnings):

• "installed packages", as the name implies
• "root" packages, i.e. the ones that have been installed manually (the complement being packages installed automatically through dependencies).
• "base" or "compiler" packages, the packages forming the compiler for the switch, which are made (somehow) immutable.

My feeling is that the root package feature is actually not fully used at the moment (maybe because it doesn't correspond to what's needed, to be fair). It's not used to weight packages in the solver, so actually only impacts opam remove -a (remove all unused, non-root packages), and opam import (in case the import is not possible, fallback to trying to import only the root packages).

"root" is always a subset of "installed" at the moment: one idea was to lift this constraint to allow easier recovery from errors (providing a command that tries to reinstall all root packages). "compiler" packages may be uninstalled, in which case any state-changing command will require them to be reinstalled, to recover switch consistency.

One more limitation is that all these sets are packages on specific versions, and can't therefore express acceptable ranges.

Putting the handling of these three sets -- and pinnings -- apart, most of the features you are asking for from mixins could be achieved by using virtual packages and leveraging their depends: field. Maybe a simpler set of features, like somehow promoting a pinned package to allow it to be shared between switches, or providing a shorter path to a local package repository, could fill the bill.

[dbuenzli]

The basic usage that we want to support for most, and in particular beginners without getting too confusing is to create a switch with some compiler inside.

The beginner argument is, as usual, fallacious. People who mention supporting beginners should read more books about usability. You don't want to support beginners, you want to expose a simple conceptual model that both the beginner and the working user are able to quickly grasp, remember and work with pleasantly.

Such a simple conceptual model is: a switch is just a named empty space where you can install packages in isolation from other switches. When you create a switch you can mention a switch init (export) file that will immediately install the packages mentioned therein.

I think that the conceptual model you expose at the moment is mess and won't scale gracefully to encompass further developments of OPAM (e.g. see below).

While I can see some advantages, I am not fully convinced: first, it will annoy me for putting in place repository signing :p.
But also, and more generally, it brings back (some) complexity that part of my goal was to avoid by removing .comp files, and re-adds multiple formats to the repository. For example, this is an issue for proper linting and checking of PRs.

These are technical issues. Favouring technical issues over usability and exposition of a simple conceptual model is not a good idea. Looking beyond, the model I propose adapts very well with the possibility of adding local, per-project, switches. In this case when you do an opam create NAME --local in a directory it would read from a local .opam_switch switch init file the universe to initialize in order to get a working environment for the project.

On mixins
Interesting ideas, and that seems like it could be useful. However, it seems a bit overkill to me, and adds a whole lot of mechanisms that could be difficult to track (think about conflict messages).

Maybe overkill and too complex, if you have better proposals I'm all for it. The problem that needs to be addressed is: an easy and efficient way to move and update package constraints between otherwise isolated switches.

In my opinion neither export in its current state, nor virtual packages, nor introducing a notion of shared pin between switches (which seems very ad-hoc and complexifies the model) fit that bill. The advantage of mixins is that they are somehow just opam switch init files --- which again fits with the conceptual model above --- that are easy to apply and update to arbitrary switches.

Maybe we can drop the idea that switch remember which mixins where applied to, i.e. updates to mixins do not affect switches, that would already be a significant improvement over the status quo. In fact mixins could simply be a command that manipulates switch init files, you have those that are provided by the repo which are immutable and you can create and update your own local init files, e.g. by adding constraints manually, by snapshooting the constraints of a set of packages in a given switch etc.

I know you'll tell me all this can already be done using export files, sure it can be done, but it's all very cumbersome. We need a good UI and conceptual model for this.

[AltGr]

Such a simple conceptual model is: a switch is just a named empty space where you can install packages in isolation from other switches.

Completely agree, and I propose nothing different, I just don't see the need to comfront users to this before they actually need it, esp. since some never will ("­— to get an ocaml installation, you need to setup a switch. — what's a switch ?", vs. "— to test on a different compiler/package set, you need to change switch. You actually were in switch <default> all along.")

When you create a switch you can mention a switch init (export) file that will immediately install the packages mentioned therein.

That indeed would be a useful feature, that should fit in the UI.

I think that the conceptual model you expose at the moment is mess and won't scale gracefully to encompass further developments of OPAM (e.g. see below).

I take it that by this you mean the fact that a subset of packages in a switch is defined as "base", and immutable. It adds some complexity to the model, but I think it's desirable, because you almost never want opam upgrade to bump the OCaml compiler to the latest version and recompiler everything — or an install to downgrade OCaml because it helps with its dependencies. I first attempted to handle this with pins, but they don't really fit the bill and it was actually more confusing.

Or, maybe, you were just complaining about the UI that doesn't perfectly reflect the actual model behind switches ?

These are technical issues. Favouring technical issues over usability and exposition of a simple conceptual model is not a good idea.

Yes and no. Given our limited resources, the consequences will be limited, or absent checks on the extra repository files, leading to possible errors or mistakes in the repository, in turn impacting usability. Having a simpler layout results in easier (hence better) maintainance in opam but also the repository. I drive these conclusions from the experience of .comp files, that were much below opam files in terms of tooling, and since deciding to drop them I have seen plenty of examples on this tracker that made me glad I took that decision. The checksums of compiler archives weren't even properly checked, for example (!)

Also, it seems to me that, regarding the global model, what you propose would move complexity around, not reduce it.

Looking beyond, the model I propose adapts very well with the possibility of adding local, per-project, switches. In this case when you do an opam create NAME --local in a directory it would read from a local .opam_switch switch init file the universe to initialize in order to get a working environment for the project.

This sounds great indeed. And I am convinced that something with the functionality of opam switch create --import-file is a must-have as well. What I am not sure about is holding these files in the repository.

Maybe we can drop the idea that switch remember which mixins where applied to, i.e. updates to mixins do not affect switches, that would already be a significant improvement over the status quo.

I really get the feeling that virtual packages could fit that bill quite gracefully, so would prefer to avoid adding such a complex mechanism. I am speaking from the implementation point of view: this is likely in the domain of "doable but not at all usable" for now, but there is a world of difference between adding a core mechanism and adding usability functions to make a given workflow efficient, while using existing core mechanisms, which I would be glad to do.

The real limitation is that by leveraging the depends: field of a package, you can only specify constraints on what you want installed, not what you want as compiler, or as roots, or pinned. Do you plan to need these ?

Also, again not usable without tooling, but an export file can now include pinnings and their definition, which may contain such virtual packages. E.g. you would get something quite close to a mixin with an export file like:

installed: "foo"
roots: "foo"
pinned: "foo.1.0"
package "foo" {
  version: "1.0"
  depends: [ selection of packages with constraints here ]
}

[dbuenzli]

Completely agree, and I propose nothing different,

Well, this:

The opam switch command is mostly backwards-compatible: opam switch [install] SWITCH will attempt to create a switch named SWITCH with SWITCH (and its dependencies) as its "compiler" packages. The selection is done among packages that have the compiler package-flag set.

is completely different from what I propose and completely ad-hoc from my perspective.

I just don't see the need to comfront users to this before they actually need it, esp. since some never will ("­— to get an ocaml installation, you need to setup a switch.

I don't care if you create a switch on init (though I wouldn't) I care about how you create switches.

I take it that by this you mean the fact that a subset of packages in a switch is defined as "base", and immutable.

No I was actually not thinking about this. But now that you mention this it seems suspicious to have that limitation.

Or, maybe, you were just complaining about the UI that doesn't perfectly reflect the actual model behind switches ?

Yes.

This sounds great indeed. And I am convinced that something with the functionality of opam switch create --import-file is a must-have as well. What I am not sure about is holding these files in the repository.

Basically you'll put the burden of managing these files on the user, separately from the opam tool, which is going to be very painful from a user perspective, especially if you'd like to be able to share switch setups. It's a little bit disappointing since you already have most of the functionality in opam itself at the moment to make a great UI with a good conceptual model that will gracefully scale to new developments (local switches, and better tools for propagating constraints across switches).

I really get the feeling that virtual packages could fit that bill quite gracefully, so would prefer to avoid adding such a complex mechanism.

TBH I don't see where the complexity lies here, it's managing a set of export files inside the opam tool and be able to apply them to a given switch. You already have all the pieces in place. Leaving that outside OPAM to the user with ad hoc --import-file options is not going to provide a good user experience at all.

The real limitation is that by leveraging the depends: field of a package, you can only specify constraints on what you want installed, not what you want as compiler, or as roots, or pinned. Do you plan to need these ?

I don't plan to need these, I need them now. When I have ~10 pins and I need to test these in another switch, it's a very painful experience. Now you can invent ad-hoc schemes about pins that are in multiple repos or whatever virtual packages to configure switches, nothing of this strikes me as being a good idea if you consider that the problem here is simply being to manipulate and propagate switch state by defining a set of package and their constraints/pins. This problem is exactly what export files are about, so if you give them first-class support in OPAM you solve a lot of problems (switch init, state/constraint propagation across switches, local switch init) in a uniform way and with a good conceptual model.

[Drup]

So, given the possibility of creating empty switches and the new scheme for ocaml packages, I don't think this is an issue anymore. @AltGr I'll let you close as you see fit, since there is another discussion going on.