Next: ocaml-version not updated with ocaml's version

[Drup]

I created an empty switch last and installed ocaml in it manually. It seems ocaml-version is not udpated.

% opam install merlin utop tuareg    
[ERROR] utop has unmet availability conditions: ocaml-version >= "4.01"
[ERROR] merlin has unmet availability conditions: ocaml-version >= "4.00.0"
% opam switch show
last
% opam list -i
base-bigarray base   Bigarray library distributed with the OCaml compiler
base-threads  base   Threads library distributed with the OCaml compiler 
base-unix     base   Unix library distributed with the OCaml compiler    
ocaml         4.03.0 Official 4.03.0 release         

[AltGr]

Ah, this part is lacking documentation, and there may be some rough edges yet to polish, so let me explain how it works at the moment (on next). And thanks again for testing ☺

How it works

So, compilers now being in packages, compiler-specific variables (like typically ocaml-version) can not generally be defined at switch creation time anymore. However, packages have been able to export opam variables since opam 1.0 by providing a <pkgname>.config file at their root (similar to .install; but although this is an older feature it probably lacks documentation too). These variables, however, are scoped with the package name to avoid clashes: if you have foo installed, you can always access foo:lib to get its lib directory (<prefix>/lib/foo normally), but also foo:bar if foo defined bar in its configuration file.

The idea, then, to keep compatibility with the previously global ocaml-* variables, is that for packages installed as compilers (i.e. "base packages" having the compiler flag set), the variables they define are re-exported in the global scope of the switch. So the ocaml package just has to define an ocaml-version variable through its .config file.

It could be much simpler to define these values in the compiler's opam file, but that would be assuming they can be statically defined. In particular, in the case of the system compiler, the package build is simply a polling script that will generate the configuration, and we don't know the OCaml version in advance (so ocaml-version and ocaml:version can be different, with the latter being system in this case). Even for compiled OCamls, we might not know in advance whether native-dynlink is available.

In your case, ocaml was installed as a "normal" package, so its variables weren't re-exported: look at files in <switch-prefix>/.opam-switch/config, global-config.config is where global variables are defined, the other files are per-package. Like we have opam install --[un]set-root, we probably need expert commands that allow adjusting the defined compiler in the current switch (you can edit <prefix>/.opam-switch/switch-state easily, but that won't do the variable export part, and is no excuse for not having it in the CLI anyway).

Now let's get to the funny details...

The reason to re-export in the global scope rather than use the variable directly (after all, we could replace uses of ocaml-native-tools with a scoped variable ocaml:native-tools, and even ocaml-version with ocaml:real-version or something like that) is that there is a difference between the two:

• package variables are only available in scripts, i.e. for actions happening after the solver solved
• global variables can be used anywhere, and, most importantly, in the available: field, which is used to filter the packages before solving, and therefore shouldn't depend on what packages are installed.

Obviously, ocaml-version is a variable we absolutely want accessible in the available: field... so we needed to have it in the global scope.

What this implies, however, is that it's inconsistent to change your compiler and non-compiler packages in the same set of actions: ocaml-version may be defined incorrectly. Currently, we do not handle this correctly, however it could happen on:

1. upgrade of the system compiler
2. manual change of the switch's compiler (e.g. through pinning)

I have a plan to solve this situation, which requires solving once, limiting the actions to the removals and compiler change, processing them, then re-running the solver in the new environment to complete the expected actions. It's a bit complex to my taste, however, and means that, by design, the user must validate (and start removing packages) before being able to know what the expected end state will be. May be acceptable for a compiler change, though, in 1.2.2, case 1. required a manual, full switch reinstall (that could fail due to broken constraints, I think), and case 2. was impossible altogether.

In the current state, the reinstallation of packages following the switch change could be tried on package unavailable for the new version and fail, and the user will need to re-run opam with a proper request to get a correct solution and retry to install them.

Another solution (or not)

Now, if you expected something simpler, you can skip this. More elegant, maybe, but not simpler. Also (spoiler), this doesn't actually solve the problem of dynamic system packages.

Generally speaking, it would be nicer to be able to write, instead of
available: ocaml-version >= "4.01"
something in the lines of
depends: [ "ocaml" >= "4.01" ]

This allows to pass all the details to the solver, and gets rid of these annoying package/global variables needed for available:. One issue is that currently, we have a mismatch between OCaml versions, and the ocaml package versions, which also encode information on the variant (4.01.0+BER, system).

The idea here is to rely on the provides: feature, which allows a package to be available under different names besides its own. This brings a whole lot of difficulties on its own, but that will be for another time -- I have lots on notes on this too. For the time being, let's assume it just works.

Using this, we could have the variants defined in different packages (e.g. ocaml+BER), and "providing" the ocaml feature with version 4.01.0.
This also works for other compiler features: we currently use two mechanisms, base- packages (e.g. base-threads) that we depend on, and ocaml- variables (e.g. ocaml-native-dynlink) that we use in scripts (and possibly, the available: field). Here a given compiler could "provide" native-dynlink and threads as packages, and other packages directly depend on that. It also seems much more appropriate on a package metadata point of view, a given compiler implementation providing a set of features.

What this doesn't solve at all, however, is the system compiler issue, since these provides: must be defined statically

Dynamic packages

We've drifted a bit, but the bulk of the issue actually boils down to this: dynamic packages, and in particular, definition of the system compiler.

• in 1.2.2, polling and generation of a compiler description, as well as further polling to detect changes, were hard-coded in opam
• in 2.0~alpha, there is no OCaml specific code in opam, the polling is done by the package itself, and the compiler-package-variable-made-global feature bridges the gap to the package selection

There may be middle-grounds or alternate solutions:

• somehow allow the definition of dynamic packages in the repository (seems complex, costly, and most importantly very difficult to do without compromising security -- any dynamic package would need to do its polling before it was selected by the user in any way)
• keep the hard-coded generation, but put it in an ocaml plugin added to the ocaml-agnostic opam.

Note that having the system package in the repository has some benefits; for example, in 1.2.2, it is impossible to change its set of base packages, which raised some difficulties when base-ocamlbuild started being needed.

---

Ok, this got quite long, I am sure you didn't expect that in your report. I'll forward to the ML since there is worthwile design discussions to be had.

[Drup]

(answering here, https://lists.ocaml.org/ is down)
I didn't expect that, indeed. :)

It's a bit complex to my taste, however, and means that, by design, the user must validate (and start removing packages) before being able to know what the expected end state will be.

As a user, I wouldn't like that. I know it sounds a bit greedy, but if compilers are packages, I want my compiler upgrade to be like a package upgrade. Consider the recent move to 4.03.0: If I decide to upgrade from 4.02.3 to 4.03.0 (with "opam upgrade ocaml"), but I see that this upgrade will remove batteries and oasis, I can decide not to upgrade just yet. It's an important information.

Another solution (or not)

Is it already possible to write ocaml > 4.03 (in next) ? If so, then we have a problem, and this solution is better. Having two (three ?) ways of putting constraints on ocaml versions that have subtle different semantics sounds like it's going to end in a disaster. :)
Also, in general, I think this solution provides a better UX. Everything I know about packages will apply to compiler-as-packages, including stuff about version constraints. It also looks like I could have richer constraints (by mixing constraints on packages and compilers).

I have no proposal to solve the issue with system, but note that we have a similar problem with the llvm package. It uses the system's llvm installation, and the version opam picked and the system's version don't match, it fails during install. In general, conf-* packages have similar issues, so having a solution that is not specific to compilers is worthwhile.

Here a given compiler could "provide" native-dynlink and threads as packages, and other packages directly depend on that.

This is also something that I find desirable, to handle the various alternative compilers, such as metaocaml. A package could depend on base-meta.

[dbuenzli]

In agreement with @Drup and the idea of not making compiler package too different from other packages and avoiding having many ways of specifying the same thing.

Regarding the system problem TBH I wonder if it something that should be supported in the long term. It seems to bring in more problems than it solves. For example we have quite a few install bugs that are system specific and we'll always be subject to the way system packagers decide to layout things which makes it harder from a support point of view than the ones we control directly in switches. Even worse it's often beginners that end-up with system switches because it seems convenient and less wasteful, but it actually provides them with the most brittle and less tested environment which can result in a sub-par initial user experience.

A possible solution to retain the possibility of having a system compiler but relying solely on an ocaml package and its version constraints would be to have mecanism to express user package configuration desires (e.g. something like the package input variables of #2247). This could be used to configure the ocaml package to request a system compiler before installing it. If that is the case the install procedure of opam install ocaml.X.Y.Z would simply check that there's indeed a reasonably working ocaml vX.Y.Z environment in the executable PATH rather than try to compile one.

[AltGr]

We seem to have the same feeling about this...

Having a package that has an effect on the set of available packages that can't be determined before the package is actually installed is indeed incredibly awkward, and that's also true from opam's point of view.

A compromise, related to what @dbuenzli suggests, and that I failed to mention above, would be to have a variant of ocaml "system" compilers for every major version, avoiding the dynamic aspect, e.g. ocaml.4.02.3+system. These packages would just do the polling on install, and fail if the appropriate OCaml version is not present in the PATH. They could even detect a different version and output the correct opam package to install.

It's not best, but this doesn't have any of the drawbacks mentioned above.

Is it already possible to write ocaml > 4.03 (in next) ? If so, then we have a problem, and this solution is better.

Yes. The current scheme is ocaml.<version>[+<variant>], so you would actually be ok in this case. But not if you wrote e.g. "ocaml" = "4.03.0". For all these reasons I really want to have provides: as soon as possible (I already made an experiment with the inverse feature, provided-by:, but it was no good). I shall open a new issue with the current state of the design. The main problem is to avoid that the solver may arbitrarily choose one implementation among several providers, and relates to the plans for signing the repo.

[AltGr]

The one very important use-case we have and that this would break is actually quite simple: I think that if we find any way to handle it it's ok:

apt-get install ocaml opam
opam init
eval `opam config env`

(and similarly for other distributions / OSes ?)

• An ocaml plugin that generates the dynamic package ocaml.xxx+system would do
• Allowing the packaged opam to come bundled with a definition for the ocaml package, since at the OS repository level we have all the information we need ?
• if depextwas included into opam, we could use it to filter available compiler packages in advance on opam init and opam switch (using only its polling functionality, not the installations). On the other hand, it's good to be able to evolve depext at a faster pace than opam...
• any other ideas ?

And the problem of dynamic packages isn't really closed for e.g. llvm...

[dbuenzli]

The one very important use-case we have and that this would break is actually quite simple

I'm not sure I understand the problem here. What is this supposed to do ? Does it already create a switch with OCaml installed ? If that's the case, it should simply be deprecated.

For me if compiler become packages, creating a switch should always result in creating an empty shell or universe (see #2532 (comment)) --- we can have switch .init files but that's just icing on the cake. So if you want to install the system you simply opam install ocaml.4.03+system and the package ocaml.4.03 has the logic for the +system bit. More precisely I don't see myself to be able to do any work before having done:

apt-get install ocaml opam
opam init
opam switch create bla
eval `opam config env`
opam install ocaml.4.03+system

[Drup]

For me if compiler become packages, creating a switch should always result in creating an empty shell or universe

I don't think that's a good idea. Beginners don't know about switches, and I don't think exposing them to that concept directly is a great idea. @AltGr's scenario seems desirable.

And the problem of dynamic packages isn't really closed for e.g. llvm...

Note that llvm is not really dynamic, and it pretty much implements your solution for system: multiple packages, and each does polling at configure time.

[dbuenzli]

I don't think that's a good idea. Beginners don't know about switches, and I don't think exposing them to that concept directly is a great idea. @AltGr's scenario seems desirable.

I disagree. First it's better not to have beginners use the system switch. Second it seems that most people coming from other environments are actually surprised by OPAM's global switch behaviour and do expect a notion of switch --- albeit a local (per directory) one. Do not try to hide things from beginners, it's contra-productive. See also my proposals in #2532.

[AltGr]

I agree with @dbuenzli on the usefulness of empty setups and switches, and that's why I spent some efforts to make sure they were possible (opam 1.2.2 simply can't work without at least one switch, and without a compiler in every switch).

Regarding beginners, there are actually two separate issues:

• having a base setup in as few commands as possible is good, and I don't think anyone who had to have a classroom get it working will disagree. This doesn't feel like hiding anything to the user, just providing a sensible default setup on init -- and the initial switch doesn't have anything special afterwards, so forcing the beginner to enter more commands only for pedagogic purposes seems dubious to me. On the other hand, people not wanting this init behaviour are generally the ones who have read some doc and know how to get an empty setup.
• there are plenty of tutorials on the web that advise a given set of commands. If possible and not too costly, we should really avoid breaking them, as it's sure to bring a lot of frustration.

On using the system switch by default, I don't have a strong opinion: it could also be the latest official release. There is a cost in build time, but system switch install can fail (already) if there is no OCaml installed on the system. And not relying on the system switch automatic detection for init removes a lot of weight from the decisions to be taken regarding the conversation above and dynamic packages. I believe @avsm, @samoht, @OCamlPro-Henry should have a say in this.

[yallop]

It also looks like I could have richer constraints (by mixing constraints on packages and compilers).

This could be rather useful. Here's a situation that came up recently that seems tricky to solve with current OPAM:

• js_of_ocaml optionally depends on ppx_deriving
• js_of_ocaml builds successfully on both OCaml 4.02 and OCaml 4.03
• with ppx_deriving installed, js_of_ocaml builds on OCaml 4.02 but ​not​ OCaml 4.03

(Advice on how best to encode this set of constraints with OPAM 1.2.2 is welcome.) With compilers-as-packages, perhaps it could be written like this:

conflicts: [
  ("ppx_deriving" & "ocaml" {>= "4.03.0"})
]

[AltGr]

Something that I forgot to mention earlier: you can actually easily set the switch-global variable ocaml-version(and others) by hand easily through: opam config set ocaml-version VALUE

@yallop the conflicts field only allows a disjunction of packages — this is a limitation of the underlying solvers, conflicts being the first source of complexity in solving installation requests. So I am afraid that, even with 2.0, you can't express a conflict with the conjunction of ppx_deriving and a specific ocaml version (without hacks¹).

The best way to handle this at the moment, imho, is to disable support of ppx_deriving on 4.03.0 rather than have a conflict (it's also more flexible for the user). This was the idea behind the experimental features: field: (example compatible with 1.2.2)

build: [
  ["./configure" "--%{ppx_deriving_support:enable}%-ppx-deriving"`]
 ...
]
features: [
  "ppx_deriving_support" "Include support for ppx deriving"
    { ppx_deriving:installed & ocaml-version < "4.03.0" }
]

What was missing from this experiment was the ability to then depend on the feature (#2499), since depending on both js_of_ocaml and ppx_deriving is not enough anymore. What 2.0 could allow us to do is replace the filter-expression based on variables { ppx_deriving:installed & ocaml-version < "4.03.0" } with a package expression { "ppx_deriving" & "ocaml" {< "4.03.0"} } that has the huge avantage that it can be understood by the solver (⇒ depends: "js_of_ocaml/ppx_deriving_support" would become possible)

In 1.2.2, you may be able to get the above without using features: with something like:

[ "./configure"
    "--enable-ppx-deriving" { ocaml-version < "4.03.0" & ppx-deriving:installed }
    "--disable-ppx-deriving" { ocaml-version >= "4.03.0" | !ppx-deriving:installed } ]

¹ like expressing conflicts in depends through a virtual no_ppx_deriving package with the corresponding conflict: depends: "ocaml" {< "4.03.0"} | "ocaml" {>= "4.03.0"} & "no_ppx_deriving"

[samoht]

(for the hack in the conflicts, see how we use the ugly mirage-no-xen package and its use in nocrypto)

My only worry about that discussion is that I'd like to keep a fast way to install and run opam and ocaml, and that's usually a job that binary distributions do very well -- hence the need of the system switch. They are other means to do that properly (e.g. not the way opam_installer.sh does it) but if we decide to kill the system switch we should have valid alternatives ready.

[carnotweat]

HO w about a package.env/include/ etc like gentoo ?