[RFC] How to handle versions

[ghost]

This ticket is to discuss how version numbers should be handled by Dune. It's still not clear to me what the best way to deal with versions is.

Problems

1. how should Dune decide what the version of a package present in the workspace is?
2. how should Dune decide what the version of an installed package is?
3. how should this information be communicated to opam? in particular when pinning a package
4. how does this interact with incremental compilation?

Current situation

For (1), Dune uses the version field specified in the <package>.opam file if any, with a fallback to some other file such as <package>.version or VERSION. It never uses the output of git describe ... except when running dune subst. Some projects such as js_of_ocaml or ocamlformat use the OCaml syntax to use the output of git describe ... in development builds. We could integrate this idea properly in Dune, however there is an issue with both (3) and (4).

For (2), Dune currently doesn't know the version of installed packages. Since it always installs the <package>.opam file as <libdir>/<package>/opam, it could simply read it from there.

For (3), the problem is described here: ocaml/opam#2932

For (4), the problem is the following: if the version changes every time we commit something, then Dune might end up rebuilding things and re-running tests on every commit.

[jberdine]

In case it is useful, note that much of the hacky complexity of building and releasing ocamlformat is due to versioning. Maybe there is a simpler way with current tools, but see the current release procedure which involves a bunch of git manipulation, jbuilder subst, and multiple rounds of topkg tag. Related to this is the script that generates version strings. Note that there are 3 distinct cases: pinning with opam, building a distribution archive, and building in a git dev tree. For reference, this script is used in the jbuild rule for the version file, and in the opam file when pinning. It seems like there should be a better way.

[ghost]

Thanks for the pointer. Looking at ocamlformat, I assume that during development the convents of Version.ml changes after every commit. Do you find that to be an issue in practice? I expect that for a project like Dune itself with a lot of tests that would be annoying as the dune binary would change and you'd have to re-run all the tests.

BTW, I notice that for pinning you let opam decide what the version is and use this version in ocamlformat. What I had in mind was the opposite: the package decide what the version is and communicates this version to opam. The latter seems more natural to me.

[jberdine]

Yes, the version changes with every commit. That is a necessary feature of the build scripts. It's important to be able to distinguish between different "unreleased" versions to make sense of the results of long-running tests done across various machines on various versions simultaneously. This is more important in other projects than ocamlformat, as ocamlformat's tests are short and can be done locally.

I would be happy for the package to tell opam the version instead of vice versa. It wasn't clear to me how to do so when I wrote these scripts. I could easily have overlooked something though.

[ghost]

Yes, the version changes with every commit. That is a necessary feature of the build scripts. It's important to be able to distinguish between different "unreleased" versions to make sense of the results of long-running tests done across various machines on various versions simultaneously. This is more important in other projects than ocamlformat, as ocamlformat's tests are short and can be done locally.

That makes sense. I was thinking in terms of a big mono-repo but in general developers work on smaller git repositories so the impact is limited. At worse we could add an option to control this behavior.

I would be happy for the package to tell opam the version instead of vice versa. It wasn't clear to me how to do so when I wrote these scripts. I could easily have overlooked something though.

It's indeed not possible at the moment to have pinned packages decide of their version.

[ghost]

We talked with @AltGr about the versioning of pinned packages, and it should be possible to do something in opam to allow using git describe ... as version number.

[rgrinberg]

@jberdine are you satisfied with the current versioning functionality in dune?

[jberdine]

@rgrinberg, by and large, yes. One point that has come up however is that some additional flexibility over raw git describe would be useful. For example, it would be useful to be able to generate package version numbers containing ~ to indicate (unreleased) versions since the last tagged release (e.g. using 1.5~10 for 10 commits since 1.5 was tagged). Since ~ can't be used in git tags, there isn't a clear way to do this. I guess that the high level point is that it would be good if the core versioning functionality in dune could exercise the functionality that is in opam. The issues with ~ is one (maybe the only?) instance of that I have run into where I couldn't use what opam could do easily from dune.

[rgrinberg]

@jberdine Thanks, that sounds reasonable. Could you make a new ticket for this issue and we can discuss this in the next dune dev meeting? The reason I'm asking for a new issue is that we go over all new issues during this meeting so this way it will not be forgotten. If that's the only outstanding issue, I'm closing this ticket.

[jberdine]

Sounds good, thanks. I've opened #3856 for this.