Update ocaml-base-compiler, ocaml-system and ocaml-variants 4.13.0+ with support for native Windows

[dra27]

This PR augments the three compiler packages (ocaml-base-compiler, ocaml-system and ocaml-variants) with support for the MSVC and mingw-w64 native Windows ports for 4.13.0+. I intend to extend this support back to 4.08 as part of an ongoing overhaul of the compiler packages, but that is beyond the scope of this PR.

Principles

The implementation presented here stems from three underlying principles:

1. All four native Windows ports should be able to be co-installed in the same opam root (i.e. ~/.opam) as separate switches (as with opam-repository-mingw). That is, the user should not be forced to choose permanently between MSVC/mingw-w64 and/or amd64/i686 at opam init.
2. The Windows ports should not be made to look like Tier 2 or alternate platforms; i.e. the instructions to create a Windows OCaml switch should not be fundamentally different from other platforms.
3. Depexts requirements should be precise, so the installation of a conf- package should not speculatively install more dependencies than necessary for the given switch. In particular, if a switch is configured with i686 OCaml, the installation of conf-libfoo should install the required packages for i686 libfoo, not both i686 and x86_64 libfoo.

Corollaries

There are some immediate observations and problems:

• The first principle matches the existing capability of opam-repository-mingw.
• The second principle prohibits either requiring opam exec or needing a Windows-specific wrapper (such as with-dkml or ocaml-env).
• The second principle also has implications for the ocaml-base-compiler package, since there is no concept of a "default" C compiler on Windows. The mingw-w64 and MSVC ports of OCaml are not interchangeable in the way that GCC or Clang-based OCaml is.
• The depext system in OCaml for Windows was based on opam 2.0's opam-depext plugin, and never adopted opam 2.1's integrated support. While the use of the plugin means that it suffers from the same solving problems that caused depext to be integrated directly into the solver in opam 2.1, it did mean that opam depext knew which compiler was installed in the switch, because it was necessarily run after the switch had been created.
• The depexts section of an opam file can only be filtered on global variables. While we can (just about) set switch-specific global variables, this would be both awkward (users would need to know of an extra step), but would also contradict the second principle, adding a requirement to Windows-specific parts of the workflow. This means that the depexts for Windows need to be recorded in separate conf- packages, and the dependency graph of a switch needs to record sufficient information about the architecture and C compiler to select the correct package (in short, we require more base--like packages).

Other related work

This work overlaps with some considerable ongoing additional work on the compiler's opam packages:

• Try a new layout for compiler flags for OCaml 4.12.0 (and add 4.12.0~alpha1) #17541 added the ocaml-option- packages to reduce the number of variants, but this applies to OCaml 4.12+ only. This facility needs extending (on principle) back to 4.11 and earlier. It makes little sense to update all the individual variants for 4.11 and earlier without first addressing this issue (it's side-stepped for now as 4.13 is the minimum version updated for a different technical reason).
• As I note in Correct the bytecode architectures for OCaml 5.1 and OCaml 5.2 (trunk) #24075 (comment), Fix ocaml-base-compiler.5.0.0~beta1 on bytecode-only architectures #22246 forged the wrong approach to fixing bytecode-only 5.x architectures started in Fix architecture support for OCaml 5.x #21510. This PR doesn't attempt to fix those issues, but the new virtual packages are added in both "input" (ocaml-option-*) form and "output" (base-*), which I believe to be the fundamental error I (/we) failed to see in Fix architecture support for OCaml 5.x #21510.

I was originally (back in September) of the opinion that it would be better to solve these two issues first and then add Windows support as an extension of these fixes. However, while it's tempting to engineer things such that Windows becomes a "minor" addition, the compatibility concerns for these two fixes make them much higher risk than the Windows packages, which have no compatibility story to worry about. I've therefore restructured the changes so that the alterations are made Windows-only for now, with the fixes to Unix following later, lifting these "limitations".

How it works

With opam 2.2.0 beta2, opam init (if pointed to this branch) will create an OCaml 5.1.1 switch. Concretely, on a clean Windows 11 system:

winget install Git.Git
winget install opam

and then in fresh terminal followed by:

rem Accept all defaults for opam init
opam init git+https://github.com/dra27/opam-repository.git#windows-initial
opam exec -- ocaml

will give OCaml 5.1.1! When creating a switch, an arch- or system- package can simply be added just as for the ocaml-option- packages. For example, assuming the user has installed Visual Studio which, amongst methods, may be done with:

winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"

then a 32-bit MSVC 4.14.2 switch may be created with:

opam switch create 4.14.2-msvc32 ocaml.4.14.2 system-msvc arch-x86_32
echo print_endline "Hello, world" > hello.ml
opam exec -- ocamlopt -o hello.exe hello.ml

Note that these steps do not require the user to start a Visual Studio Tools Command Prompt or do anything beyond installing Visual Studio.

Under the hood

In more detail, at present, the ocaml.x.y.z package encodes the version of OCaml being installed. To this, at present for Windows only, I have added two more sets of packages:

• arch-x86_32 and arch-x86_64 allow the choice between the i686 and amd64 architectures.
• system-mingw and system-msvc provide the choice between the mingw-w64 and Microsoft Visual Studio (MSVC) ports.

Both ocaml-base-compiler and ocaml-variants use these two sets of packages. ocaml-base-compiler remains "OCaml in its default configuration", but it becomes possible to control exactly which C compiler configuration it's using.

The default compiler is amd64 mingw-w64 (i.e. arch-x86_64 and system-mingw will be automatically added if no other arch- or system- package has been selected) for two reasons:

• We can't detect MSVC using opam's depext system at present (but we can automatically install mingw-w64), so mingw-w64 as a default means that opam init always builds a working OCaml
• Cygwin (and MSYS2) are not available for 32-bit systems, so users will be on 64-bit Windows.

For now, it is intentionally not possible to install the Cygwin port of OCaml using native Windows opam.

Where the user installs a system- and an arch- package, there are also new sets of host-arch- and host-system- which are installed by all opam switches. The idea here is that one host-system- and one host-arch- package are always installed in a switch (be that ocaml-base-compiler, ocaml-variants or ocaml-system).

More information for users

The key rule is that arch- and system- should never be used in opam files because there are packages (such as ocaml-system) which don't use them. These are also packages which may also disappear in the future if opam gains a way to specify configuration options for to packages at installation time.

From the user's perspective, specifying arch-x86_64 vs host-arch-x86_64 is similar to the difference between specifying ocaml-base-compiler.4.14.1 vs ocaml.4.14.1. ocaml-base-compiler.4.14.1 instructs opam to build 4.14.1 from source, where ocaml.4.14.1 permits the use of a system compiler.

The motivation for this change is to be able to indicate precisely where packages are not supported:

• available: os != "win32" is the sledgehammer: no Windows support at all
• conflicts: "host-system-msvc": this package works with the mingw-w64 ports, but doesn't work with the Visual Studio ports
• conflicts: "host-arch-x86_32": this package doesn't work on 32-bit Intel
• depends: "host-system-mingw": this package only works with the mingw-w64 ports.

Notes

I've attempted to organise the changes into a meaningful commit series, which is slightly easier to review than the entire diff in one go. I've added missing metadata fields packages in order to pass opam lint.

[dra27]

I have a lot of markdown notes on the full compiler package infrastructure, as well as further notes on the changes here which I'll consolidate into a single piece of markdown which can finally form some documentation (probably to go in the wiki here).

[dra27]

(Some of the packages are triggering a lint warning which will be addressed in opam 2.2.0 beta3 by the final version of ocaml/opam#5927)

[jonahbeckford]

If you are making changes to ocaml-config, I have been stuck not being able to submit DkML into the opam repository because of https://gitlab.com/dkml/distributions/dkml/-/releases/2.1.1#likely-permanent-incompatibilities

The root cause are my changes at https://github.com/diskuv/diskuv-opam-repository/tree/main/packages/ocaml-config so that the delegate executables (ex. ocamlc-real.exe) used by the DkML shims (ex. ocamlc.exe) are recognized. There was some restriction on OCaml 5 only for ocaml-config.3 which makes it impossible for me to upstream.

[dra27]

ocaml-config.3 is OCaml 5 only when not on Windows - is that insurmountable? The change made here is to be explicit that Windows always requires ocaml-config.3. I think this may be a separate discussion (especially if those delegate executables relate to relocation).

[avsm]

Just a "nit" question here: why ppc64 instead of the OCaml power term in the package?

[dra27]

Indeed - the namespacing here is just horrible from a usability perspective (recalling near errors over riscv vs riscv64, etc.!) - so many different names, and often in strings 😱

I opted that opam-based consistency was the best thing (or "least-worst"), so went with the values opam would use for %{arch}% on the basis that both are likely to be used in opam files and while it's odd that ocaml -config-var architecture gives power, it's probably stranger to have to write "host-arch-power" {arch = "ppc64"} in the opam files.

opam's arch variable is whatever uname -m gives with normalisations as given in OpamSysPoll.normalise_arch which pretty much reduce them to x86_64/x86_32, ppc64/ppc32, arm64/arm32, falling back to the uname -m value.

[avsm]

A very charismatic PR, thank you @dra27 :-) I'm not quite sure what the best way to review this; might you have any suggestions? There are three considerations:

• new arch/host package scheme: this looks useful for other packages as well (for example for simpler selection of 64-bit architectures ?).
• changes to ocaml-base-compiler packages for existing non-Windows architectures: is the expected standard that "no behavioural changes are expected" here?
• do the new compilers work on Windows: do we have a story aside from manual testing yet? Manual testing is just fine for now, but it would be a useful guidance to opam-repo maintainers to be explicit here.

[dra27]

Thanks 🙂 I certainly intend to extend the arch- and system- packages to allow better control of the compiler build on Unix systems too (for example, selecting between gcc and clang and being able to control the selection of arm64 or x86_64 on macOS). The host-arch- and host-system- packages work completely on Windows because opam is able to encode the defaults, but to do this on Unix will require some changes in opam (which, probably unsurprisingly, I'm already scheming towards!), but the packages as given here should at least encode the current slightly limited behaviour (cf. things like ocaml/opam#5949). The main thing is that we need a way to communicate a tiny bit more information about the C compiler to the solver (as, for example, we already do about any OCaml "system" compiler).

It should indeed be the case that nothing has changed for non-Windows apart from the installation of host-system-other and a host-arch- package.

There isn't a testing story as yet beyond manual testing, but there isn't a story for the Unix compiler packages either. At the moment, ocaml-variants packages are over-tested because of a "bug" in docker-base-images and ocaml-base-compiler packages aren't tested at all.

[dra27]

I wonder whether a synchronous review may be best?

[raphael-proust]

I think a synchronous review would definitely help me… I'm pinging you offband about it

[dra27]

At the moment, ocaml-variants packages are over-tested because of a "bug" in docker-base-images and ocaml-base-compiler packages aren't tested at all.

A slight refinement - new compiler releases aren't tested at all, but patches to the latest release of existing branches are (although, again, it is more that they're tested by accident than deisgn; cf. ocaml-base-compiler.4.14.2, ocaml-base-compiler.5.0.0, ocaml-base-compiler.5.1.1)

[dra27]

First rebase to clear conflicts with #25875 and #25911; now addressing review

[raphael-proust]

The changeset is good for me. But I think we need to make some changes to the "analysis" part of the CI. Currently the host-arch and host-system packages are queued-up for build on each runner config which obviously is not what we want:

arm32-ocaml-5.2

    host-arch-s390x.1 (failed: Failed: Build failed)
    host-arch-x86_64.1 (failed: Failed: Build failed)
    host-system-mingw.1 (failed: Failed: Build failed)
    host-system-msvc.1 (failed: Failed: Build failed)
    msys2-mingw64.1 (failed: Failed: Build failed)

It's nice to see that the build actually fails with an assertion check for, e.g., host-arch-s390x on an arm32 runner. But I don't think we should test this going forward.

Ping @shonfeder

[raphael-proust]

hmm

There are actually some weird results like successes for installing all the arch package in some default runners. Anyway, I still think that teh analysis phase needs to be updated.

---

This MR can be merged before we update the analysis phase bc those arch/system packages shouldn't be modified too often.

[raphael-proust]

These new packages should have available fields to avoid the CI attempting to install them.

[raphael-proust]

ping @dra27 this can be done later in another PR, I'll let you manage your own TODO list

[raphael-proust]

FYI this is the errors in the CI

extras
    arm32-ocaml-4.14
        host-arch-arm32.1 (failed: Failed: Build failed)
        host-arch-x86_64.1 (failed: Failed: Build failed)
        host-system-mingw.1 (failed: Failed: Build failed)
        host-system-other.1 (failed: Failed: Build failed)
        msys2-clang64.1 (failed: Failed: Build failed)
        ocaml-base-compiler.4.13.0 (failed: Failed: Build failed)
        ocaml-config.0 (failed: Failed: Build failed)
        ocaml-env-msvc64.1 (failed: Failed: Build failed)
        ocaml-system.4.13.0 (failed: Failed: Build failed)
        ocaml-variants.4.13.2+trunk (failed: Failed: Build failed)
        ocaml.4.14.0 (failed: Failed: Build failed)
        system-mingw.1 (failed: Failed: Build failed)
    arm32-ocaml-5.2
        host-arch-s390x.1 (failed: Failed: Build failed)
        host-arch-x86_64.1 (failed: Failed: Build failed)
        host-system-mingw.1 (failed: Failed: Build failed)
        host-system-msvc.1 (failed: Failed: Build failed)
        msys2-mingw64.1 (failed: Failed: Build failed)
        ocaml-base-compiler.4.14.1 (failed: Failed: Build failed)
        ocaml-base-compiler.5.2.0~alpha1 (failed: Failed: Build failed)
        ocaml-option-mingw.1 (failed: Failed: Build failed)
        ocaml-options-only-no-flat-float-array.1 (failed: Failed: Build failed)
        ocaml-system.5.1.1 (failed: Failed: Build failed)
        ocaml-variants.4.14.3+trunk (failed: Failed: Build failed)
        ocaml.4.13.0 (failed: Failed: Build failed)
    arm64-ocaml-4.14
        conf-g++.1.0 (failed: Failed: Build failed)
        msys2-clang64.1 (failed: Failed: Build failed)
        msys2.0.1.0 (failed: Failed: Build failed)
        ocaml-base-compiler.5.2.0 (failed: Failed: Build failed)
        ocaml-base-compiler.5.2.0~beta2 (failed: Failed: Build failed)
        ocaml.4.14.0 (failed: Failed: Build failed)
        ocaml.4.14.3 (failed: Failed: Build failed)
    arm64-ocaml-5.2
        host-arch-x86_64.1 (failed: Failed: Build failed)
        host-system-msvc.1 (failed: Failed: Build failed)
        msys2-clang64.1 (failed: Failed: Build failed)
        msys2-clangarm64.1 (failed: Failed: Build failed)
        msys2-mingw32.1 (failed: Failed: Build failed)
        system-mingw.1 (failed: Failed: Build failed)
        system-msvc.1 (failed: Failed: Build failed)

[dra27]

What available field should they have? No other base packages have available fields?

[dra27]

Although I can see they might temporarily want to conflict older compilers which haven't been updated to use them - I will indeed put that on the TODO list

[raphael-proust]

They should have available fields matching their architecture. E.g., host-arch-s390x should have available: [ arch = "s390x" ]. This will avoid the CI trying to build the package on, say, arm32.

[dra27]

Thank you for the reviews and for the merge, @raphael-proust!

[avsm]

Congratulations on the merge, @dra27! A huge amount of work from you on this over many years.

[nojb]

Congratulations @dra27 (and everyone else who helped)!

[MSoegtropIMC]

@dra27 : can it be that this breaks OCaml on MacOS. I updated my repo and now get for e.g. 4.14.2:

Switch invariant: ["ocaml-variants" {= "4.14.2+options"} "ocaml-option-flambda"]
[ERROR] Could not determine which packages to install for this switch:

• Missing dependency:
  • ocaml-option-32bit
    unmet availability conditions: 'arch = "x86_64" & (os = "linux" | os = "macos")'

I am installing opam and switches with scripts I didn't change, so I guess it is a result of this PR.

[MSoegtropIMC]

I am on on ARM Mac, btw, so no idea how I get to x86_64 - my opam is clearly arm.

[MSoegtropIMC]

Update: I rolled back to the previous version of ocaml-variants.4.12.2+options and this works.

[dra27]

Sorry about this, it turns out it's a silly mistake in this conjunction:

 (("host-arch-x86_32" {os != "win32" & arch = "x86_64" & post} & "ocaml-option-32bit" {os != "win32"}) |
   "host-arch-x86_64" {os != "win32" & arch = "x86_64" & post})

For arch != "x86_64" and os != "win32" this is reducing to the single atom "ocaml-option-32bit" which is of course failing, as that package is not available on arm64.

I have already tested a fixing commit (dra27@aef43ad), but as there's a related issue with ocaml-ci and this disjunction, I'm just double-checking what needs to happen (that commit will definitely be in a PR by the end of today, no matter what).

[MSoegtropIMC]

Thanks for the quick work - I appreciate it!