depexts for MSYS2 #5348

jonahbeckford · 2022-11-11T05:14:34Z

MSYS2 on Windows uses the same package manager as Arch Linux (pacman). It targets several ABIs (compilers, bitiness and Windows C libraries).

This PR builds on the opam global variables that are registered when Diskuv OCaml (DKML) is installed. Those variables became available with DKML 1.1: https://gitlab.com/diskuv/diskuv-ocaml/-/blob/bd5b461b7d267ff897d053a67cce6ed5c26f27b4/CHANGES.md. Any MSYS2 distribution (not just DKML) can set those opam variables:

$ opam var --global
arch                 x86_64                                                         # Inferred from system
exe                  .exe                                                           # Suffix needed for executable filenames (Windows)
...
mingw-chost          x86_64-w64-mingw32                                             # Set through 'opam var'
mingw-package-prefix mingw-w64-clang-x86_64                                         # Set through 'opam var'
mingw-prefix         /clang64                                                       # Set through 'opam var'
msys2-nativedir      C:\Users\beckf\AppData\Local\Programs\DiskuvOCaml\tools\MSYS2\ # Set through 'opam var'
msystem              CLANG64                                                        # Set through 'opam var'
msystem-carch        x86_64                                                         # Set through 'opam var'
msystem-chost        x86_64-w64-mingw32                                             # Set through 'opam var'
msystem-prefix       /clang64                                                       # Set through 'opam var'
opam-version         2.2.0~alpha0~20221104                                          # The currently running opam version
...

Testing

With the following change to conf-pkg-config:

 depexts: [
   ["pkg-config"] {os-family = "debian"}
   ["pkgconf"] {os-distribution = "arch"}
   ["pkgconf-pkg-config"] {os-distribution = "fedora"}
   ["pkgconfig"] {os-distribution = "centos" & os-version <= "7"}
   ["pkgconf-pkg-config"] {os-distribution = "mageia"}
   ["pkgconfig"] {os-distribution = "rhel" & os-version <= "7"}
   ["pkgconfig"] {os-distribution = "ol" & os-version <= "7"}
   ["pkgconf"] {os-distribution = "alpine"}
   ["pkgconfig"] {os-distribution = "nixos"}
   ["devel/pkgconf"] {os = "openbsd"}
   ["pkg-config"] {os = "macos" & os-distribution = "homebrew"}
   ["pkgconfig"] {os = "macos" & os-distribution = "macports"}
   ["pkgconf"] {os = "freebsd"}
   ["pkgconf-pkg-config"] {os-distribution = "rhel" & os-version >= "8"}
   ["pkgconf-pkg-config"] {os-distribution = "centos" & os-version >= "8"}
   ["pkgconf-pkg-config"] {os-distribution = "ol" & os-version >= "8"}
   ["system:pkgconf"] {os = "win32" & os-distribution = "cygwinports"}
+  # Although not supported in Opam 2.2 (and earlier), we want the equivalent of:
+  #   ["%{mingw-package-prefix}%-pkg-config"] {os = "win32" & ?mingw-package-prefix}
+  ["mingw-w64-clang-i686-pkg-config"]   {os = "win32" & msystem = "CLANG32"}
+  ["mingw-w64-clang-x86_64-pkg-config"] {os = "win32" & msystem = "CLANG64"}
+  ["mingw-w64-i686-pkg-config"]         {os = "win32" & msystem = "MINGW32"}
+  ["mingw-w64-x86_64-pkg-config"]       {os = "win32" & msystem = "MINGW64"}
+  ["mingw-w64-ucrt-x86_64-pkg-config"]  {os = "win32" & msystem = "UCRT64"}
 ]

we can now do:

C:\> with-dkml `
  Z:\source\dkml-component-opam\_opam\share\dkml-component-staging-opam64\staging-files\windows_x86_64\bin\opam.exe `
  install Z:\source\opam-repository\packages\conf-pkg-config\conf-pkg-config.2\opam 
[conf-pkg-config.2] synchronised (no changes)
The following actions will be performed:
=== install 1 package
  ✶ conf-pkg-config 2 (pinned)

The following system packages will first need to be installed:
    mingw-w64-clang-x86_64-pkg-config

<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>

opam believes some required external dependencies are missing. opam can:
> 1. Run C:\Users\beckf\AppData\Local\Programs\DiskuvOCaml\tools\MSYS2\usr\bin\pacman.exe to install them (may need root/sudo access)
  2. Display the recommended C:\Users\beckf\AppData\Local\Programs\DiskuvOCaml\tools\MSYS2\usr\bin\pacman.exe command and wait while you run it manually (e.g. in another
     terminal)
  3. Attempt installation anyway, and permanently register that this external dependency is present, but not detectable
  4. Abort the installation

[1/2/3/4] 1

+ C:\Users\beckf\AppData\Local\Programs\DiskuvOCaml\tools\MSYS2\usr\bin\pacman.exe "-Su" "--noconfirm" "mingw-w64-clang-x86_64-pkg-config"
- :: Starting core system upgrade...
-  there is nothing to do
- :: Starting full system upgrade...
- resolving dependencies...
- looking for conflicting packages...
- 
- Packages (1) mingw-w64-clang-x86_64-pkg-config-0.29.2-3
- 
- Total Installed Size:  1.39 MiB
- 
- :: Proceed with installation? [Y/n] 
- checking keyring...
- checking package integrity...
- loading package files...
- checking for file conflicts...
- :: Processing package changes...
- installing mingw-w64-clang-x86_64-pkg-config...

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
▼ retrieved conf-pkg-config.2  (file://Z:/source/opam-repository/packages/conf-pkg-config/conf-pkg-config.2)
✶ installed conf-pkg-config.2
Done.
# Run eval $(opam env) to update the current shell environment

(If it wasn't a test the command line would simply be opam install conf-pkg-config)

kit-ty-kate

Thanks a lot! That looks great! My only issue is on the msys2-nativedir variable / msys2 detection story, but the rest looks ok.

src/state/opamSysInteract.ml

kit-ty-kate · 2022-11-11T13:20:12Z

Oh, also I think the os-distribution should be also set to msys2 (see how it's done for macOS (homebrew vs. macports) in src/state/opamSysPoll.ml). Otherwise we can't distinguish between msys2 and other packages managers.

kit-ty-kate · 2022-11-11T13:25:48Z

src/state/opamSysInteract.ml

@@ -130,6 +138,15 @@ let family ~env () =
        "External dependency handling for macOS requires either \
         MacPorts or Homebrew - neither could be found"
    | "suse" | "opensuse" -> Suse
+    | "windows" ->


Something like that would also need to be done (see how it's done for BSDs above)

Suggested change

| "windows" ->

| "windows" ->

begin match OpamSysPoll.os_distribution env with

| Some "msys2" ->

Fixing! Did a search in fdopen for os-distribution = "win32"; nothing found, so it is safe for existing Diskuv users. Thanks

For posterity the setting of os-distribution was done in diskuv/dkml-runtime-distribution@ff42f08 (used by the Diskuv OCaml installer).

Added the Some "msys2" on top of existing match. Once we decide on a global variable for the package manager binary (C:\Users\beckf\AppData\Local\Programs\DiskuvOCaml\tools\MSYS2\usr\bin\pacman.exe) then the match clause will simplify.

jonahbeckford · 2022-11-11T17:54:41Z

Oh, also I think the os-distribution should be also set to msys2 (see how it's done for macOS (homebrew vs. macports) in src/state/opamSysPoll.ml). Otherwise we can't distinguish between msys2 and other packages managers.

Can you suggest to me what the values should be? I have no idea what the distinction is between os-distribution and os-family. Today it is:

opam-version         2.2.0~alpha0~20221104                                          # The currently running opam version
os                   win32                                                          # Inferred from system
os-distribution      win32                                                          # Inferred from system
os-family            windows                                                        # Inferred from system
os-version           10.0.22621                                                     # Inferred from system

Do you want:

opam-version         2.2.0~alpha0~20221104                                          # The currently running opam version
os                   win32                                                          # Inferred from system
os-distribution      msys2                                                          # Inferred from system
os-family            msys2                                                        # Inferred from system
os-version           10.0.22621                                                     # Inferred from system

Note: I've been using opam switch variables rather than opam global variables for the wrap-build-commands so that DKML created switches (ie. opam dkml init) can avoid conflicts with other Windows switches (I guess Cygwin). And I was going to use ?mingw-package-prefix to check for MSYS2 if needed. However perhaps it is time to ditch the interoperability ... either Opam will globally use MSYS2 or Opam will globally use Cygwin.

Also, which opam variable can be used to indicate which Windows C library (SDK+headers) to use when compiling C programs? The choices boil down to Windows SDK (I've been calling this native Windows) or the GNU C library (MinGW/Cygwin). Rust for example equates the binary choice to a choice in the ABI: https://rust-lang.github.io/rustup/installation/windows.html. It would be nice to have an opam variable that clearly said which Windows ABI to use. Today a package maintainer can look at the lack of the cygwinports opam value to compile for native Windows, but that is hacky. They can also look at the ocamlc -config ccomp_type for msvc, but that is insufficient because clang can compile both the Windows SDK and the GNU C library.

kit-ty-kate · 2022-11-11T18:12:30Z

I think the values should be:

os-distribution: msys2
os-family: windows
os: win32

Also thinking about the msystem values in your first comment: do all packages have versions for each subsystems? If so maybe it would be better to be able to just do:

  ["pkg-config"] {os = "win32" & os-distribution = "msys2"}

and let opam infer which is the current subsystem (using either the MSYSTEM or MINGW_PACKAGE_PREFIX environment variable) or use the opam variable for that (whichever you prefer)

jonahbeckford · 2022-11-16T18:37:20Z

I think the values should be:
os-distribution: msys2
os-family: windows
os: win32

Thanks.

Also thinking about the msystem values in your first comment: do all packages have versions for each subsystems? If so maybe it would be better to be able to just do:
  ["pkg-config"] {os = "win32" & os-distribution = "msys2"}
and let opam infer which is the current subsystem (using either the MSYSTEM or MINGW_PACKAGE_PREFIX environment variable) or use the opam variable for that (whichever you prefer)

Sigh; most but not all packages have versions for each subsystem. Common executables like bash and p7zip do not have subsystem packages since those tools are only for the build machine ABI. But almost anything that exposes a C library (including pkg-config) are only available as subsystem packages since C libraries by definition are ABI specific.

Perhaps we could use a label? Example:

["p7zip"]              {os = "win32" & os-distribution = "msys2"}
["rsync"]              {os = "win32" & os-distribution = "msys2"}
["abi:pkg-config"]     {os = "win32" & os-distribution = "msys2"}
["abi:SDL2"]           {os = "win32" & os-distribution = "msys2"}

dra27

Thanks for this!

We need to be concerned about OPAMCONFIRMLEVEL being respected - and we're going to be meeting shortly to discuss how to deal with the detection of MSYS2 in this case, so that can be added separately.

master_changes.md

src/state/opamSysInteract.ml

* Part of ocaml/opam#5348

rjbou · 2022-12-15T15:26:23Z

After discussion, we introduce sys-pkg-manager-path as a variable that contains system package manager full path. It is needed in this PR for Msys2, and in the future will be needed for cygwin support.
I updated the PR consequently, if it ok for you @jonahbeckford @dra27 @kit-ty-kate , i'll rebase/clean history.

Aligns with ocaml/opam#5348

jonahbeckford · 2022-12-16T00:35:04Z

After discussion, we introduce sys-pkg-manager-path as a variable that contains system package manager full path. It is needed in this PR for Msys2, and in the future will be needed for cygwin support. I updated the PR consequently, if it ok for you @jonahbeckford @dra27 @kit-ty-kate , i'll rebase/clean history.

Thanks. I added sys-pkg-manager-path (diskuv/dkml-runtime-distribution@45b9997 for posterity).

Looks good!

dra27 · 2022-12-19T14:22:35Z

Sorry, @jonahbeckford, we've just been discussing again trying to reconcile how this variable works for all the package managers, and in doing so we've come back round to something that's closer to where you started! For this PR, with apologies for yet another tweak needed in dkml-runtime, could the variable by sys-pkg-manager-cmd-msys2?

For a later PR, the plan is then to have sys-pkg-manager-cmd-cygwin (for Cygwin's installer) and to alter the rest of the function so that in theory you could define sys-pkg-manager-cmd-alpine, sys-pkg-manager-cmd-debian, etc. The previous sys-pkg-manager-path proposed having one variable shared by all package managers, but having a common prefix with an optional variable for each package manager seems better, especially in the case where you actually choose to change depext system (which can happen on macOS or with nix, and would mean that you could switch an entire opam root from MSYS2 to Cygwin or vice versa, if you really wanted).

In particular, that means that for everything which isn't Cygwin or MSYS2, opam would just use a default. i.e. on Debian/Ubuntu, you could choose to set sys-pkg-manager-cmd-debian to /usr/bin/apt-get, but if you don't then opam will just do apt-get as it does now but on Cygwin/MSYS2, you must the appropriate sys-pkg-manager-cmd-cygwin/sys-pkg-manager-cmd-msys2 or opam will instead say that it can't run the package manager.

- Second alignment with ocaml/opam#5348

jonahbeckford · 2022-12-20T15:50:10Z

... could the variable by sys-pkg-manager-cmd-msys2?

No problem. diskuv/dkml-runtime-distribution@316f67e for posterity.

src/state/opamSysInteract.ml

rjbou · 2023-01-06T15:19:56Z

gentoo fix in #5410

dra27 · 2023-01-09T09:15:11Z

Thanks!

jonahbeckford force-pushed the feature-msys2-depexts branch from e603a40 to c67efb9 Compare November 11, 2022 05:17

kit-ty-kate reviewed Nov 11, 2022

View reviewed changes

src/state/opamSysInteract.ml Outdated Show resolved Hide resolved

src/state/opamSysInteract.ml Outdated Show resolved Hide resolved

src/state/opamSysInteract.ml Outdated Show resolved Hide resolved

kit-ty-kate added this to the 2.2.0~alpha milestone Nov 11, 2022

kit-ty-kate added AREA: PORTABILITY AREA: DEPEXTS PR: WAITING FOR REVIEW labels Nov 11, 2022

kit-ty-kate reviewed Nov 11, 2022

View reviewed changes

dra27 reviewed Nov 18, 2022

View reviewed changes

master_changes.md Outdated Show resolved Hide resolved

src/state/opamSysInteract.ml Outdated Show resolved Hide resolved

src/state/opamSysInteract.ml Outdated Show resolved Hide resolved

jonahbeckford added a commit to diskuv/dkml-runtime-distribution that referenced this pull request Nov 18, 2022

Add os-distribution=msys2 if using MSYS2 during install

ff42f08

* Part of ocaml/opam#5348

jonahbeckford added a commit to diskuv/dkml-runtime-distribution that referenced this pull request Dec 16, 2022

Set sys-pkg-manager-path as pacman.exe

45b9997

Aligns with ocaml/opam#5348

jonahbeckford added a commit to diskuv/dkml-runtime-distribution that referenced this pull request Dec 20, 2022

Set sys-pkg-manager-cmd-msys2 as path to pacman

316f67e

- Second alignment with ocaml/opam#5348

rjbou reviewed Jan 6, 2023

View reviewed changes

src/state/opamSysInteract.ml Outdated Show resolved Hide resolved

rjbou force-pushed the feature-msys2-depexts branch from ce80b31 to d435910 Compare January 6, 2023 15:04

rjbou approved these changes Jan 6, 2023

View reviewed changes

rjbou added STATE: READY TO MERGE and removed PR: WAITING FOR REVIEW labels Jan 6, 2023

rjbou added the PR: QUEUED Pending pull request, waiting for other work to be merged or closed label Jan 6, 2023

rjbou and others added 2 commits January 7, 2023 13:46

update gitignore

787e977

depext: Add MSYS2 support, add opam variable 'sys-pkg-manager-cmd-msys2'

14ee446

update changes

b3d5666

rjbou force-pushed the feature-msys2-depexts branch from d435910 to b3d5666 Compare January 7, 2023 12:47

dra27 approved these changes Jan 9, 2023

View reviewed changes

dra27 merged commit 9ff2373 into ocaml:master Jan 9, 2023

rjbou mentioned this pull request Feb 7, 2023

Variable for package manager #5436

Merged

rjbou removed the STATE: READY TO MERGE label Jun 13, 2023

This was referenced Feb 15, 2024

Opam 2.2 init with MSYS2 #5843

Merged

Init config file does not define sys-pkg-manager-cmd #5844

Closed

dra27 mentioned this pull request Jun 13, 2024

Does OPAM work on windows? #246

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

depexts for MSYS2 #5348

depexts for MSYS2 #5348

jonahbeckford commented Nov 11, 2022

kit-ty-kate left a comment

kit-ty-kate commented Nov 11, 2022 •

edited

Loading

kit-ty-kate Nov 11, 2022

jonahbeckford Nov 18, 2022

jonahbeckford Nov 18, 2022

jonahbeckford Nov 18, 2022

jonahbeckford commented Nov 11, 2022

kit-ty-kate commented Nov 11, 2022

jonahbeckford commented Nov 16, 2022

dra27 left a comment

rjbou commented Dec 15, 2022

jonahbeckford commented Dec 16, 2022

dra27 commented Dec 19, 2022

jonahbeckford commented Dec 20, 2022

rjbou commented Jan 6, 2023

dra27 commented Jan 9, 2023

depexts for MSYS2 #5348

depexts for MSYS2 #5348

Conversation

jonahbeckford commented Nov 11, 2022

Testing

kit-ty-kate left a comment

Choose a reason for hiding this comment

kit-ty-kate commented Nov 11, 2022 • edited Loading

kit-ty-kate Nov 11, 2022

Choose a reason for hiding this comment

jonahbeckford Nov 18, 2022

Choose a reason for hiding this comment

jonahbeckford Nov 18, 2022

Choose a reason for hiding this comment

jonahbeckford Nov 18, 2022

Choose a reason for hiding this comment

jonahbeckford commented Nov 11, 2022

kit-ty-kate commented Nov 11, 2022

jonahbeckford commented Nov 16, 2022

dra27 left a comment

Choose a reason for hiding this comment

rjbou commented Dec 15, 2022

jonahbeckford commented Dec 16, 2022

dra27 commented Dec 19, 2022

jonahbeckford commented Dec 20, 2022

rjbou commented Jan 6, 2023

dra27 commented Jan 9, 2023

kit-ty-kate commented Nov 11, 2022 •

edited

Loading