Download and manage internal copy of MSYS2 for ./configure-only libraries

[ras0219-msft]

Since it's unfortunately common to require autoconf on Windows (ffmpeg, ipopt, ...), we probably need vcpkg to download and manage an internal copy of MSYS2 for just this reason.

It is possible to use MSYS2 without a global install (https://github.com/StephanTLavavej/mingw-distro), but those instructions require user interaction as written. I'm sure we can mimic whatever effects the msys2_shell.bat script is having and internally perform those.

Depending on the size/time cost of this, we may want to allow the user to provide their own MSYS2 (or autodetect a system-installed one).

Needed for: #233 #2

[kryptan]

MSYS2 relies on its own package manager (pacman) which downloads binary packages. This will make one package manager (vcpkg) to depend on the other.

1. This has security implications, if MSYS2 binary repository is compromised this will mean that vcpkg is also compromised.
2. AFAIK pacman always installs the latest version of each package and there is no easy way to install specific version. This interferes with the reproducibility - an update in MSYS2 can potentially break something in vcpkg.

Maybe vcpkg should provide prebuilt versions of each required MSYS2 package and not rely on pacman at all?

Also I'm not sure if autoconf scripts can understand how to use msvc.

Since it's unfortunately common to require autoconf on Windows (ffmpeg, ipopt, ...)

ffmpeg doesn't use autoconf, there is a handwritten configure bash script. Still requires msys though.

[ras0219-msft]

Thanks for bringing up the point about autoconf; I believe all the libraries above use bash ./configure scripts and not the full autotools chain.

• This would indeed mean a dependence for those particular packages, however only to acquire the buildsystem tools. Once the build is complete, there would be no MSYS2 dependency. Users who don't need those packages would of course not need MSYS2.
• (1) This is somewhat true for the packages that require MSYS2 dependence. We may be able to mitigate this by downloading and hash-checking the packages from http://repo.msys2.org/msys/x86_64/ directly before installing them with pacman -U.
• (2) Unfortunately, I don't see a better option here. In combination with the answer to (1), we can give users a way to cache the specific version they used which would help reproducibility, but as long as libraries require MSYS2 as part of their build (and MSYS2 doesn't host packages indefinitely), we will be subject to this issue.

[weltling]

Just to give an example - MSYS2 doesn't require any user interaction. It is possible to compose a set of required tools, that can be provided. I'm currently working on the PHP Windows build toolkit, where we use many OSS tools like fe. bison, re2c, zip, etc. The actual PHP binaries and dependencies are produced with the conventional Visual Studio by standard. So we don't depend on the MSYS2 libraries for the compilation, but only on the tools, that MSYS2 provides or that are impossible to provide using the Windows toolchain.

What I'm literally doing, is picking the required tools and their dependencies, thus avoiding to have the whole MSYS2 tree, even avoiding to have the whole base tools package. Also, even if the original MSYS2 might not contain some tool - well, you can compile it with the toolchain provided by MSYS2. It'll have then same dependencies as the MSYS2 distribution, so can be shipped with it. Now, as i understand, the tools like autotools, maybe make and some others, are only what is needed for VCPKG as well. Otherwise, it could be done the other way round - take the MSYS2 distribution, but only leave the tools that can be useful and strip others out.

As mentioned, the big pluses of doing it this way are

• smaller end size
• control over the exact toolset (fe you don't have to add pacman)
• no installation is required

Thanks.

[tkelman]

Does invoking msvc through a configure script work correctly in msys2? The way I've done that in the past, using the compile wrapper script from automake (roughly ./configure CC="$PWD/compile cl") doesn't work in msys2 last time I checked.

[codicodi]

@tkelman I recently compiled libffi this way, so it's definitely possible. Here (also in comments) are some tips to get it to work.

[tkelman]

Don't think that --toolchain is a standard autoconf flag. If a project is using autoconf and automake in the conventional way, then the generated makefiles will try to invoke the compiler with unix cc-style flags that msvc won't handle.

Long term it's probably most worthwhile to translate and contribute cmake ports of various projects' build systems for them to support msvc in a predictable way.

[ras0219-msft]

I've added initial support for a minimal internal msys with 2f8d8d8, which also demonstrates using it to build FFMPEG. Please open new issues or pull requests for improvements!

A notable issue at the moment: we do not hash check the msys2 packages. This means they can (and will) update underneath us, which could cause intermittent or unreproducible errors for users. We should try to improve this in the future, such as by individually downloading packages ourselves, checking them, then installing them with pacman -U.