Move to vc142 toolchain on windows

[h-vetinari]

TL;DR

Move to newer toolset because:

• vc141 is disappearing in the CI of many projects (due to the compiler where it's the default being EOL), leading to bitrot, breakage, and extra work (both upstream and in packaging)
• ABI not hard-linked with OS as on unix
• better performance & less bugs
• it's trivial for conda-forge to keep a consistent toolchain
• the set of users being broken by this change is either empty or very small (<- happy to be challenged on this)

Background

Visual Studio version vs. Toolset version

The MSVC compiler comes in different versions (..., VS2015, VS2017, VS2019, VS2022); these compilers come with a default toolset, but are able to target older ones. E.g. VS2017 comes with vc141, but can target vc140 (the default toolset of VS2015). vc142 is the default toolset of VS2019. What is the toolset exactly? From the docs:

The Microsoft C++ compiler, linker, standard libraries, and related utilities make up the MSVC compiler toolset.

ABI compatibility

Microsoft has been extremely conservative with ABI-compatibility from VS2015-VS2022 (the story about [[msvc::no_unique_address]] is instructive for that), meaning that a given project can use the new compilers while using all the old binaries that flow into the build. The one caveat is that any library built with a given toolset version can only be consumed (e.g. linked) by a toolset version that's the same or newer.

No OS coupling

Furthermore, unlike on linux, the version / ABI of the compiler is not coupled as tightly to a distribution (mostly through the ABI of the libstdcxx that comes with the compiler). As such, changing the toolchain version does not affect which platforms a binary built by a given toolset will run on.

VS2017 EOL & impact on open-source CI

VS2017 has reached EOL, and has been aggressively removed by microsoft from its public CIs (Azure & GHA), see previous discussion. This is IMO both understandable (avoid maintaining 3 separate compilers), as well as not very impactful, due to the very strong ABI-compatibility between VS2017 and VS2019.

As such, many open source projects using these CI providers have forcibly moved to VS2019, and in almost all cases, adopted its default toolset vc142 (it's possible to change, but presumably most maintainers don't know or don't care until an issue is raised). This means that compatibility with vc141 is bitrotting at an accelerated pace, because the respective CIs are not trying to compile against that anymore. LLVM 14+ also hard-requires vc142, and we have followed suit there already.

This happened even to numpy (originally the 1.22.0 release was not compilable with the vc141 toolchain), and similar issues keep cropping up. I argue that spending time on this has a bad cost/benefit, and the numpy devs voiced interest in moving to vc142 as well, but mentioned the current restriction of conda-forge.

Since I've already argued for this move in the context of the windows-2016 image deprecation and the numpy 1.22.rcs, I'm now (reluctantly) picking up the mantle for this, c.f. Isuru's comment:

[...] until someone champions the move to vs2019 which means getting approval from core to move, announcing the move and giving time for users and then doing the switch itself.

Impact

On conda-forge packaging

None. We have a consistent toolchain, and feedstocks upgrading to a new compiler can continue to use all previously built artefacts. That means we don't even need a migration, we can just switch to vs2019 in conda-smithy.

On conda-forge users

For the vast majority of users who just consume packages, there is no impact.

The main objection to bumping the toolset version along with the image & compiler in was the impact on users that are using a conda-based environment to develop on windows, and have VS2017 installed.

I claim (and would be happy to be challenged on this) that the set of affected users - i.e. who develop on windows using conda, aren't already directly or indirectly affected by CI providers dropping VS2017 (e.g. due to vc141-support bitrotting for their dependencies), and cannot upgrade to a 100% ABI-compatible, 1:1 drop-in & free compiler - is empty, or very very close to it.

If anyone would be affected by this (or knows someone who is), I'd really like to know the constraints of the situation, because I have trouble imagining what those could be.

Compatibility with mingw

By default, vc142 emits some assembly sections that mingw chokes on. However, this can be fixed with the addition of a single flag (-d2VolatileMetadata-). Given that the previous interaction was working only incidentally (cf. the following comment from upstream dev), this isn't a substantial change in circumstances:

Mixing static libraries or object files between mingw and msvc is not
supported, and not expected to work, in general. You might have been lucky
and your library might have been simple enough not to hit any problematic
case though.

Other

• Many feedstocks are using VS2019(+vc142) already, this would harmonize things.
• Many bugs fixed -> less compilation errors on windows
• Better performance in some cases (e.g. for numpy, this would enable compiling down to instrinsics for __popcnt)

In summary

It's time to move on - the trade-offs for staying on vc141 are getting less and less justifiable by the day, the benefits are substantial (as time is the most precious resource in upstream & packaging), and the breakage (if any) is negligible.

[h-vetinari]

@conda-forge/core

Could someone comment on this?

[hmaarrfk]

I think this makes sense, but I don't know too much about windows compilers. I do agree that we should likely attempt to keep up.

[h-vetinari]

Move to newer toolset because:

• vc141 is disappearing in the CI of many projects (due to the compiler where it's the default being EOL), leading to bitrot, breakage, and extra work (both upstream and in packaging)

Case in point, scipy added a new submodule in the 1.9.x cycle, and this fails with the compiler generating non-copyable types in some templated code.

The issue does not appear with VS2019, which is why it didn't show up in upstream CI. The upstream wheel builds were also simply moved to VS2019.

This leaves the scipy-feestock in a bind¹, so could we please restart this discussion?

Footnotes

1. unless someone comes up with a patch, but this time it's not a simple header inclusion problem like for numpy last time ↩

[hmaarrfk]

I also had to update the compiler for opencv due to a similar issue

[jaimergp]

This might be a nice item in the agenda for the next conda-forge call!

[h-vetinari]

This might be a nice item in the agenda for the next conda-forge call!

I always forget where to look up the next calls. The minutes don't give an indication...

[jaimergp]

They are listed here, but yes, it's not a very visible place. I think there's a PR changing that now. In the meantime: next one is Wednesday, next week, 5PM UTC. Send me a DM on gitter with your preferred email and I'll add you to the invite!

[h-vetinari]

Sent you a PM, thanks! I'm visiting my folks and that is not a good time for me next Wednesday, but I'll join if I should happen to be free.

[h-vetinari]

I'm visiting my folks and that is not a good time for me next Wednesday, but I'll join if I should happen to be free.

As feared I won't be able to make it. Feel free to discuss without me, the OP should be pretty comprehensive. Otherwise I'll try to join again next time.

PS. SciPy 1.9 is now requiring vc142 as well, so the question for the scipy-feedstock has been made obsolete.

[h-vetinari]

As requested in the core meeting just now, I've opened a PR: conda-forge/conda-forge-pinning-feedstock#3167

PS. This is the issue @mattip mentioned about the future of the static npymath library currently shipped by numpy: numpy/numpy#20880