META: {{ stdlib("c") }} migration

[h-vetinari]

If you got directed here by the linter...

A short overview of the situation and what to do can be found is in this announcement. For the gory details, see below...

---

One of the few things that conda cannot ship sanely & safely is the C standard library that is an integral part of the operating system (certainly on UNIX), and very ABI-sensitive. This is also the reason why distributions that do ship the C standard library (glibc on linux) stay on the same version for the lifetime of that distribution. Which in turn is why the glibc version is the "clock" measuring distro age in PEP600, and thus manylinux wheels (statistics).

For almost the entire duration of conda-forge's existence (AFAIU; some of this is way before my time), the underlying C standard library has not changed from:

• glibc 2.12 from CentOS 6 (though this represents the baseline we build against; the CI image already has glibc 2.17 from CentOS 7; see here)
• the C stdlib as of macOS 10.9 (last bump)

Given that glibc 2.12 is almost 14 years old (and CentOS 6 is EOL for over 3 years), and that macOS 10.9 is EOL since over 7 years (with Apple aggressively pushing users to upgrade), a lot of modern packages are starting to require newer, sometimes way newer glibc versions (the reason we were able to hold out so long w.r.t. OSX is that we ship our own C++ standard library, but we cannot do that for C). With the breadth of feature expansion in the C23 standard, this will also continue well into the future, as projects eventually start picking up these things.

For packages already requiring a newer C stdlib, there have been work-arounds in place for a long time:

• - sysroot_linux-64 2.17 # [linux64]
• - MACOSX_DEPLOYMENT_TARGET

But these are ultimately hacks, and not ready to support a wide-scale roll-out. This is what has blocked us from releasing libcxx 17.0 for example, since it requires macOS >=10.13, and would virally spread that requirement to essentially all packages that use C++ in any of their dependencies (which means +/- all of conda-forge).

The need to move to greener pastures has been clear for a long time, though the work involved is painful and it wasn't clear how to do it exactly. In any case, we've already announced that:

• We're moving to glibc 2.17 / CentOS 7 this summer (meta-issue)
• We're moving to macOS 10.13 as the baseline as soon as our infrastructure allows (meta-issue)

The latter announcement already explicitly added a caveat that we need to figure out how to do this first, because the fact that we've never really bumped the C stdlib baseline version on linux or osx, means that it's crept into lots of places as a silent assumption waiting to break. The most critical thing being correct metadata for the produced packages - in particular, if some package requires a newer C stdlib than a user has (as identified by the __glibc / __osx virtual packages; which need at least conda >=4.8), the package should not ever be installed.

After several discussions in cf/core, as well as with people from Anaconda & Prefix, we now have identified a way forward which involves a new jinja-function {{ stdlib("c") }} to match {{ compiler("c") }}. This is a welcome increase in expressivity, as indeed the version of the C standard library and compiler are very distinct things (as for other languages as well...), but with the substantial downside that no recipes are using this functionality yet. This means we have to add this function to essentially all compiled recipes, to ensure we have the mechanics in place for later bumping the minimum required version (while also allowing selective opt-ins to newer or older versions per feedstock).

The function is designed in many ways in parallel with the {{ compiler("...") }} jinja, and so the corresponding keys in conda_build_config.yaml should be unsurprising:

c_stdlib:
  - sysroot                    # [linux]
  - macosx_deployment_target   # [osx]
  - vs                         # [win]
c_stdlib_version:              # [unix]
  - 2.12                       # [linux and x86_64]
  - 2.17                       # [linux and not x86_64]
  - 10.9                       # [osx and x86_64]
  - 11.0                       # [osx and arm64]

The required functionality was merged into conda-build as of 3.28, but some bugs are still being ironed out and it'll likely not yet be fully functional until the upcoming conda-build 24.3.

This issue is intended for discussion, questions, etc. for this entire effort, beyond the specifics that will arise for glibc 2.17 resp. macos 10.13.

---

• Come up with a whole new bit of conda infrastructure:
  • New jinja-function for handling sysroot conda/conda-build#4981 / Reopen the discussion on stdlib jinja function conda/conda-build#5053
  • Implement that: support stdlib() jinja function  conda/conda-build#4999
  • Wait for new conda-build version (released in 3.28)
  • ENH: Improve stdlib-handling conda/conda-build#5190
  • Fix stdlib being recognized in variant hash inputs conda/conda-build#5195
  • Backport patch fixing stdlib detection conda-build-feedstock#220 (came through conda-build 24.3)
• Stdlib metapackages
  • ENH OSX sysroot/sdk package staged-recipes#23777
  • Use existing sysroot_{{ target }} on linux and vs_{{ target }} on windows
  • Add c_stdlib{,_version} to global pinning: add stdlib metapackages to pinning conda-forge-pinning-feedstock#5499
  • BUG fix zipping for c stdlib conda-forge-pinning-feedstock#5592
  • fix c_stdlib_version zip with cdt_name conda-forge-pinning-feedstock#5607
  • add MACOSX_DEPLOYMENT_TARGET to same zip as c_stdlib_version conda-forge-pinning-feedstock#5669
  • Only zip c_stdlib_version with compilers for CUDA conda-forge-pinning-feedstock#5672
  • also set c_stdlib_version on windows conda-forge-pinning-feedstock#5736
  • Revert "also set c_stdlib_version on windows" conda-forge-pinning-feedstock#5756
  • add definition of m2w64_c_stdlib on windows conda-forge-pinning-feedstock#5832
• Infrastructure fixes
  • ENH: correctly rerender CI configs with new stdlib jinja-function conda-smithy#1840
  • ENH: add handling for stdlib-jinja function conda-smithy#1846
  • add test for correctly resolving stdlib conda-smithy#1888
  • ENH: Take into account stdlib_version for determining macOS SDK version in .ci_support/ and cf-ci-setup conda-smithy#1884
  • merge c_stdlib_version & MACOSX_DEPLOYMENT_TARGET on osx conda-smithy#1889
  • remove c_stdlib{,_version} from always_keep_keys in configure_feedstock.py conda-smithy#1908
  • provide linter hints about adding stdlib, removing outdated ways to do it conda-smithy#1909
  • Linter hint about {{ stdlib("c") }} variable is incorrect for noarch packages conda-smithy#1924
  • fix overzealous linter hint for __osx conda-smithy#1925
  • BUG: MACOSX_SDK_VERSION not participating in merge logic conda-smithy#1927
  • ensure MACOSX_SDK_VERSION is at least c_stdlib_version conda-smithy#1928
  • Add linter hints for issues with OSX configuration in CBC conda-smithy#1929
  • Fix corner-case in stdlib linting conda-smithy#2042
  • take c_stdlib_version into account for MACOSX_DEPLOYMENT_TARGET / DEFAULT_LINUX_VERSION staged-recipes#26012
  • bump conda-forge-ci-setup version & conda-build lower bound staged-recipes#26084
  • Add c_stdlib_version to Linux variants staged-recipes#26206
  • BUG: Add missing channel pinnings staged-recipes#26238
  • Set c_stdlib_version for macos staged-recipes#26578
  • add stdlib to example recipe staged-recipes#27852
  • Add run_constrained to vs activation vc-feedstock#75
  • ensure vs20YY_{{ target }} constrains vs_{{ target }} 20YY conda-forge-repodata-patches-feedstock#701
  • Allow compiling with vs2022 rust-activation-feedstock#50
  • add a test to https://github.com/regro/conda-forge-feedstock-check-solvable
• documentation
  • add announcement for upcoming stdlib migration #2126
  • First iteration of stdlib docs #2155
  • write more extensive docs (after infrastructure page refactor)
• Migrate conda-forge
  • Write piggyback migrator: Piggyback migrator for stdlib-migration regro/cf-scripts#2135
  • Roll it out to one migration (boost 1.84)
  • Fix up Stdlib migration issues regro/cf-scripts#2328 as they appear
  • switch on stdlib-piggyback for all migrations regro/cf-scripts#2372
  • add stdlib piggyback to version_migrator regro/cf-scripts#2459
  • don't remove m2w64_ prefix in stdlib piggyback regro/cf-scripts#2551
• Increase C stdlib baseline on linux/osx 🥳
  • increase minimum macOS deployment target to 10.13 conda-forge-pinning-feedstock#5829
  • Drop cos6 builds conda-forge-pinning-feedstock#6070
• Continued roll-out to stragglers afterwards
  • Keep piggyback running on top of all migrators & version bumps - will ensure feedstocks get {{ stdlib("c") }} if they don't have it already
  • Keep linter active to ensure feedstocks add it even if there are no migrators of version bumps coming through.

[h-vetinari]

@conda-forge/core, I think this would be worth an announcement?

[h-vetinari]

@conda-forge/core, the blockers for regro/cf-scripts#2135 are all close to being resolved. In case someone still has feedback on that PR, please let me know! I'm planning to start this as soon as the we have smithy and https://github.com/regro/conda-forge-feedstock-check-solvable done.

[h-vetinari]

Exciting times - the piggy back migrator was merged, and should start working in the next couple of hours in conjunction with the boost 1.84 migration. 🤩🥳

I'll try to keep an eye on this, but please ping me (or comment here) if something seems to be going wrong!

[h-vetinari]

I went through the open PRs of the boost migration today, and I think we've reached a point where we could switch on the piggyback for all migrations now. In any case, if you find an issue with what the bot is proposing (w.r.t. stdlib-changes), please let us know in regro/cf-scripts#2328.

[carterbox]

Are the Windows pinnings missing c_stdlib_version? It looks like this key exists for unix, but not Windows. I noticed as part of this migration, that using the stdlibc template on Windows causes two Windows compiler wrappers to be installed in my build environment. I assume because the compiler is pinned to vs2019, but the stdlib floats to the latest version (vs2022).

Something like:

c_stdlib_version:
  - 2019                     # [win and x86_64]
  - 2022                     # [win and arm64]

[h-vetinari]

Are the Windows pinnings missing c_stdlib_version?

AFAIU, this shouldn't be necessary (kinda like we don't have c_compiler_version on windows either). That said, the double-compilers are a bit weird (even though vs2019 gets chosen correctly as the compiler according to the logs).

CC @isuruf, since this was based on his suggestion originally.

[h-vetinari]

FWIW, pulling in vs2022 is problematic in the sense that the windows STL (C++ standard library) requires clang >=16, and any recipes using clang <16 + win are broken by pulling in newer vs2022.

There also seem to be some problems in gazebo w.r.t. pulling in vs2022 (CC @Traverso), so considering all that, I'm going to add the version pins for windows for now (which stops pulling in both vs2019 & vs2022, and fixed compilation in conda-forge/dcgp-python-feedstock#24, for example).

[h-vetinari]

Another issue related to pulling in vs2022 in arrow:

-- Providing CMake module for zstdAlt as part of Gandiva CMake package
CMake Error at src/gandiva/precompiled/CMakeLists.txt:44 (message):
  Unsupported MSVC_VERSION=1938

[carterbox]

AFAIU, this shouldn't be necessary (kinda like we don't have c_compiler_version on windows either).

I would say that is because the version (year) of the msvc package is part of the package name whereas gcc and clang do not include the version in the name.

c_compiler:
  - gcc                        # [linux]
  - clang                      # [osx]
  - vs2019                     # [win and x86_64]
  - vs2022                     # [win and arm64]
c_compiler_version:            # [unix]
  - 12                         # [linux]
  - 16                         # [osx]
  - 10                         # [os.environ.get("CF_CUDA_ENABLED", "False") == "True" and linux]
  - 11                         # [os.environ.get("CF_CUDA_ENABLED", "False") == "True" and linux]

[isuruf]

That was my bad. We need a repodata patch that does https://github.com/conda-forge/vc-feedstock/pull/75/files

[xhochy]

It seems that now builds that have previously installing both vsYYYY versions fail now with a solving error. In my case, this is due to rust forcing 2019 https://github.com/conda-forge/rust-activation-feedstock/blob/492cf0058a561fbe9e2666099a4adb77bbe2d0ab/recipe/meta.yaml#L43

We shouldn't have both at the same time, i.e. we need to change rust. But how should the rust-activation be setup now?

[h-vetinari]

But how should the rust-activation be setup now?

It should be fine to use vs_{{ cross_target_platform }}, which will use either vs2019 or vs2022.