Timeline for closing out OpenSSL 3 migration / dropping OpenSSL 1.1.1

[h-vetinari]

One of the longest-running migrations (and painful for the solver, see here) has been the one for OpenSSL 3.

However, due to the efforts of many people, this is now turning into home stretch (more below).

The reason for opening this issue is that people might not be ready for a world in which there are no more 1.1.1 builds, if only due to the sheer lengths of time it's been there.

In particular, anaconda does not yet build for OpenSSL 3, which would mean that after dropping 1.1.1, many of their packages would not be co-installable with continuing-to-be-migrated conda-forge ones anymore.

I don't know Anaconda's timeline, but the EOL for OpenSSL 1.1.1 is in 2023/09/11, so only 9 months away.
CC @chenghlee @jezdez @skupr-anaconda @AndrewVallette

Status of remaining OpenSSL 3 PRs

(not everything shows up in the migrator status; I also used the recently much-improved GH search)

Essential

• Rebuild for openssl3 qt-webengine-feedstock#19
• Rebuild for openssl3 rstudio-feedstock#13
• tensorflow 2.11 tensorflow-feedstock#286

Ready to merge

• Rebuild maintenance branches for aws-sdk-cpp 1.9 & openssl 3? arrow-cpp-feedstock#918
• Migrations (arrow, geos, libarchive, openssl) heavydb-ext-feedstock#33
• Rebuild for openssl3 (main branch = v10.1.x) htcondor-feedstock#136
• Rebuild for openssl3 (v10.0.x branch) htcondor-feedstock#134
• Rebuild for openssl3 (v9.0.x branch) htcondor-feedstock#128
• Rebuild for openssl3 katago-feedstock#2
• Drops openssl pin kubo-feedstock#11
• Rebuild for openssl3 m2crypto-feedstock#70
• Rebuild for openssl3 subversion-feedstock#18
• virtuoso-opensource v7.2.8 virtuoso-opensource-feedstock#7
• Rebuild for openssl3 yarp-feedstock#18

Todo:

• Rebuild for openssl3 capnproto-feedstock#32 (single failing test with OpenSSL 3)
• Rebuild for openssl3 libpcap-feedstock#14 (newly added feedstock; windows build errors)
• https://github.com/conda-forge/libtins-feedstock (migrator thinks it depends on libpcap, however one is # [osx] and one is # [win] for OpenSSL)
• Rebuild for openssl3 rb-eventmachine-feedstock#4 (dependent on ruby 3 migration)

Problematic

Ruby 2.5 & 2.6 will very likely not support OpenSSL 3

• Rebuild for openssl3 (2.6.x branch) ruby-feedstock#110
• Rebuild for openssl3 (2.5.x branch) ruby-feedstock#109
• Rebuild for openssl3 (14.x) nodejs-feedstock#242 (this still has some users, but upstream is not compatible)

Started a ruby 3 migration:

• Update pin for ruby #2347

Feedstocks that still pin to 1.1.1

(sidenote: loving the new GH search functionality 🤩 )

• Rely on global pinning rvlib-feedstock#22
• Remove openssl pins tmate-feedstock#16

Abandoned / Obsolete Packages

• Rebuild for openssl3 hdf5-feedstock#168 (for 1.10 branch; global pinning is already at 1.12)
• Rebuild for openssl3 mysql-connector-c-feedstock#21 (5 years old; seems to have been replaced by mysql-connector-cpp)
• Rebuild for openssl3 postgres-xl-feedstock#8 (last release ~4 years ago)
• salt (pinned to openssl 1.0.*!)

PS. There are some grumblings in upstream CPython about not moving to OpenSSL 3, and the cryptography devs are also not enthusiastic about OpenSSL 3 (despite supporting it from the start, and defaulting to it nowadays). However, there are no really compelling alternatives, so assuming OpenSSL 3 stabilizes a bit in the perception of the community, it might be the least bad option after all.

[h-vetinari]

From the side of conda-forge, aside from all other concerns, there's obviously an interest in closing this migration sooner rather than later, because it keeps many feedstocks on a doubled build matrix, which is especially painful for packages that need manual builds.

[pkgw]

Considering conda-forge on its own, I would really like to declare this migration complete (or, "as complete as it's going to get") sooner rather than later — the massive build matrixing is indeed very painful, IMO. I don't understand the migration machinery super well but I'd be tempted to say that we declare it done when the "Essential" packages above are dealt with, and mop up the rest as their maintainers deal with them.

However, the status with regards to Anaconda is certainly important, and I don't think we should drop support for the 1.x series until we understand the timeline there. So hopefully at some point we'll get an update on that.

Finally, regarding CPython and cryptography, I think that's actually a bit of a red herring — it's not like they're going to stick with OpenSSL 1.x rather than 3.x, and indeed 3.x is now supported; it's just that they may wish to move away from OpenSSL altogether in the future. That would be something to deal with, but is orthogonal to the OpenSSL update.

[h-vetinari]

Finally, regarding CPython and cryptography, I think that's actually a bit of a red herring — it's not like they're going to stick with OpenSSL 1.x rather than 3.x, and indeed 3.x is now supported

It's supported in cryptography, but incomplete in CPython. Also, there's at least some speculation about pushing 1.1.1 past its EOL:

I think the options are:

• Wait and see: half the internet is in the same position as us re: openssl 1.1.1/3.0, so as the planned 1.1.1 EOL date approaches I expect some kind of consensus will develop about what to do – whether that’s pushing back the EOL date, or openssl upstream getting things together to fix 3.0, or a group coalescing to maintain 1.1.1 past the EOL or what.
• [...]

[jakirkham]

We discussed this at the core meeting. The main thing to discuss from our perspective is when we consider this done. Given OpenSSL 3 will become the only supported version in the Fall of 2023 and we are still migrating a few things (notably TensorFlow), we thought moving to OpenSSL 3 only by Spring of 2023 seemed like a reasonable timeline to be done, but we can pull this in sooner if the remaining pieces are in before then. Will discuss again in the New Year to see where things are and if that needs to change.

[chrisburr]

As there hasn't been a contradicting view: I'm keen to give this a little longer as I still find I have too many issues with OpenSSL 3.x for it to be usable (hopefully I'll deal with them over the next few weeks).

[jakirkham]

Yeah the general consensus at the meeting was this won't be closed until 2023. Spring 2023 seemed reasonable by most of those gathered.

Maybe it is worth spelling out the specific issues encountered so they are tracked here?

[jakirkham]

However, the status with regards to Anaconda is certainly important, and I don't think we should drop support for the 1.x series until we understand the timeline there. So hopefully at some point we'll get an update on that.

@chenghlee could you or someone on the Anaconda side please comment on your OpenSSL 1 to 3 plans?

[chenghlee]

could you or someone on the Anaconda side please comment on your OpenSSL 1 to 3 plans?

Anaconda is planning to kick off its repositories' OpenSSL 3.x migration in early 2023; the goal is definitely to get it done before September 2023.

[hmaarrfk]

I think i resolved the tmate issue a while back. it was a different package that had an issuie.

[h-vetinari]

I think i resolved the tmate issue a while back. it was a different package that had an issuie.

tmate still pins openssl 1 in the recipe, so it's not done yet.

[h-vetinari]

I think i resolved the tmate issue a while back. it was a different package that had an issuie.

tmate still pins openssl 1 in the recipe, so it's not done yet.

On second look, those pins are only there for testing. Still, I guess we should ensure that it works for both...

[h-vetinari]

Tensorflow is finally done! 🥳

In fact, because the manual builds there are so much work (~roughly one build per overnight run), it already preempted the closing of the migration and only built for OpenSSL 3.

That leaves the ruby ecosystem, where the ruby 3 migration got started (ruby 2.x won't support OpenSSL 3), but it seems that there were some layout changes and several packages there don't build anymore.

[pkgw]

At this point in the process, it might make sense to reorganize the (very helpful!) lists at the top of this issue to more clearly separate what's done and what's left to do. As far as I can tell, the remaining to-dos are all approximately equal priority?

[h-vetinari]

At this point in the process, it might make sense to reorganize the (very helpful!) lists at the top of this issue to more clearly separate what's done and what's left to do.

I tried to do this already - all that's left (to my knowledge) is under the "Todo:" heading. I don't know how to judge the importance of ruby, but from my POV the remaining packages are not important enough to hold up the closing of the migration.

PS. Happy for suggestions about how to make the OP more clear

[pkgw]

@h-vetinari Yes, at this point I think the closing of the migration is more about a policy choice of when to start de-matrixing OpenSSL 1.1, rather than about resolving technical blockers.

OK if I edit the top post?

[h-vetinari]

OK if I edit the top post?

Go right ahead!

[jakirkham]

Status is here. Looks all PRs are opened and nearly all merged.

There's a PR to close the migration ( #3892 ). Let's discuss wrapping this up there.

[h-vetinari]

Status is here. Looks all PRs are opened and nearly all merged.

For various reasons¹, the list in the OP here is more comprehensive than what's shown on the migrator status.

There's a PR to close the migration ( #3892 ). Let's discuss wrapping this up there.

Would suggest to keep it in this issue, but don't mind moving to the PR if that's what people prefer.

Footnotes

1. e.g. the migrator will not issue PRs if the feedstock has a hard pin on OpenSSL, and I explicitly looked for those too ↩

[jakirkham]

Question on this item from the list:

https://github.com/conda-forge/libtins-feedstock (migrator thinks it depends on libpcap, however one is # [osx] and one is # [win] for OpenSSL)

Should we open a PR manually migrating this feedstock?

Edit: Actually looks like this was done quite some time ago ( conda-forge/libtins-feedstock#2 )

[jakirkham]

On a different list item:

conda-forge/rb-eventmachine-feedstock#4 (dependent on ruby 3 migration)

We now have a Ruby 3 migrator for rb-eventmachine ( conda-forge/rb-eventmachine-feedstock#5 ). Have merged the OpenSSL 3 migrator into the Ruby 3 migrator. There are some build issues that need debugging

[jaimergp]

Hello folks! As per the last core meeting, there have been no comments against closing this migration once and for all. In other words, we can proceed and close the OpenSSL 3 migration.

cc @h-vetinari

[jakirkham]

Marking PR ( #3892 ) ready for review. Think we can merge it

[traversaro]

conda-forge/libpcap-feedstock#14 (newly added feedstock; windows build errors)

This is now fixed by conda-forge/libpcap-feedstock#18 .

[h-vetinari]

Just an FYI for anyone curious: OpenSSL 1.1.1 is end-of-life as of today.