Version 0.15.2 with support for Pypy 3.9 and Python 3.6

[felipou]

Hi! First of all, thanks for building and maintaining this project!

The cryptography project, which depends on this PyO3, is currently unable to provide support for Pypy with Python 3.9, because a large portion of their users still use Python 3.6, which was dropped in version 0.16 of PyO3, the same that first includes support for Pypy 3.9.

This is the relevant issue: pyca/cryptography#6924

It has been suggested that a new patch release could be used to solve this issue, for example a version 0.15.2 adding support for Pypy 3.9 while also keeping support for Python 3.6.

Is that feasible? Would a pull request be accepted? If yes, can you give me some pointers or tips so that I can start working on this?

[birkenfeld]

Personally, I think that is reasonable. Maybe make 0.15 a kind of "LTS" version?

[alex]

First, thanks! In the immediate term I think PyPy 3.9 support is the only compatibility issue we're currently facing with pyo3 0.15.

To take a broader view, I'd love to think about what ways we (pyca/cryptography) can maintain our current, relatively conservative, support policies without placing too much burden on packages like pyo3.

Notwithstanding my long-running frustration at support things because distros support them beyond the life of upstream (https://alexgaynor.net/2015/mar/30/red-hat-open-source-community/), I expect we'll continue to support Python 3.6 for quite a while; we also currently support rustc 1.41.0 and we'll be conservative in increasing that as well. I'd like to believe there's an outcome where we can maintain these without burdening the pyo3 team too much!

[mejrs]

Would it be feasible to fix cryptography to be compatible with both pyo3 0.15 and 0.16, and just choose 0.16 for pypy 3.9?

As far as I can see in pyca/cryptography#6935 the only breaking change is the removed PartialEq impl, but substituting it with a.as_ptr() == b.as_ptr() works with both pyo3 versions.

[alex]

Hmm, a few thoughts:

1. There's a bunch of other changes in that PR besides the .is() change, I guess those were all warnings?
2. Theoretically I'm ok with that, but I'm not actually sure how we'd operationalize it. Right now we ship a Cargo.lock in our sdist. How would we switch which versions we use 0.15 vs. 0.16?

[mejrs]

1. I think everything else is just warnings, but I've not tried compiling it myself.
2. You would have to run something like cargo update -p pyo3 --precise <version> as part of your workflow.

What kind of timescale do you have for dropping Python 3.6 support? Personally I'm lukewarm about maintaining multiple pyo3 versions, especially if it's for years..

[alex]

I was just pointed at https://github.com/TrueLayer/reqwest-middleware/blob/main/reqwest-tracing/Cargo.toml as a potential method to address this, and then we set the --features we want in setup.py.

Our plan is to be data driven in when we drop it, based on usage going low enough. https://pypistats.org/packages/cryptography shows that 3.6 is ~13% ATM, I think we'd want it to be <5% before we consider dropping.

Based on supporting both looking tractable, I think I'm going to try it!

[alex]

Unfortunately the pattern I linked doesn't work, because cargo doesn't allow multiple dependencies to specify the same links value.

[davidhewitt]

I would be happy to release a PyO3 0.15.2 with PyPy 3.9 support. Unfortunately I don't have time to implement this release myself at present, so community input is very welcome. @felipou I suggest looking at #2217 and #2143

Note that we split pyo3-ffi as a separate crate in 0.16 so I'm unsure how easy cherry picking will be.

[messense]

Opened #2262

[felipou]

You people are awesome! 👏