Support for older Rust versions (1.41)?

[tiran]

Feature request

pyca/cryptography users have been asking for support of Debian Buster (current stable), Alpine 3.12, and other platforms with older Rust compiler versions. The common baseline seems to be 1.41. pyo3 does not compile with Rust 1.41 and seem to require at least Rust 1.45. Would it be feasible to support 1.41?

https://buildd.debian.org/status/package.php?p=rustc&suite=buster

🌍 Environment

• Your operating system and version: Debian Buster, Alpine <= 3.12
• Your python version: /
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: system python
• Your Rust version (rustc --version): 1.41
• Your PyO3 version: git master
• Have you tried using latest PyO3 master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?:

Reproducer

$ podman run -ti debian:stable
# apt update
# apt install -y git cargo python3-dev build-essential
# git clone https://github.com/PyO3/pyo3.git
# cd pyo3
# rustc -V
rustc 1.41.1

# cargo build
   Compiling pyo3-macros-backend v0.13.1 (/root/pyo3/pyo3-macros-backend)
error[E0658]: subslice patterns are unstable
   --> pyo3-macros-backend/src/pyfunction.rs:177:36
    |
177 |         [_first_attr, second_attr, ..] => bail_spanned!(
    |                                    ^^
    |
    = note: for more information, see https://github.com/rust-lang/rust/issues/62254

error[E0658]: subslice patterns are unstable
   --> pyo3-macros-backend/src/pymethod.rs:707:14
    |
707 |         [py, args @ ..] if utils::is_python(&py.ty) => (Some(py), args),
    |              ^^^^^^^^^
    |
    = note: for more information, see https://github.com/rust-lang/rust/issues/62254

error[E0658]: use of unstable library feature 'str_strip': newly added
   --> pyo3-macros-backend/src/method.rs:144:18
    |
144 |                 .strip_prefix(prefix)
    |                  ^^^^^^^^^^^^
    |
    = note: for more information, see https://github.com/rust-lang/rust/issues/67302

error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0658`.
error: could not compile `pyo3-macros-backend`.
warning: build failed, waiting for other jobs to finish...
error: build failed

[birkenfeld]

There was some discussion of this in #1076 leading to the current policy. However, with wider adoption comes heavier compatibility pressure, and IMO we should make cryptography's life a little easier if possible - it is already facing a lot of backlash for the Rust dependency.

[mtreinish]

I will second this sentiment in retworkx I reverted the pyo3 0.13.0 bump (Qiskit/rustworkx#237) because it would break compatibility for piwheels which is using the debian buster rustc package which is also 1.41 (although in that case the ndarray and rust-numpy versions also complicates because the latest ndarray version has a msrv of 1.42).

[tiran]

@birkenfeld Thanks Georg! Your PR does the trick.

# cargo build
    Updating crates.io index
  Downloaded indoc v0.3.6
  Downloaded paste v0.1.18
  Downloaded indoc-impl v0.3.6
  Downloaded proc-macro-hack v0.5.19
  Downloaded paste-impl v0.1.18
   Compiling proc-macro-hack v0.5.19
   Compiling pyo3-macros-backend v0.13.1 (/root/pyo3/pyo3-macros-backend)
   Compiling paste-impl v0.1.18
   Compiling indoc-impl v0.3.6
   Compiling paste v0.1.18
   Compiling pyo3-macros v0.13.1 (/root/pyo3/pyo3-macros)
   Compiling indoc v0.3.6
   Compiling pyo3 v0.13.1 (/root/pyo3)
    Finished dev [unoptimized + debuginfo] target(s) in 14.13s

[davidhewitt]

I'm in favour of backporting support for older versions. (Thanks @birkenfeld!)

We should probably use this issue to discuss what a better Rust support policy is than what we last settled on.

To simplify packaging changes downstream, I suggest we keep MSRV bumps to the same release where we deprecate old Python versions. E.g. this year 3.6 will become end-of-life. In the release we drop Python 3.6, we can also bump the MSRV.

A reasonable floor could be the minimum version supported by the current Debian / RHEL / Alpine versions at the time of the MSRV bump. I'm open to other suggestions too.

To keep version support broad, I propose we also make use of feature gating to add optional support for newer functionality. E.g. min_const_generics: we could definitely make a feature to support that on Rust 1.51+.

[tiran]

How much effort is Python 3.6 compatibility for you? I would appreciate if you could keep Python 3.6 support for a little longer. CentOS 8 and RHEL 8 use Python 3.6 as platform Python interpreter. I just started the process to get setuptools_rust into RHEL 8 buildroot environment today...

• RHEL 8.2.1: rust 1.42
• RHEL 8.3: rust 1.45
• CentOS 8 (as of today): rust 1.45

Next RHEL 8 update should contain more recent versions of Rust.

[zeha]

Chiming in here because I asked for 1.41.1 (=Debian buster) support over in pyca/cryptography#5769.
We deploy our own Python 3.7 (soon 3.8) builds on top of Debian buster. So, older Python-version support does not matter that much to me, but older Rust-version support does.

Thanks for working on this, much appreciated!

[reaperhulk]

cryptography typically uses the only metrics we have available (PyPI statistics) to help determine when we drop versions. In our case, here's the last 30 days of downloads grouped by version (we have dropped 2.7 in the release that added rust, so you can ignore that line item):

Based on this I'd definitely urge you to keep 3.6 despite upstream dropping support. It is very heavily used.

[davidhewitt]

Perhaps I didn't communicate well; Python 3.6 end-of-life is in December of this year. We won't drop support in PyO3 until after that - so it could be as long as a year before that release happens. Or do you think we'll need support for even longer support than that?

[reaperhulk]

@davidhewitt It's hard to say, but the primary driver lowering 3.6 usage will be people moving off distributions like Ubuntu 18.04. We've found in the past that there's reasonably rapid migration off the oldest LTS when LTS+2 comes out (e.g. 22.04 in April 2022), but 18.04's EOL is April 2023, so at worst ~a year more.

[reaperhulk]

That's probably a long-winded way of saying that I would expect 3.6 usage to be quite high at the end of 2021, but start dropping off relatively quickly starting in May 2022. cryptography typically considers deprecating support and removing it relatively quickly once usage drops < 5%, but PyO3 may choose to use different thresholds! In an ideal world we would always be willing to drop before you are, but I recognize you may have drivers that are different from ours in the future. 😄

[tiran]

I hate to break it to you, but I expect 3.6 to drop off a bit slower than that. The default Python interpreter on RHEL 8 and CentOS 8 is Python 3.6. RHEL 9 hasn't been announced yet. This fact is likely prolonging prevalence of Python 3.6.

Please note that users have options. The Python maintenance team at Red Hat did a fantastic job and made Python 3.8 a fully supported first-class-citizens in RHEL. Python 3.8 is available in the default AppStream. dnf install python3.8 provides /usr/bin/python3.8 with working venv and /usr/bin/pip3.8. Python 3.9 will be available soon, too.

[birkenfeld]

As long as cryptography is happy with what it gets in PyO3 0.13, this doesn't have to mean a need to support 3.6 in all later versions though. It just means a commitment to release bugfix versions, if serious bugs surface, of 0.13 (or the latest one to support 3.6).

[kngwyu]

Thanks for reporting, we are happy to release the patch version 😉

A reasonable floor could be the minimum version supported by the current Debian / RHEL / Alpine versions at the time of the MSRV bump. I'm open to other suggestions too.

👍🏼

RHEL 8.2.1: rust 1.42
RHEL 8.3: rust 1.45
CentOS 8 (as of today): rust 1.45

So perhaps Debian is the most conservative?

[reaperhulk]

As long as cryptography is happy with what it gets in PyO3 0.13, this doesn't have to mean a need to support 3.6 in all later versions though. It just means a commitment to release bugfix versions, if serious bugs surface, of 0.13 (or the latest one to support 3.6).

Ah, yes this is a good point. And since setuptools-rust is only a build time dependency we don't even have to worry about pinning conflicts there. That's nice!

As an aside, I personally would like to thank every PyO3 maintainer. The open source ecosystem requires projects to stand on each others' shoulders and @alex and I couldn't do what we're doing in cryptography without the foundational work this team chose to tackle.

[davidhewitt]

👐 @alex did some much appreciated work to help get the abi3 support here; we definitely benefit from help with implementation and plenty of new interest by having you folks embrace Rust!