Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for older Rust versions (1.41)? #1420

Closed
tiran opened this issue Feb 10, 2021 · 15 comments · Fixed by #1421
Closed

Support for older Rust versions (1.41)? #1420

tiran opened this issue Feb 10, 2021 · 15 comments · Fixed by #1421

Comments

@tiran
Copy link
Contributor

tiran commented Feb 10, 2021

Feature request

pyca/cryptography users have been asking for support of Debian Buster (current stable), Alpine 3.12, and other platforms with older Rust compiler versions. The common baseline seems to be 1.41. pyo3 does not compile with Rust 1.41 and seem to require at least Rust 1.45. Would it be feasible to support 1.41?

https://buildd.debian.org/status/package.php?p=rustc&suite=buster

🌍 Environment

  • Your operating system and version: Debian Buster, Alpine <= 3.12
  • Your python version: /
  • How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: system python
  • Your Rust version (rustc --version): 1.41
  • Your PyO3 version: git master
  • Have you tried using latest PyO3 master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?:

Reproducer

$ podman run -ti debian:stable
# apt update
# apt install -y git cargo python3-dev build-essential
# git clone https://github.com/PyO3/pyo3.git
# cd pyo3
# rustc -V
rustc 1.41.1
# cargo build
   Compiling pyo3-macros-backend v0.13.1 (/root/pyo3/pyo3-macros-backend)
error[E0658]: subslice patterns are unstable
   --> pyo3-macros-backend/src/pyfunction.rs:177:36
    |
177 |         [_first_attr, second_attr, ..] => bail_spanned!(
    |                                    ^^
    |
    = note: for more information, see https://github.com/rust-lang/rust/issues/62254

error[E0658]: subslice patterns are unstable
   --> pyo3-macros-backend/src/pymethod.rs:707:14
    |
707 |         [py, args @ ..] if utils::is_python(&py.ty) => (Some(py), args),
    |              ^^^^^^^^^
    |
    = note: for more information, see https://github.com/rust-lang/rust/issues/62254

error[E0658]: use of unstable library feature 'str_strip': newly added
   --> pyo3-macros-backend/src/method.rs:144:18
    |
144 |                 .strip_prefix(prefix)
    |                  ^^^^^^^^^^^^
    |
    = note: for more information, see https://github.com/rust-lang/rust/issues/67302

error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0658`.
error: could not compile `pyo3-macros-backend`.
warning: build failed, waiting for other jobs to finish...
error: build failed
@birkenfeld
Copy link
Member

birkenfeld commented Feb 10, 2021

There was some discussion of this in #1076 leading to the current policy. However, with wider adoption comes heavier compatibility pressure, and IMO we should make cryptography's life a little easier if possible - it is already facing a lot of backlash for the Rust dependency.

@mtreinish
Copy link
Contributor

I will second this sentiment in retworkx I reverted the pyo3 0.13.0 bump (Qiskit/rustworkx#237) because it would break compatibility for piwheels which is using the debian buster rustc package which is also 1.41 (although in that case the ndarray and rust-numpy versions also complicates because the latest ndarray version has a msrv of 1.42).

birkenfeld added a commit to birkenfeld/pyo3 that referenced this issue Feb 10, 2021
This version is currently supported by Debian stable and Alpine Linux.

Fixes PyO3#1420
@tiran
Copy link
Contributor Author

tiran commented Feb 10, 2021

@birkenfeld Thanks Georg! Your PR does the trick.

# cargo build
    Updating crates.io index
  Downloaded indoc v0.3.6
  Downloaded paste v0.1.18
  Downloaded indoc-impl v0.3.6
  Downloaded proc-macro-hack v0.5.19
  Downloaded paste-impl v0.1.18
   Compiling proc-macro-hack v0.5.19
   Compiling pyo3-macros-backend v0.13.1 (/root/pyo3/pyo3-macros-backend)
   Compiling paste-impl v0.1.18
   Compiling indoc-impl v0.3.6
   Compiling paste v0.1.18
   Compiling pyo3-macros v0.13.1 (/root/pyo3/pyo3-macros)
   Compiling indoc v0.3.6
   Compiling pyo3 v0.13.1 (/root/pyo3)
    Finished dev [unoptimized + debuginfo] target(s) in 14.13s

birkenfeld added a commit to birkenfeld/pyo3 that referenced this issue Feb 10, 2021
This version is currently supported by Debian stable and Alpine Linux.

Fixes PyO3#1420
@davidhewitt
Copy link
Member

davidhewitt commented Feb 10, 2021

I'm in favour of backporting support for older versions. (Thanks @birkenfeld!)

We should probably use this issue to discuss what a better Rust support policy is than what we last settled on.

To simplify packaging changes downstream, I suggest we keep MSRV bumps to the same release where we deprecate old Python versions. E.g. this year 3.6 will become end-of-life. In the release we drop Python 3.6, we can also bump the MSRV.

A reasonable floor could be the minimum version supported by the current Debian / RHEL / Alpine versions at the time of the MSRV bump. I'm open to other suggestions too.

To keep version support broad, I propose we also make use of feature gating to add optional support for newer functionality. E.g. min_const_generics: we could definitely make a feature to support that on Rust 1.51+.

@tiran
Copy link
Contributor Author

tiran commented Feb 10, 2021

How much effort is Python 3.6 compatibility for you? I would appreciate if you could keep Python 3.6 support for a little longer. CentOS 8 and RHEL 8 use Python 3.6 as platform Python interpreter. I just started the process to get setuptools_rust into RHEL 8 buildroot environment today...

  • RHEL 8.2.1: rust 1.42
  • RHEL 8.3: rust 1.45
  • CentOS 8 (as of today): rust 1.45

Next RHEL 8 update should contain more recent versions of Rust.

@zeha
Copy link

zeha commented Feb 10, 2021

Chiming in here because I asked for 1.41.1 (=Debian buster) support over in pyca/cryptography#5769.
We deploy our own Python 3.7 (soon 3.8) builds on top of Debian buster. So, older Python-version support does not matter that much to me, but older Rust-version support does.

Thanks for working on this, much appreciated!

@reaperhulk
Copy link

cryptography typically uses the only metrics we have available (PyPI statistics) to help determine when we drop versions. In our case, here's the last 30 days of downloads grouped by version (we have dropped 2.7 in the release that added rust, so you can ignore that line item):
image
Based on this I'd definitely urge you to keep 3.6 despite upstream dropping support. It is very heavily used.

@davidhewitt
Copy link
Member

Perhaps I didn't communicate well; Python 3.6 end-of-life is in December of this year. We won't drop support in PyO3 until after that - so it could be as long as a year before that release happens. Or do you think we'll need support for even longer support than that?

@reaperhulk
Copy link

@davidhewitt It's hard to say, but the primary driver lowering 3.6 usage will be people moving off distributions like Ubuntu 18.04. We've found in the past that there's reasonably rapid migration off the oldest LTS when LTS+2 comes out (e.g. 22.04 in April 2022), but 18.04's EOL is April 2023, so at worst ~a year more.

@reaperhulk
Copy link

That's probably a long-winded way of saying that I would expect 3.6 usage to be quite high at the end of 2021, but start dropping off relatively quickly starting in May 2022. cryptography typically considers deprecating support and removing it relatively quickly once usage drops < 5%, but PyO3 may choose to use different thresholds! In an ideal world we would always be willing to drop before you are, but I recognize you may have drivers that are different from ours in the future. 😄

@tiran
Copy link
Contributor Author

tiran commented Feb 11, 2021

I hate to break it to you, but I expect 3.6 to drop off a bit slower than that. The default Python interpreter on RHEL 8 and CentOS 8 is Python 3.6. RHEL 9 hasn't been announced yet. This fact is likely prolonging prevalence of Python 3.6.

Please note that users have options. The Python maintenance team at Red Hat did a fantastic job and made Python 3.8 a fully supported first-class-citizens in RHEL. Python 3.8 is available in the default AppStream. dnf install python3.8 provides /usr/bin/python3.8 with working venv and /usr/bin/pip3.8. Python 3.9 will be available soon, too.

@birkenfeld
Copy link
Member

birkenfeld commented Feb 11, 2021

As long as cryptography is happy with what it gets in PyO3 0.13, this doesn't have to mean a need to support 3.6 in all later versions though. It just means a commitment to release bugfix versions, if serious bugs surface, of 0.13 (or the latest one to support 3.6).

birkenfeld added a commit to birkenfeld/pyo3 that referenced this issue Feb 11, 2021
This version is currently supported by Debian stable and Alpine Linux.

Fixes PyO3#1420
birkenfeld added a commit to birkenfeld/pyo3 that referenced this issue Feb 11, 2021
This version is currently supported by Debian stable and Alpine Linux.

Fixes PyO3#1420
birkenfeld added a commit to birkenfeld/pyo3 that referenced this issue Feb 11, 2021
This version is currently supported by Debian stable and Alpine Linux.

Fixes PyO3#1420
@kngwyu
Copy link
Member

kngwyu commented Feb 11, 2021

Thanks for reporting, we are happy to release the patch version 😉

A reasonable floor could be the minimum version supported by the current Debian / RHEL / Alpine versions at the time of the MSRV bump. I'm open to other suggestions too.

👍🏼

RHEL 8.2.1: rust 1.42
RHEL 8.3: rust 1.45
CentOS 8 (as of today): rust 1.45

So perhaps Debian is the most conservative?

@reaperhulk
Copy link

As long as cryptography is happy with what it gets in PyO3 0.13, this doesn't have to mean a need to support 3.6 in all later versions though. It just means a commitment to release bugfix versions, if serious bugs surface, of 0.13 (or the latest one to support 3.6).

Ah, yes this is a good point. And since setuptools-rust is only a build time dependency we don't even have to worry about pinning conflicts there. That's nice!

As an aside, I personally would like to thank every PyO3 maintainer. The open source ecosystem requires projects to stand on each others' shoulders and @alex and I couldn't do what we're doing in cryptography without the foundational work this team chose to tackle.

@davidhewitt
Copy link
Member

davidhewitt commented Feb 11, 2021

👐 @alex did some much appreciated work to help get the abi3 support here; we definitely benefit from help with implementation and plenty of new interest by having you folks embrace Rust!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants