Python 2.7 support timeline

[reaperhulk]

We deprecated 2.7 support in our 3.0 release, but we're blocked on very significant performance improvements in our CI (#5246) by continued 2.7 support. This issue is just to discuss and track when we're going to actually drop support. At the moment we continue to see ~20% of our downloads come from Python 2 (I haven't tried to run the numbers on the latest release so that's just a total across all possible versions).

I'd also like to see what our downstream consumers are thinking about regarding 2.7 support in their own projects. /cc @bmw @mattsb42-aws @glyph @bitprophet.

[alex]

Twisted has dropped Python 2, so I assume they're fine with this.

[alex]

Another thing that's blocked by this is us starting to use Rust instead of C. pyo3 is Python3 only.

[bmw]

I think 6 months is a fairly conservative estimate for how much longer Certbot plans to support Python 2.7. We're in the process of migrating our users and how long it takes depends on how that transition goes.

If it helps, all we'd really be worried about is making sure our users get any relevant security fixes while we still support Python 2. New cryptography features and releases could be Python 3 only if any security fixes were backported to a version that supports Python 2 that we could use.

[reaperhulk]

@bmw we've talked a bit about that and as long as the timeline isn't super long we're probably just going to eat the pain to keep our releases and codebase unified. Supporting to end of 2020 won't kill us 😄

[mattsb42-aws]

I'll do some digging and get an official answer.

My personal unofficial answer is that we've historically used pyca as our max support window (mainly because it was the narrowest window of our dependencies at launch), but we do still have a large percentage of our PyPI downloads coming from Python 2. Looking at June 2020, 20% for aws-encryption-sdk, 95% (eek) for aws-encryption-sdk-cli, and 25% for dynamodb-encryption-sdk. That said, we all know that dropping 2 support is a matter of when, not if.

When pyca does drop 2.7 support, would the idea be to immediately block 2.7 installs or only stop testing/supporting 2.7, like you did with 3.4, and only start blocking installs when known incompatible components (ex: pyo3) are added?

[1] https://aws.amazon.com/blogs/compute/continued-support-for-python-2-7-on-aws-lambda/

[alex]

When we drop Python2, we'll do in the same way we've dropped other Python versions: We'll change our setup.py metadata, we'll stop testing it, we'll stop building wheels for it, and we'll delete all the python2 specific code which will result in the library not working.

[reaperhulk]

That means Python 2 will install the last compatible version, but we’re not planning on shipping updated static wheels if/when openssl does security releases. Thus Python 2 users will “work” but become increasingly insecure (absent building it against newer versions themselves) over time.

[reaperhulk]

Just cryptography 3.0 downloads bucketed by Python version:

If you assume both 2.7 and null are Python2 that puts us around 24% right now.

[reaperhulk]

Just as a note, we've also deprecated 3.5 support both due to low usage and because, coupled with dropping 2.7, we can finally support inline type hints. Our current timelines put 3.5 being dropped somewhere around December/January, with 2.7 hopefully dropping around the same time (with some potential flex based on downstream needs).

[bitprophet]

All else equal I'd been thinking I'd cut off 2.7 on my end once downloads get below, I don't know, 10%? 5%?

But I haven't looked at my stats since before CPython dropped the axe, and given where I'm at these days, I wouldn't feel all that bad if a certain major Paramiko dependency provided timeline pressure giving me more of an excuse to simplify my support window.

See paramiko/paramiko#1739 - tl;dr I am still at 40% of downloads coming from Python 2.7 or unspecified, over the last 12mo. Reckon once y'all are ready to drop the axe I'd do a shorter scan to get a more up to date figure.

[reaperhulk]

Statistics for cryptography 3.1 in the past 30 days:

19.5% Python 2.7 (assuming null+2.7 like before) and 7% Python 3.5. So we've had ~4% drop from 2.7 in the last 6-7 weeks.

[reaperhulk]

Latest numbers here for cryptography 3.2.1 only (which has only been out for 2 weeks): Python 2.7 is 22.9% with another 1.5% unknown. A significant upswing. For now we're still planning on dropping support despite this significant percentage since we can significantly improve our personal quality of life from a maintenance and development perspective, but we'll keep watching this.

[WhyNotHugo]

I think it's important to remember that dropping Python2 does not leave those users in the dark; they can continue using the latest release that supports Python2 indefinitely.

When it comes to updates, they're not getting security updates from Python or pretty much any other library anyway either; the lack of support is clear and not something that should burden cryptography developers.

[potiuk]

I think it's time to say goodbye to 2.7 and possibly even yank 3.2 and 3.2.1 release. I have just debugged and found a very annoying issue in 2.7 version of https://github.com/apache/airflow which I pinpointed to upgrade to latest cryptography upgrade (I tested different versions and 3.1.1 and 3.1.2 worked fine - 3.2 and 3.2.1 are broken.

I am not sure if I should open a separate issue for that - or whether you decide to "get rid" of python 2. I'd love to help you and put one more proverbial nail into Python2 coffin.

The problem:

We are using Fernet Key in Apache Airflow and we use it to store encrypted data in the database. So we are really doing this:

self._extra = fernet.encrypt(bytes(value, 'utf-8')).decode()

to encrypt, and

fernet.decrypt(bytes(self._extra, 'utf-8')).decode()

to decrypt some values.

This worked fine, and we usually get as output exactly what we started with before encrypt/decrypt sequence, for all python versions until cryptography 3.2.

But that has changed in 3.2, all continues to work for python 3 but for python 2 when we encrypt/decrypt this:

'{"s3_config_format": "aws", "profile": "test", "s3_config_file": "aws-credentials", "region_name": "us-east-1"}'

we get this:

u'b\'{"s3_config_format": "aws", "profile": "test", "s3_config_file": "aws-credentials", "region_name": "us-east-1"}\''

Not only Unicode but with the "b" and some escaped characters.
While I can guess where it came from, it's likely not really desired outcome :).

My proposal: Yank 3.2 and 3.2.1 and release 3.2.2 or even 3.3 without python 2 support.

Let's kill that Python 2 vampire once and for all.

[reaperhulk]

@potiuk #5548 fixed this I believe. The 3.3 release will include that fix for a final py2 compatible release.

[potiuk]

Cool. Glad it's taken care of. Thanks @reaperhulk for the info! Does not change my feelings towards Python 2 after today's debugging.

BTW. I do recommend yanking for both 3.2, 3.2.1 in PyPI once 3.3 is out -> it works exactly as you'd expect and we've only recently learned about it: https://pypi.org/help/#yanked

[potiuk]

FYI. We've workarounded it for now with apache/airflow@bbf122b

[alex]

👋 @bmw @mattsb42-aws @bitprophet I just wanted to flag that we're probably a few weeks out from our 3.3 release, which will be the last cryptography release to support Python 2 (a 3.4 release which is python3 only come at some undetermined point). If this is going to be a problem for y'all, please flag now.

[bitprophet]

I'm committed to dropping 2 on my end sometime in 2021. Could be sooner, could be later.

Assuming your 3.4 release has the right metadata prompting pip on Python 2 to disregard it, I don't think I need to bother pinning to 3.3 on my end, yea? (Unless a user is on a Very Old Pip in which case I have a harder time caring.)

[alex]

Yes, we have the correct metadata, so modern pips will do the right thing, no pinning required.

…

On Fri, Dec 4, 2020 at 5:17 PM Jeff Forcier ***@***.***> wrote: I'm committed to dropping 2 on my end sometime in 2021. Could be sooner, could be later. Assuming your 3.4 release has the right metadata prompting pip on Python 2 to disregard it, I don't think I need to bother pinning it on my end, yea? (Unless a user is on a Very Old Pip in which case I have a harder time caring.) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5359 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBGZLPCZCRIXBUT4AG3STFNZHANCNFSM4PKZN6PA> .

-- All that is necessary for evil to succeed is for good people to do nothing.

[alex]

We just did our 3.3 release. We'll start landing PRs which remove our 2.7 support shortly.

Thanks to everyone who contributed to this conversation, see y'all on py3k in 2021!