Official support for Python 3.12

[SnoopJ]

Requested Feature

Python 3.12 has entered beta, so it's a good time to start thinking about Sopel on that version.

• Check Sopel tests against 3.12 (done in Add Python 3.12 support #2516)
  • Installing dev-requirements.txt was blocked by lack of support in greenlet, but a PR has since been merged with a fix for this, presumably the next release will include that fix.
  • Looks like we may need to adjust the version requirement for setuptools, I see the following error with setuptools == 67.8.0 but pinning setuptools < 67 in build-system.requires resolves it.

  AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
  

  • There was additional upstream breakage that has now been fixed in aiohttp (fixed on develop, should be included in 3.9) and in frozenlist (released in 1.4.0)
• Update CI to run with 3.12 (done in Add Python 3.12 support #2516)
  • Looks like the python-versions Action is capable of providing the 3.12.0b1 pre-release, so we could run against the beta version(s) before the official release (scheduled for 2 October 2023).
• Update classifiers list in pyproject.toml (done in Add Python 3.12 support #2516)
• [ ] Check on 3.12 support concerns for official non-core plugins Not core's problem — @dgw

[Exirel]

Just a note here: there are feature we use in Py3.7 that are deprecated in Py3.9, Py3.10, or Py3.11, and may be removed in version as early as Py3.10 and Py3.11. So I'm pretty sure it'll be quite hard to support such a big range of Python version.

I believe it will be more reasonable to drop anything below Py3.9 to support Py3.12.

[SnoopJ]

3.7 reaches its EOL in the next month, so that one is basically free. If we can continue to support 3.8, that would be nice to our downstream, although I don't know what similar issues there might be for that version.

[SnoopJ]

This PR is probably a better thing to track for the upstream blockage from aiohttp than the issue linked above: bartfeenstra/betty#1008

This issue is still blocked, but it sounds like 3.12 support might be in the next release if things go well.

[SnoopJ]

The entire Sopel test suite passes on 3.12.0rc1 with a bit of hand-holding.

1317 passed, 8 xfailed, 1 warning in 39.80s

Recipe to get a Sopel install on this version:

• Install greenlet from source (pip install git+https://github.com/python-greenlet/greenlet) (c2416a1)
• Install aiohttp from source (clone and run make, see instructions here: https://docs.aiohttp.org/en/stable/contributing.html) (cf97e5b)
  • NOTE: I had some checksum integrity problems with this project's dependencies, but manually dropping the problematic checks in the npm machinery got me a working build. Not an issue for us, building a real release wheel is their problem.
• Install geoip2 from source (clone, edit setup.cfg to relax aiohttp requirement, and install) (7769666)
• Install Sopel from source (with the build-system.requires updated to setuptools>=63.0,<67) (bc688b4)

So things are a looking good for adding Python 3.12 support, we're just waiting for new upstream releases from greenlet, aiohttp, geoip2 before we're ready to do the rest of the chores.

[Exirel]

@SnoopJ this is nice! Thank you very much for that! ❤️

[dgw]

"The time has come," the walrus said, "to test on three-point-twelve!"

Translation: Python 3.12 went stable today. Blame @xnaas for reminding me, else this ticket would've stayed silent.

However, the actions/python-versions run that was supposed to generate 3.12 failed with errors, so the stable release won't be available through GHA for some unspecified amount of time until the build is fixed.

We probably need to wait on upstream dependencies to fix some number of incompatibilities with the new version anyway before we can officially declare support, anyway. Once GHA gets 3.12 up, we can look into the packages @SnoopJ investigated that were awaiting a 3.12-compatible release as of late August.

[SnoopJ]

However, the actions/python-versions run that was supposed to generate 3.12 failed with errors, so the stable release won't be available through GHA for some unspecified amount of time until the build is fixed.

If I understand right, the thing that failed was the MacOS build specifically? I'm not sure if this failure means that the action we need from GHA isn't available at all or not, that's beyond my ken.

---

I installed 3.12.0 and to my surprise, we seem to have no upstream blockage. I have no idea what changed with aiohttp because they definitely haven't done a release with the fix we previously needed, but the only thing I had to do to get the test suite (and my reference test instance) running was to adjust the setuptools build requirement. I guess maybe the source of that dependency shifted in the interim.

I'll start a branch later this week for that change, the metadata churn, and whatever needs to be done for CI, but things look pretty promising from a quick check.

[dgw]

If I understand right, the thing that failed was the MacOS build specifically? I'm not sure if this failure means that the action we need from GHA isn't available at all or not, that's beyond my ken.

The workflow graph (see below) indicates that any failing build skips testing/publishing any of the successful builds too, so 3.12 stable won't be available yet in GHA and there hasn't been another build attempt yet that I can see.

[dgw]

[ ] Check on 3.12 support concerns for official non-core plugins

Is this realistically something that even belongs in the core issue tracker?

Presumably only plugins that have dependencies beyond Sopel itself might be affected. The only one we know of for sure is sopel-iplookup, which has an install workaround with --pre until aiohttp gets together a fully 3.12-compatible release.

Everything else, if it doesn't have an 8.0-tested release yet (i.e. still requires sopel>=xx,<8), we can test on 3.12 as part of that update. That includes github (depends on bottle), and youtube (depends on google-api-python-client) still needs converting to an entrypoint package. The few "official" (or official-ish) plugins that have already been formally bumped to 8.0-compatibility (like twitter) should get sorted out through user bug reports as py3.12 adoption grows.

I guess when one of us is bored, we can pick a plugin from the org's repo list and run down a checklist ensuring things like: the plugin 1) is in entrypoint style, 2) works against latest 8.0.0.dev0, and 3) is installable/functional with both py3.8 and py3.12. A ton of the older ones still need converting to entrypoint with sopel-<pluginname> package names, and a shim dropped at the old namespace-package location on PyPI; just testing 3.12 then is the most logical approach.

[Exirel]

Nah, I don't think we should track that in core, for most plugin except the sopel-help plugin, which is kind of a big deal.

However, I'm pretty sure sopel-help shouldn't have a problem, as long as requests works fine.