[Python] Add support for Python=3.11

[valeriupredoi]

Description

This is THERE 😁

Steps I took:

• esmpy - complicated, see Rebuild for python311 conda-forge/esmpy-feedstock#57
• iris (via netCDF4 repo patch) - this will take a while to be unpatched, depends how fast the iris folk will understand and fix the issues from netCDF4 > 1.6.0
• iris-esmf-regrid (needs esmpy)

As of 10 May, 2023: we have support for Python=3.11:

• prospector still not supporting Py=3.11 - this is work in progress by me, see [Python] Add support for Python=3.11 #1832 (comment)
• this is still a valid issue, but not affecting us too much: sphinx needs to be unpinned so 5.0 gets pulled, 6+ has issues with the rtd theme, see example env file below; the issue is a pin on docutils the sphinx_rtd_theme has in their meta file, see Relax docutils pin to allow build to support sphinx>=6 and python>=3.11 conda-forge/sphinx_rtd_theme-feedstock#28 - it looks like it'll have to stay like this for a while: sphinx >=6 can not be installed because sphinx_rtd_theme are pinning docutils<0.19 which is the only practical version that supports python=3.11; see Support for docutils 0.19.x readthedocs/sphinx_rtd_theme#1323 and github.com/Relax docutils pin to allow build to support sphinx>=6 and python>=3.11 conda-forge/sphinx_rtd_theme-feedstock#28
• RTD build fails related to this New behaviour for esmfmkfile in v8.4.0 conda-forge/esmf-feedstock#91 since the RTD mechanism does not activate an env, but rather, uses the (base) env (silly!) neatly sorted out by @zklaus with a bit of magic 🪄 but the RTD interface is still obtuse, at least docs build now
• the tests fail massively due to geopy's geoextractor known HTTP request timeout; this should be fixed by Add custom pytest marker: skip extract location tests in case of RequestTimeoutError from geolocator Nominatim #2002 - @zklaus found out that Nominatim (the locator engine) should not be used in CIs - a bit of a letdown (not Klaus' work, which is great, but the robustness of Nominatim), for now I am pytest-skipping tests/integration/preprocessor/_regrid/test_extract_location.py fixed by merging main that now contains @schlunma 's mockingbird from Use mocked geopy.geocoders.Nominatim to avoid ReadTimeoutError #2005

Apart from that above, all goes spiffy!

Closes #1776 and #2001

[valeriupredoi]

an absolute fail so far, only Codacy passed 🤣 Give it another couple of months 😁

[codecov]

Codecov Report

Merging #1832 (2d63d99) into main (bd76eb1) will not change coverage.
The diff coverage is n/a.

❗ Current head 2d63d99 differs from pull request most recent head 18668e7. Consider uploading reports for the commit 18668e7 to get more accurate results

@@           Coverage Diff           @@
##             main    #1832   +/-   ##
=======================================
  Coverage   92.80%   92.80%           
=======================================
  Files         236      236           
  Lines       12446    12446           
=======================================
  Hits        11550    11550           
  Misses        896      896           

[valeriupredoi]

OK current roadblock is esmpy which is an old package (by Python age). If I recall correctly but @zklaus or @bouweandela can correct me - this is not maintained anymore, or it's been transferred someplace else?

[bouweandela]

I don't know about that, but indeed it looks like it's over a year old.

[valeriupredoi]

I don't know about that, but indeed it looks like it's over a year old.

yeah the ESMPY itself is one year old but xESMF is actually not maintained anymore (there was that guy that went under a bus), so am not sure how to ping these folks - am gonna try and open a PR in their feedstock update the build for 3.11

[valeriupredoi]

actually there is a PR awaiting for exactly that (albeit not in good shape) conda-forge/esmpy-feedstock#57 - I asked if we can get done linux64-only PR to start supporting 3.11

[valeriupredoi]

OK I've managed to get more work done on this, with the new iris 3.4.1 now (indirectly supporting 3.11, via the unpin of netCDF4), the only two issues in the environment are:

• the pin on esmpy <8.4 (I'll investigate the associated issue)
• and, as always, prospector - I might just go open a PR on prospector's feedstock

[valeriupredoi]

@bouweandela @zklaus heads up I've started working on this, to bump prospector to a modern version on conda, and hopefully get support for Py311 working there, feel free to visit conda-forge/prospector-feedstock#39

[valeriupredoi]

OK folks here's how things stand now (April 20, 2023) wrt to prospector:

• I opened a PR to bump prospector to a modern version on conda, and hopefully get support for Py311 working there, Bump to version 1.9.0 conda-forge/prospector-feedstock#39
• that PR is currently failing them tests since requirements-detector can not be installed from conda with a modern enough version and satisfying their pin for it; apart from that all else seems to be fine
• as such, I opened a PR to update the requirements-detector conda package at bump version to v1.2.1 conda-forge/requirements-detector-feedstock#18
• but that is a bit sick too since requirements-detector depends on poetry-semver which is now archived, so what I'll do I'll propose a PR that replaces it with semver, see Retire the use of poetry-semver prospector-dev/requirements-detector#43

[bouweandela]

That's turning into a nice bit of yak shaving you're doing there V! Much appreciated!

[valeriupredoi]

"yak shaving" 🤣 A fairly furry yak too, bud 😁

[bouweandela]

https://www.youtube.com/watch?v=AbSehcT19u0

[valeriupredoi]

https://www.youtube.com/watch?v=AbSehcT19u0

hahaha - excellent analogy! So this is the last missing piece, the WD40 being purchased, as per the vid conda-forge/prospector-feedstock#39 - seems the hardware store is a bit closed though 😁

[valeriupredoi]

OK - conda-forge/prospector-feedstock#39 has now been merged and that means we should be fine for Py311 here - let's see how it goes that WD40 😁

[valeriupredoi]

@bouweandela @zklaus looky-look! We have us full support for Python 3.11, gents, no corners cut, no prospectors left behind 😁 I'll now clean up this and we are ready to plug it in 🍻

[bouweandela]

Looks good to me!