Build wheels for Python 3.12

[phillipuniverse]

Long story short

Publishing CPython 3.12 wheels to PyPI would help other projects that use multidict and are working on their own CPython 3.12 support and distribution.

Also this issue is effectively copied from #778

Expected behaviour

pip is able to install multidict wheels from PyPI with CPython 3.12.

Actual behaviour

The sdist is downloaded instead such that building is required.

Steps to reproduce

Look on PyPI and note there are no cp312 wheels available.

Your environment

CPython 3.12

[Hrxn]

Yes, this is also a requirement of sigstore, and is the reason why installing sigstore currently fails on Windows for example, with a fresh installation of Python 3.12 - so the Python 3.12 installation package cannot be verified with sigstore at the moment.

[Hrxn]

Any ETA for this?

[pythonweb2]

The blackd server is dependent on this project and cannot be built on windows. It would be great if python 3.12 wheels could be hosted on PyPI soon.

[kraktus]

It seems that 3.12 is not supported yet: #877, not just that the wheels have been forgotten.

[fenchu]

Please create wheels. on windows the workaround is to install the 6.72GB visual studio build tools for c++.

[webknjaz]

I'll get to it once frozenlist is ready.

[fenchu]

Since I have tons of windows and really hate to add the visual studio bloat, I added the whee here:
pip install https://github.com/fenchu/python312-wheels/raw/main/multidict/multidict-6.0.4-cp312-cp312-win_amd64.whl

[leirn]

I'll get to it once frozenlist is ready.

It seems the PR for python 3.12 on frozenlist had been merged now. Multidict is the new blocking point for many packages for python 3.12

[stonebig]

All necessary pull requests are there. anaconda did workaround temporarly to not be stopped. At same time I see workloads in the cloud in Python-3.8, on premise load on python-3.11, and the python-3.13 twisters of multi-interpreters, jit, and free threads are coming... how can our benevolant giants withstand this ?

[Jamim]

I'll get to it once frozenlist is ready.

Hello @webknjaz,

Given frozenlist is ready, would you mind taking care of the related PRs?

• Adapt the CPython 3.9+ C-API usage to use newer macros #864
• Add support for Python 3.12 #877

Thanks in advance!

[marhoy]

I'm also waiting on wheels for Python 3.12, as LangChain depends on multidict (and e.g. for docker images, I don't want to add compilers etc.).

[a5r0n]

as the wheels building run only for new tags, and 3.12 released after the latest tag, we need to wait for new multidict release..

[vrdn-23]

Can we get an estimate on when the next release is going to be cut?

[webknjaz]

I'm still doing housekeeping, cleaning PRs/issues, packaging and CI infra. I've spent a lot of time on this already but it's not enough. I'll make a release when I'll feel like the repo is in a good shape. I'm planning to get a slightly changed variant of #909 in (done!), before that and get rid of the import loops that caused some tests to be skipped as they still seem to cause side effects.
If anybody wants to help — send PRs cleaning the ignored rules/error codes from .mypy.ini, one per file, could be partial (with only some errors fixed and the rest postponed).
I want to at least have 100% coverage for the tests/ directory — previously, it wasn't measured which caused us to miss that there was an import loop resulting in the C-extension tests being skipped. Currently, only gen_pickles.py isn't covered. It'd be nice to integrate/move it into the pytest infra as a session-scoped fixture.
Here's an issue documenting this task: #923.
Another bit that will ensure we don't have loops (I think we still have one, FTR) is adding a test — https://github.com/aio-libs/aiohttp/blob/master/tests/test_circular_imports.py needs to be copied over and adapted a bit for this repo.

[webknjaz]

Here, I filed an issue on improving coverage #921 — it's a chance to make a series of easy contributions, if anybody's up to it. It can also be worked on by multiple people simultaneously.

And a bit more advanced contribution: #922 (pytest experience would be useful for this one).

[elia-bracci-hs]

Sounds good. When is planned the next release including your changes?

[jonaslb]

Fixing long-standing bugs, getting 100% test coverage, improving the testing and CI setup - these are all obviously good things, but they don't really seem like actual blockers for a 3.12-compatible release..? I think I can say for all of us who are a little impatient to use 3.12 in practice: A multidict release which is "equally buggy and untested" as the previous 3.11-releases would be perfectly fine as step 1!

[kraktus]

Tying those goals to 3.12 release is also a way to encourage external contributions (which is fair imho).

[martin-traverse]

FYI - this is a dependency for some of the core Azure libraries, so it is blocking the ability to use Python 3.12 with Azure. I agree that tech debt, build robustness etc. are super important. Perhaps there are some pragmatic choices around what improvements can be made reasonably quickly before releasing 3.12 support and what might be more appropriate to resolve as tech debt tickets over the next few months?

[matrey]

FYI pip install --only-binary :all: aiohttp on Python 3.12 ends up "silently" installing aiohttp version 0.13.1 from 10 years ago (the last one without dependency on multidict I guess). Quite a nasty surprise when upgrading to Python 3.12

[BjoernPetersen]

@martin-traverse If a giant like Azure is depending on this project for their commercial product, it might be fair for them to spend some resources on maintaining it.

@matrey I agree that the behavior you mentioned could be a nasty surprise. Another possible surprise would be to accidentally install a new version of aiohttp that you haven't integration tested, breaking your app as well. That's why I'd strongly recommend specifying a version (or a version range) for each of your direct dependencies.

[astrojuanlu]

Both issues mentioned in #887 (comment) have open PRs with feedback from @webknjaz that hasn't been addressed by their respective authors.

Maybe stop pushing the maintainers for shipping a convenience wheel and spend that energy helping finishing those two PRs?

[webknjaz]

for each of your direct dependencies.

..as well as transitive: https://github.com/jazzband/pip-tools

[webknjaz]

So.. I wanted to take a moment to respond to some comments since my last update. Here we go.

Fixing long-standing bugs, getting 100% test coverage, improving the testing and CI setup - these are all obviously good things, but they don't really seem like actual blockers for a 3.12-compatible release..?

@jonaslb The segmentation faults you found seem like deal breakers to me. It's more or less the main blocker.

I think I can say for all of us who are a little impatient to use 3.12 in practice: A multidict release which is "equally buggy and untested" as the previous 3.11-releases would be perfectly fine as step 1!

Sure, an additional maintainer might prioritize things to their liking. I don't get to work on this very often and so I want a few things fixed. The test coverage is not a blocker but a nice to have while waiting for the blockers to be addressed. I was hoping to maybe hit more segfaults revealed while covering the Python and C modules.

I revamped the packaging setup so there's no "equally" in the case of the next release compared to the previous one, honestly. I thought I completed repackaging but apparently, the changes are still uncommitted locally and incomplete — I won't block the release on this, then. I'd rather prefer to have less surprises down the pipe and if there's known fixable problems, that reduces the probability of combined bugs popping up that need hotfixing.

One thing that is important to me is reducing the maintenance burden and pressure related to it. I don't like being blamed for things, you see.. So I'll cut the release as soon as I feel comfortable with what we have here. I understand the frustration but that's the price for having somebody other than you maintain a thing you use.

That said, @jonaslb I'm grateful to you personally (and many others) for helping out with the analysis of #926. And I'm willing to let more people do release management here if that's something they're interested in.

Maybe stop pushing the maintainers for shipping a convenience wheel and spend that energy helping finishing those two PRs?

@astrojuanlu thank you for being so supportive! While those two issues are not ultimate blockers, they are indeed important. It looks like the most important patch is going to be merged soon.

FYI - this is a dependency for some of the core Azure libraries, so it is blocking the ability to use Python 3.12 with Azure. I agree that tech debt, build robustness etc. are super important. Perhaps there are some pragmatic choices around what improvements can be made reasonably quickly before releasing 3.12 support and what might be more appropriate to resolve as tech debt tickets over the next few months?

@martin-traverse I suppose a pragmatic choice is to use redistributors that provide the kind of support companies like the one behind Azure value so much. There's entire ecosystems of curated software like RHEL, that maintain patches LTS-style, for example. I'm but a little human and have my vision and priorities 🤷‍♂️
If you're seeking somebody to pressure, may I offer M$? It would be amazing if they started caring about their own dependencies, for once. And it would be a great service to the FOSS community if they were to really contribute to maintenance and reducing its burdens.
By the way, installing from Git (while pinning to a commit SHA, of course) is a quite realistic solution. It is also rather pragmatic.

Tying those goals to 3.12 release is also a way to encourage external contributions (which is fair imho).

You're right, @kraktus. This is fair. Thanks for pointing this out!
Although, there's no conspiracy here — I don't seek to blackmail the consumers my free labor (that's mostly happening in the middle of the night instead of me getting enough sleep), it's nice to see several individuals contributing back and I'm thankful to them.
I was mostly trying to communicate what's happening.
Hopefully, I'll be able to attempt triggering the first release build process before the sun rises. If not — that'll be a plan for another night.

[martin-traverse]

Thank you for your comments and discussion on this issue. I can certainly appreciate the position you are in as a private individual trying to look after a project and balance against work / life commitments. I think the point that major commercial platforms have these kind of dependencies way down the stack is a big issue with several implications and certainly this is not the only place it is happening! I do think there is a general discussion to be had about how commercial support can be provided to these kind of projects, particularly in cases where they are critical, widely used components near the bottom of the stack and no support structure has formed organically.

I have raised issues against the Azure Python SDK here and the Azure fsspec implementation here. Perhaps this will help get some attention.

As an FYI, I believe the Google cloud SDK also has this dependency, although presently they still have some of their own dependencies with broken builds for Python 3.12 which are masking the issue. AWS are ahead of the game, their boto3 library has a much smaller dependency tree, demonstrating the up-side of minimising dependencies where possible!

[webknjaz]

@martin-traverse thank you! I'm personally not blocked by this missing release, just trying to be helpful since some people here were asking nicely. Otherwise, I'd be doing doing the updates I wanted slower, as an opportunity presents itself. These days, my focus is on simplifying and unifying processes and workflow configurations in a way that would enable me and other people spend less time on the boring stuff. I have a maintainer (ish) status in a lot of projects and looping over them always takes a minute.

Regarding the AWS, I happened to randomly meet one of their dev tools engineers at PyCon last year. Apparently, they even vendor stuff, sometimes stripped down. It was interesting to discuss how and why they do this, and what the drawbacks might be. It sounds like the ecosystem is moving towards not being tied to the upstream software repos for apps that aren't libs. That's why their deps are smaller — it's also easier to audit and be in control, in their use case.

Release update

I triggered the workflow and it failed because of the release automation problem (I didn't complete synching many packaging-related things with yarl/frozenlist and must've forgotten at least one place). The next step will be to fix that and re-attempt.

[webknjaz]

v6.0.5 is out: https://github.com/aio-libs/multidict/releases/tag/v6.0.5 / https://pypi.org/project/multidict/6.0.5/ / https://github.com/aio-libs/multidict/actions/runs/7746701945. Sigstore signing failed, it probably needs updating in CI. Hopefully, updating it will fix stuff for the next release.

[martin-traverse]

Thank you, this has unblocked using Python 3.12 on Azure for my project.