Support generating non-abi3 pythonXY.dll?

[messense]

In theory we can also generate a pythonXY.dll from a .def file as long as we can obtain it somehow. Unlike stable API that has a stable_api.toml file, the obvious way to obtain a .def file for pythonXY.dll is using dumpbin.exe, for example dumpbin /exports python310.dll.

Unfortunately, there seems no easy way to differentiate between function and data in the output of dumpbin /exports, it treats them all as function.

• Collect DLLs and generate DEF file on CI
• Use DEF files to generate pythonXY.dll import library

[ravenexp]

gendef can do this in one step.

Whether we should do this for all of python37.dll, python38.dll, python39.dll, python310.dll and python311.dll is another question.

[messense]

gendef looks nice, thanks for the pointer!

Whether we should do this for all of python37.dll, python38.dll, python39.dll, python310.dll and python311.dll is another question.

Right, I wonder whether we can just generate a single pythonXY.dll for all of them, but it will cause problems if later python versions removed some APIs. The .def text file isn't very big(python311.def is about 34K uncompressed), bundling all of them isn't a big issue.

[ravenexp]

Some API are being deprecated and removed between versions, like Py_UNICODE.

I also wonder if people should build new non-abi3 binary wheels in this day and age. Most projects continue to do so mainly because of inertia and "it's always been done this way". But if they ever stop publishing new wheels they are locking their users into an ancient Python version with no obvious upgrade path.

Maybe easy cross-compilation can become one of the reasons for people to enable abi3 in the first place.

[messense]

non-abi3 binary might have access to more performant APIs/optimizations I think?

[ravenexp]

This is definitely true, but not everyone is building the next NumPy.

[messense]

My goal here is to make creating Windows wheels using maturin as plain less as possible, just like the current Unix experience (See PyO3/maturin#896). With this we can collect python sysconfigs from Windows and make maturin build -i pythonX.Y --target x86_64-pc-windows-gnu works out of the box.

[ravenexp]

Right, I wonder whether we can just generate a single pythonXY.dll for all of them, but it will cause problems if later python versions removed some APIs. The .def text file isn't very big(python311.def is about 34K uncompressed), bundling all of them isn't a big issue.

I've been thinking about this once again, and I think a single python311.def file would suffice, since as far as PyO3 is concerned,
all its Py_3x cfgs are strictly additive, so there is no way to remove extern symbols for Python 3.11 without also removing them for Python 3.7. We can safely generate python37.dll.a from python311.def and everything should just work (tm).

[davidhewitt]

all its Py_3x cfgs are strictly additive, so there is no way to remove extern symbols for Python 3.11 without also removing them for Python 3.7

I don't think the above statement is correct - you can have e.g. #[cfg(all(Py_3_8, not(Py_3_10)))] to define a symbol that only exists on Python 3.8 and 3.9.

[ravenexp]

I don't think the above statement is correct - you can have e.g. #[cfg(all(Py_3_8, not(Py_3_10)))] to define a symbol that only exists on Python 3.8 and 3.9.

Oh, indeed. Trying to unify the exported symbol definitions probably not worth it, then.

[ravenexp]

Do you plan to use it in pyo3-build-config directly or will this be a maturin-only feature?

[messense]

I plan to use in pyo3-build-config directly, maybe adding a new generate-import-lib feature to replace generate-abi3-import-lib.