build.rs fails to parse hand-written pyconfig.h headers installed by python for windows (MSVC-compiled)

[ravenexp]

🐛 Bug Reports

When trying to cross-compile a python extension for windows on linux using the mingw target I get the following error:

$ cargo build --target x86_64-pc-windows-gnu

error: failed to run custom build command for `pyo3 v0.13.0`

Caused by:
  process didn't exit successfully: `./target/debug/build/pyo3-7cf3ac8bb8af293b/build-script-build` (exit code: 1)
  --- stderr
  Error: "Py_ENABLE_SHARED is not defined"

pyconfig.h from python 3.9.1 for windows (built with the MSVC compiler), which I have installed, contains the following lines:

/* For Windows the Python core is in a DLL by default.  Test
Py_NO_ENABLE_SHARED to find out.  Also support MS_NO_COREDLL for b/w compat */
#if !defined(MS_NO_COREDLL) && !defined(Py_NO_ENABLE_SHARED)
#       define Py_ENABLE_SHARED 1 /* standard symbol for shared library */
#       define MS_COREDLL       /* deprecated old symbol */
#endif /* !MS_NO_COREDLL && ... */

When I manually add the line

#define Py_ENABLE_SHARED 1

to pyconfig.h the error disappears.

I believe the following code fails to parse C headers with white space between # and define,
as well as with the comments:

https://github.com/PyO3/pyo3/blob/master/build.rs#L224

The C header parsing code should somehow be restricted to generated pyconfig.h headers only.
Manually written pyconfig.h from python for MSVC does not work.

🌍 Environment

Arch Linux
python 3.9.1
mingw-w64-python39-bin 3.9.1 <- this is the MSVC-compiled python, not the MinGW-compiled python
rustc 1.48.0
pyo3 0.13.0

[davidhewitt]

Hi @ravenexp thanks for the report. Is there a _sysconfigdata.py file (might have a slightly different name) anywhere in your mingw lib directory? Perhaps we can parse this; it's usually more reliable than header parsing.

(See find_sysconfigdata function in build.rs for more detail.)

[ravenexp]

No, there isn't. It could be that the ArchLinux package author has chosen not to install it, though.

BTW, what kind of configuration data pyo3 needs to build an extension module for Windows?
I'm asking because by applying the above workaround, as well as some DLL import library symlinking hacks,
I've successfully compiled an extension module for Windows python that loads and works under WINE.

Perhaps it is possible to provide the same information via environment variables and avoid looking at the cross python installation files at all (except for the python DLL import library)?

[kngwyu]

Hi @ravenexp, thank you for reporting.

what kind of configuration data pyo3 needs to build an extension module for Windows?

See https://github.com/PyO3/pyo3/blob/v0.13.0/build.rs#L198.
We need eight variables. It would be good to allow some manual configuration(json or Python dict?) instead of pyconfig.h.

[ravenexp]

Thank you for your prompt responses.

I've stumbled upon this function:

pyo3/build.rs

Line 507 in 26ca1f5

fn get_config_vars_windows(_: &Path) -> Result<HashMap<String, String>> {

and "PC\pyconfig.h" is exactly the header I've quoted before. As I've said, the arch mingw python package installs MSVC headers.
These hardcoded config vars would probably have worked for my use case.

But is it really feasible to build a working python 3.5+ for windows without wide Unicode, dynamic linking and threading support?
AFAIK on windows the extension modules have to link to python3y.dll or python3.dll (ABI3 modules) because the source DLL has to be defined explicitly for every imported symbol.
And dynamic linking on windows pretty much implies multi-threading, at least when using MSVC.

[ravenexp]

I did some research on the impact of the build config variables on the extension module ABI.

The ./configure flag --with-wide-unicode has been removed in Python 3.3.
It is no longer possible to select the Py_UNICODE representation at the compile time.
I think it becomes entirely platform-dependent.
Also, Py_UNICODE typedef is not a part of the stable ABI.

Py_DEBUG is not compatible with the stable ABI.

What's left is Py_ENABLE_SHARED and WITH_THREAD, which I believe can be assumed to be set on Windows.

[davidhewitt]

Thanks, I've opened #1350 based on that research! I think it might even be reasonable to remove the header parsing entirely, though that's a breaking change so will wait until 0.14.

[ravenexp]

I have retried building my extension module with pyo3 git master after #1350 was merged.

When PYO3_CROSS_INCLUDE_DIR is not set, I'm greeted with the following new error message:

Error: "One of PYO3_CROSS_INCLUDE_DIR, PYO3_CROSS_PYTHON_VERSION, or an abi3-py3* feature must be specified when cross-compiling for Windows."

When I set the PYO3_CROSS_INCLUDE_DIR to point to the windows python headers, I get the same old:

Error: "Py_ENABLE_SHARED is not defined"

but, when I set either PYO3_CROSS_PYTHON_VERSION, or an abi3-py3* feature everything works as expected.

My cross-compilation problem is essentially solved by #1350, but the reported windows pyconfig.h parsing bug is still there.
Also the first error message the user encounters when trying to cross-compile for windows is very misleading, as the very first suggested solution fails with a cryptic error message.

[davidhewitt]

Thanks for following up @ravenexp .

I agree now I think about it that the error messages are still not great. I am aware the original INCLUDE_DIR issue still remains.

I think that as the other options now work well, I'd be keen to remove the header parsing entirely in pyO3 0.14. This'll "fix" the bug and I'll improve the help messages at the same time.