Remove pyconfig.h header parsing

[ravenexp]

The config header parsing code was supposed to be only invoked when
cross-compiling for Windows, but in reality it fails to correctly parse
the config header files shipped with the upstream Python for Windows.

Given that there are now better options for reliable cross-compiling
for Windows such as PYO3_CROSS_PYTHON_VERSION or the abi3-py3* features,
it should be OK to remove this config for v0.14.

Update the cross-compilation instructions section of the user guide.

Fixes #1337

[davidhewitt]

Thanks, yes this is probably fine to go, however it does lead to a possible situation where a user trying to cross-compile for a very unusual windows build might not be able to configure pyo3 as they need.

I've been conisidering a PR which enables users to provide exact configuration externally (maybe as JSON), but didn't finish it yet. We might want to consider adding that before merging this, just incase anyone needs the flexibility.

[ravenexp]

I agree that a configuration tool which is independent of the target Python installation would be nice to have,
but unfortunately pyconfig.h parsing as it presently exists is of little help here.

Mainly because it is only capable of parsing autoconf/autoheader generated headers, but the Windows Python port does not use these tools. It instead supplies a hand-written config header containing multiple #ifdef clauses and whatnot, which the current PyO3 code can't possibly make sense of.

One such example is the Py_ENABLE_SHARED definition.
#1359 makes the problem even worse by assuming Py_ENABLE_SHARED is not set when the parser fails, which is simply wrong when targeting Windows.

In the case someone customizes their Python build on Windows, they are supposed to define some magic preprocessor macros like Py_NO_ENABLE_SHARED, which in turn trigger the #ifdef clauses inside pyconfig.h. But then this decision is not represented in the config header file contents at all. As such, Py_NO_ENABLE_SHARED is only mentioned in the comments.

[kngwyu]

I think it's OK to remove this if we can prepare a more flexible solution until the next release.

I've been conisidering a PR which enables users to provide exact configuration externally (maybe as JSON),

Maybe [package.metadata] is better?

[ravenexp]

Maybe [package.metadata] is better?

This must be a per-target setting, and the target triple is supplied externally via --target or CARGO_BUILD_TARGET.
Environment variables are currently being used for this kind of configuration.

Can we tolerate another 3-4 PYO3_CROSS_* variables?
Using JSON will likely introduce a build dependency on serde and serde_json, which take quite a long time to compile.

[davidhewitt]

Yes, if it were an JSON file it would be an environment variable specifying where to find the JSON.

Using JSON will likely introduce a build dependency on serde and serde_json, which take quite a long time to compile.

I was thinking I would make this "advanced" build configuration opt-in via a feature :)

---

Regarding this PR specifically, I'm now convinced that leaving the header parsing in causes more headaches than it solves. Users who really need advanced configuration can always patch their local PyO3 temporarily!

[kngwyu]

I merged main to fix CI.
BTW, we can use nanoserde as a lightweight (but limited) alternative of serde.

[ravenexp]

I've looked through the Windows pyconfig.h contents once more, and I'm now fairly convinced that there are only two switches the downstream builders can flip without modifying the vanilla CPython sources:

• define Py_NO_ENABLE_SHARED which maps to !Py_SHARED in PyO3
• define _DEBUG which maps to Py_DEBUG in PyO3

Defining _DEBUG does not enable Py_TRACE_REFS though, so the debug Python builds should remain ABI-compatible, at least for version 3.7+.

Disabling WITH_THREAD is not supported without modifying the headers manually, this fact is also mentioned here

pyo3/build.rs

Line 296 in 3bc5caa

// WITH_THREAD is always on for Python 3.7, and for PyPy.

As a follow-up, I propose adding PYO3_CROSS_NO_ENABLE_SHARED environment variable, which can enable users to statically link the Python interpreter library into their Rust programs for Windows. It doesn't make sense for the extension modules though, and should be ignored with a warning when "extension-module" feature is enabled.
I'll try to implement it myself and submit a new PR in the nearest future.

IMO, as far as cross-compiling for Windows is concerned, the JSON configuration feature may be an overkill.

[davidhewitt]

As a follow-up, I propose adding PYO3_CROSS_NO_ENABLE_SHARED environment variable, which can enable users to statically link the Python interpreter library into their Rust programs for Windows. It doesn't make sense for the extension modules though, and should be ignored with a warning when "extension-module" feature is enabled.
I'll try to implement it myself and submit a new PR in the nearest future.

Note that the last time I experimented with this, I concluded linking the Python interpreter statically doesn't work very well (probably not until we have some upstream support). When linking binaries and tests statically Rust seems to trim out the symbols that aren't used directly by the program, so when other Python extension modules are imported by the statically-linked binary, often there are missing symbol errors. E.g. #742

So it may not be worth your effort implementing that variable. Thanks for looking though!

[ravenexp]

I was thinking about programs embedding a Python interpreter for application-level scripting, just like Lua and Tcl interpreters are commonly used. Loading Python extension modules is not a requirement in this case, because it would allow to escape the user script sandboxing...

[davidhewitt]

Mmm. This would include parts of the Python standard library which are implemented as C extension modules!

[ravenexp]

As a side note, apparently it is possible to link in the standard library modules statically, as described here:

https://wiki.python.org/moin/BuildStatically