Are static library builds on Windows still supported?

[FFY00]

Bug report

Bug description:

This is a followup from #110049 (comment), where @colesbury pointed out that we are assuming in a couple places that Windows builds are using a shared library.

I think this might steam from the "availability" section of the sys.dllhandle documentation only stating "Windows".

If static library builds on Windows are still supported, we should document that sys.dllhandle will only be available on shared library Windows builds.

This likely hasn't come up until now because we don't test this in the CI. AFAICT this is a "best-effort" option, where we are not really paying much attention to it, but will try to fix bugs if reported. It'd probably make sense to document that somewhere.

But with this going unnoticed for a while, it also raises the question, do we even want to support this use-case? The maintenance is minimal, so IMO it should be fine, but we definitely need to fix the sys.dllhandle documentation.

---

Possibly problematic code

ctypes

cpython/Lib/ctypes/__init__.py

Lines 469 to 474 in 8d92b6e

	if _os.name == "nt":
	pythonapi = PyDLL("python dll", None, _sys.dllhandle)
	elif _sys.platform == "cygwin":
	pythonapi = PyDLL("libpython%d.%d.dll" % _sys.version_info[:2])
	else:
	pythonapi = PyDLL(None)

On POSIX, no object name or handle is provided, because we want to get the handle for the running executable.

From dlopen's manpage

void *dlopen(const char *filename, int flags);

If filename is NULL, then the returned handle is for the main program.

On Windows, we load the dynamic library instead. I am a bit out of my depth here, but AFAICT we do this because the dynamic library symbols are not present on the current process. Again, not sure about this, but AFAICT that should not be the case when using a static libpython, so, if I understand everything correctly, we would want to take the same approach as POSIX in this situation, and from what I can tell, that can be done with GetModuleHandle(NULL).

@zooba, could check if my analysis of the issue on Windows is correct? 🤣

Tests

Failures from this wouldn't show up on CI, but should when running the tests on static library Windows builds. This is something we should probably fix, but is low priority.

cpython/Lib/test/test_site.py

Line 569 in 8d92b6e

dll_src_file = _winapi.GetModuleFileName(sys.dllhandle)

cpython/Lib/test/test_import/__init__.py

Line 742 in 8d92b6e

dllname = GetModuleFileName(sys.dllhandle)

cpython/Lib/test/support/__init__.py

Line 1552 in 8d92b6e

dll = _winapi.GetModuleFileName(sys.dllhandle)

---

Relevant information

Window static library build documentation

cpython/PCbuild/readme.txt

Lines 266 to 274 in 8d92b6e

	Static library
	--------------
	The solution has no configuration for static libraries. However it is
	easy to build a static library instead of a DLL. You simply have to set
	the "Configuration Type" to "Static Library (.lib)" and alter the
	preprocessor macro "Py_ENABLE_SHARED" to "Py_NO_ENABLE_SHARED". You may
	also have to change the "Runtime Library" from "Multi-threaded DLL
	(/MD)" to "Multi-threaded (/MT)".

---

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

[zooba]

Also relevant is the python-build-standalone project which uses the functionality (their build script). I've previously invited them to submit patches upstream for things that belong on our side, and having an easier option to enable statically linked builds would be one such thing.

Making a statically linked build and then using extension modules is probably not really viable, as Windows only does import resolution on dynamic libraries. But I believe there are uses for a static library runtime, particularly if we lighten the impact of the import system (esp configuration) and improve subinterpreters, which would make it more viable to have embedded apps with many small Python interpreters in it.

So right now, it apparently works well enough for people to use, though it's for specific cases and not a real alternative to a general purpose interpreter build. I'd like to keep it supported and try to move toward better supporting some of those cases - formally un-supporting it basically cuts those off for good.

But if we're genuinely having to twist and turn to keep support for things that are difficult to use, let's consider it. I haven't noticed anything like that though.

[FFY00]

But if we're genuinely having to twist and turn to keep support for things that are difficult to use, let's consider it. I haven't noticed anything like that though.

I agree, the thing requiring the most effort to fix right now would be the ctypes issue, but I think it's low priority. For now, we just need to fix the docs, so that we prevent more bad code to be written.

[jairov-apptio]

@zooba I think it is already possible to build Windows static library, I confirmed it using 3.11 under msys2 using --disable-shared.
But probably the next issue is there is no documented mechanism to build the builtin extensions statically linked and initialized too. They are linked against the DLL even when I use disable-shared.

[zooba]

--disable-shared only exists in the makefile, which means you are doing a POSIX build under emulation. That's fine if it works for you, but it's often not what people want - they prefer to use a native build.

Updating the native Windows build to use static libraries is likely possible, but may require significant changes. We're not opposed to having those changes, but someone has to do them and preserve the existing build process.

[jairov-apptio]

I though it was native because it is the only one documented in the readme. what is the root of the native build?
Also, I would consider your answer a confirmation that currently there is no way to build modules as static libraries too.
I will evaluate what changes are required.

[zooba]

Look in PCbuild and the dev guide for more info about the Windows native build.

"Currently there is no way" just means there isn't a simple option for it. But there's no way without making modifications to at least PC\config.c (which is not auto-generated as on POSIX), and life is probably going to be simpler modifying the other build files (primarily pythoncore.vcxproj to include extra source files and pcbuild.proj to omit extension module projects), even though you could probably craft command lines to avoid changing them.

[melroyvandenberg]

Are static Linux builds still supported?

There are multiple questions about that.

I tried for now configure options / compile flags:

./configure --enable-optimizations --with-ensurepip=install prefix="$rollout" \
    --disable-shared LDFLAGS="-static -static-libgcc" CFLAGS="-static" CPPFLAGS="-static" CFLAGSFORSHARED=" " CCSHARED=""

Set static to the Setup.local configuration modules file (it's empty by default, so *static* will be added correctly and this file be read first, before the other Setup files):

# Set static to Python setup modules
echo "*static*" >> Modules/Setup.local

And make via: make LINKFORSHARED=" " -j 10

Then the build seems to start fine, until...
It fails with 😢 :

gcc -pthread -shared -static -static-libgcc  -fno-semantic-interposition -fprofile-generate  Modules/arraymodule.o   -o Modules/array.cpython-312-x86_64-linux-gnu.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/usr/bin/ld: Modules/arraymodule.o: relocation R_X86_64_PC32 against undefined symbol `PyExc_ValueError' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status

For some reason I can't get rid of the -shared flag... Why is the shared build in general so poorly documented?

Could somebody improve this wiki page: https://wiki.python.org/moin/BuildStatically

ps. @zooba The project you pointed to (python-build-standalone), does seems to build Linux shared libs using -fPIC and --enable-shared flags. For example: https://github.com/indygreg/python-build-standalone/actions/runs/7843195688/job/21403138066#step:7:44766

These "standalone" builds are not all static builds. I'm unable to find the static build to be honest... Could just be me.

[zooba]

@melroyvandenberg Linux static builds are off-topic for this issue, and you'll likely get a better response to your question on https://discuss.python.org/ rather than tacking it on here.

Though since you've dropped in, could you enlarge a bit on your scenario? I'm being led to understand that embedders prefer to reference the distro-installed libpython3 and use whatever is found there - why is your scenario different from that?

[melroyvandenberg]

@melroyvandenberg Linux static builds are off-topic for this issue, and you'll likely get a better response to your question on https://discuss.python.org/ rather than tacking it on here.

Though since you've dropped in, could you enlarge a bit on your scenario? I'm being led to understand that embedders prefer to reference the distro-installed libpython3 and use whatever is found there - why is your scenario different from that?

Yea sorry I was aware that Linux static build was off-topic. 😊 I think the Python forum might be a better place to discuss these issues indeed.. That being said, my scenario is quite simple: I want to have a stand-alone Python binary build from source, which I can deploy to various Linux environments, thus it shouldn't depend on the local glibc/shared .so libraries of the distro. Hence the reason for a static build.

The -shared flag keeps present:

gcc -pthread -shared -static -static-libgcc  -fno-semantic-interposition -fprofile-generate  Modules/xxlimited.o   -o Modules/xxlimited.cpython-312-x86_64-linux-gnu.so

Or during test module build:

gcc -pthread -shared -static -static-libgcc build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.8/Modules/_bisectmodule.o -L/sdev_shared/sdev_custom_scripts_tst/lib -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -o build/lib.linux-x86_64-3.11/_bisect.cpython-311-x86_64-linux-gnu.so

Despite I use:

export LINKFORSHARED=" "

./configure \
    --enable-optimizations \
    --with-ensurepip=install \
    --prefix="$rollout" \
    --disable-shared \
    --with-static-libpython \
    LDFLAGS="-static -static-libgcc" \
    CFLAGS="-static" \
    CPPFLAGS="-static" \
    CFLAGSFORSHARED="" CCSHARED="" LDSHARED="" LDCXXSHARED="" \
    MODULE_BUILDTYPE=static

It seems newer Python versions (eg. 3.12.x) seems to be effected more regarding static Linux builds. As if the developers of Python do not verify for static builds. Causing more regression over time.