Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CPython: Use shared by default on Windows #23648

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Ahajha
Copy link
Contributor

@Ahajha Ahajha commented Apr 19, 2024

Specify library name and version: cpython/all

(See python/cpython#110234 for more context, I will summarize below)

On Windows, static builds haven't worked since 3.10. Additionally, when using static builds (in the versions that support it), the extension modules aren't built, making the package less useful. Due to this, it makes sense to make shared the default on Windows, otherwise the default state would be either invalid or not very useful. (Generally, static Windows builds are for specific use cases.)

This will also fix/avoid issues like this one which arise from static being the default. I've also seen a few other PRs pop up (now that this recipe is usable) which set shared to true, so that provides some more evidence, plus some places to clean up just a little.

@conan-center-bot

This comment has been minimized.

@Ahajha
Copy link
Contributor Author

Ahajha commented Apr 19, 2024

@RubenRBS @uilianries Can an exception be made to the hook? (please see above for reasoning). If not (but you still agree), perhaps I could trick the hook by using configure()?

@AbrilRBS
Copy link
Member

@Ahajha thanks for the PR and the reasoning behind it (Along with the one in Slack!)
We'll bring this up with this the team next Tuesday, we'll let you know how that goes :)

@AbrilRBS AbrilRBS self-assigned this Apr 19, 2024
@jcar87
Copy link
Contributor

jcar87 commented Apr 19, 2024

Hi @Ahajha - thanks for this!

I'm actually surprised by this! Not so much on Windows - but I thought that a statically-linked-python-runtime-into-the-interpreter was actually widely used in some distros (Debian based) and Conda?

Historically:

  • the statically linked runtime performed better
  • loadable modules are expected to not record a dependency on libpython (that is, do not force the linker to try and load it during dlopen(), because the symbols are already provided by the executable)

I think other distros did not follow the path - apparently there's a flag you can pass at compile time to also get performance benefits for the shared one as well, see this thread: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/NWPVQSKVWDKA75PDEIJNJIFL5C5SJXB2/

Just a curiosity - if the static variant doesn't work on Windows, I would make it an invalid option in validate() only on that platform.
The problem with making the shared variant with default for all platforms is that it introduces the risk of the runtime library embedding the contents of all the other library dependencies (static) if no care is taken. Potentially leading to runtime issues loadable modules then need to load the shared variant of those shared libraries.

@Ahajha
Copy link
Contributor Author

Ahajha commented Apr 19, 2024

@jcar87 Thanks for the context, I was not aware (though also not surprised) that there's more to the story for non-Windows platforms. I'll do some reading and think about that case separately. I think I recall there being a lengthy discussion in nanobind about... something related (see here wjakob/nanobind#360 (comment), may or may not be related).

As for Windows, static builds are already invalid, but I think it provides for a poor user experience for the default state to be invalid. Every single user would have to set it to shared anyways, we might as well set that as the default.

So perhaps maybe it should only be default shared on Windows only?

@Ahajha
Copy link
Contributor Author

Ahajha commented Apr 19, 2024

@jcar correct me if I'm wrong: So from my rough understanding, the difference between shared and static is that the static package ships the binary with a statically linked libpython. For our purposes, we also need to ship the static library, since it's a "generic" package that can also be used for embedding. The shared package ships an exe linked to libpython dynamically, again with both shipped.

The part that I don't quite understand is where it would be necessary for a C extension to link to libpython, but I'm not sure if that's relevant here.

I think I'm okay with leaving non-Windows platforms as static in this case, after reading about the performance increase that comes with static.

@valgur
Copy link
Contributor

valgur commented Apr 19, 2024

As for Windows, static builds are already invalid, but I think it provides for a poor user experience for the default state to be invalid. Every single user would have to set it to shared anyways, we might as well set that as the default.

We could possibly add

def config_options(self):
    if self.settings.os == "Windows":
        self.options.shared = True

to set the default to True for Windows only.

@Ahajha
Copy link
Contributor Author

Ahajha commented Apr 19, 2024

@valgur I think that's the change I'll end up going with. I'll push that later today.

@conan-center-bot

This comment has been minimized.

@Ahajha Ahajha changed the title CPython: Use shared by default CPython: Use shared by default on Windows Apr 26, 2024
@conan-center-bot

This comment has been minimized.

@Ahajha
Copy link
Contributor Author

Ahajha commented May 21, 2024

@jcar87 Can you take another look at this one?

@conan-center-bot
Copy link
Collaborator

Conan v1 pipeline ✔️

All green in build 4 (443d8841ac3d1e1f05c0005bc18698e8bd93f329):

  • cpython/3.12.2:
    All packages built successfully! (All logs)

  • cpython/3.11.9:
    All packages built successfully! (All logs)

  • cpython/3.10.14:
    All packages built successfully! (All logs)

  • cpython/3.8.19:
    All packages built successfully! (All logs)

  • cpython/3.9.19:
    All packages built successfully! (All logs)


Conan v2 pipeline ✔️

Note: Conan v2 builds are now mandatory. Please read our discussion about it.

All green in build 4 (443d8841ac3d1e1f05c0005bc18698e8bd93f329):

  • cpython/3.10.14:
    All packages built successfully! (All logs)

  • cpython/3.12.2:
    All packages built successfully! (All logs)

  • cpython/3.11.9:
    All packages built successfully! (All logs)

  • cpython/3.8.19:
    All packages built successfully! (All logs)

  • cpython/3.9.19:
    All packages built successfully! (All logs)

@danimtb danimtb requested a review from jcar87 July 4, 2024 12:02
@AbrilRBS AbrilRBS removed their assignment Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants