test_embed fails if Python is built with LTO and LLVM clang on macOS #110313
Labels
type-bug
An unexpected behavior, bug, or error
Comments
|
|
Building Python 3.12.0 on the same system with the same toolchain results in the following: I assume that this is not a fatal error? |
|
We are running into this bug on the CPython benchmarking infrastructure, too. |
|
Does test_decimal fail if you run it alone? How did you built Python? |
|
Here is the result of running |
|
And |
To be clear: this is top of tree |
|
To answer the question about how I am building Python: |
|
It is not clear why |
|
Something must have changed in the runtime module loading in |
|
It's syntax that Python cannot locate the unicodedata shared library. Can you import it if you run Python manually? Was it built? |
|
Can the issue be reproduced on other platforms than macOS? |
|
Perhaps @mdboom can comment about what platform he is using in his comment above. |
|
I see now that Under Under |
|
I don't understand this test_embed error neither: In Log: |
|
|
|
I think it is a problem with embedding on my platform in |
|
I suspect that the problem occurs when Python code being executed in an embedded environment tries to import a runtime loaded module. That is why much of |
In the Python API, the variable is called |
Do you reproduce the issue with a more classic command?
then:
|
That is because macOS uses the traditional C symbol naming, which places an underscore before C symbols. |
|
Aha, I can reproduce some issues on Linux: 8 tests failed with "env changed". Example: Each subprocess spawned by tests create a |
|
Okay, here is what I have determined so far: |
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Oct 10, 2023
|
Tests spawning subprocesses create many PR #110654 avoids the warning, but |
|
I don't think we have established that LLVM 17 is the problem. To do that we would need to try building with LLVM 16. |
|
Building with PGO and LTO are fine now. |
|
Test on Fedora Rawhide with LLVM clang 17: It works as expected. By the way, PGO+LTO build is tested on Fedora Rawhide on multiple architectures:
|
Would you mind to elaborate "now"? Before you wrote that it didn't work. What changed? Are you still testing the latest main branch? You're doing tests on macOS 10.12.6 with Clang 17.0.1, right? |
vstinner
changed the title
I added "on macOS" to the issue title :-) |
Sorry, with 3e3a7da reverted. |
If you can run your test with LLVM 16 on macOS, that would be nice yes, since I fail to reproduce the issue on Linux with LLVM 17. |
Yes, I will try that, but it will have to be later. It is past dinnertime now. |
|
The issue exists when building with LLVM 16, and backing out commit 3e3a7da resolves it. |
@debohman Are you using a custom compiler toolchain rather than using the Apple basic toolchain? |
I am okay with reverting the PR, IIUC it was just intended to speed up the build, but I need to know what effect will occur if drop the "NOLTO" idea. |
|
If we drop the "NOLTO" idea, it will occur #96761 again. Please let me know If I misunderstand. |
|
Or do you intend to revert #29859 too? |
The stock linker is being used, the rest of the toolchain is the newer llvm / clang. |
It's macOS 13.6 I'm building CPython as follows: |
|
I can confirm that backing out 3e3a7da also resolves things for me. |
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Oct 11, 2023
…e and _testembed (python#109581)" This reverts commit 3e3a7da.
|
I wrote PR #110720 to revert my "NoLTO" change which introduced the bug. I don't understand exactly why/how Programs/_testembed is miscompiled on macOS using LLVM 16 or 17, but it's bad: we should not miscompile Python. My change was just to optimize "build Python" command faster (run It would be nice if someone can dig into the issue, but I don't have the bandwidth for that. I cannot reproduce the issue on Linux, and I don't have access to any macOS machine. |
debohman
changed the title
|
If I may hazard a guess as to why this is failing, we seem to be using objects built with LTO and linking them to create In the case which fails, the generated |
|
I suppose that the best way to build with LTO only once instead of 3 times (freeze, testembed, libpython) is to use If I recall correctly, GCC implementation of LTO is very different and so the NOLTO hack to reduce build time when enable-shared is not used still works. Maybe we just use -flto=full in NOLTO on macOS (when cland is used). |
|
Shall we close this issue? I have successfully built |
|
I am also using the latest Clang |
|
Ok, I close the issue. |
Glyphack
pushed a commit
to Glyphack/cpython
that referenced
this issue
Sep 2, 2024
…e and _teste… (python#110720) pythongh-110313: Revert "pythongh-90108: Disable LTO on _freeze_module and _testembed (python#109581)" This reverts commit 3e3a7da.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug report
Bug description:
This is on macOS 10.12.6, building with LLVM / Clang 17.0.1. The system otherwise has Python 3.11.6 installed.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Linked PRs
The text was updated successfully, but these errors were encountered: