Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

_cext.pyd depends on MSVCP140.dll, which is not distributed #187

Open
LordAro opened this issue Nov 20, 2024 · 13 comments
Open

_cext.pyd depends on MSVCP140.dll, which is not distributed #187

LordAro opened this issue Nov 20, 2024 · 13 comments

Comments

@LordAro
Copy link

LordAro commented Nov 20, 2024

Version: 1.4.7

_cext.pyd still depends MSVCP140.dll, even though this was supposed to have been fixed by #179 (?)

$ python3.12 -m pip download kiwisolver
Collecting kiwisolver
  Using cached kiwisolver-1.4.7-cp312-cp312-win_amd64.whl.metadata (6.4 kB)
Using cached kiwisolver-1.4.7-cp312-cp312-win_amd64.whl (55 kB)
Saved c:\foo\kiwisolver-1.4.7-cp312-cp312-win_amd64.whl
Successfully downloaded kiwisolver

$ file kiwisolver-1.4.7-cp312-cp312-win_amd64.whl
kiwisolver-1.4.7-cp312-cp312-win_amd64.whl: Zip archive data, at least v2.0 to extract, compression method=deflate

$ unzip kiwisolver-1.4.7-cp312-cp312-win_amd64.whl
Archive:  kiwisolver-1.4.7-cp312-cp312-win_amd64.whl
  inflating: kiwisolver/__init__.py
  inflating: kiwisolver/_cext.cp312-win_amd64.pyd
  inflating: kiwisolver/_cext.pyi
  inflating: kiwisolver/exceptions.py
  inflating: kiwisolver/py.typed
  inflating: kiwisolver-1.4.7.dist-info/LICENSE
  inflating: kiwisolver-1.4.7.dist-info/METADATA
  inflating: kiwisolver-1.4.7.dist-info/WHEEL
  inflating: kiwisolver-1.4.7.dist-info/top_level.txt
  inflating: kiwisolver-1.4.7.dist-info/RECORD

$ objdump -p kiwisolver/_cext.cp312-win_amd64.pyd | grep DLL
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: MSVCP140.dll
        DLL Name: python312.dll
        DLL Name: KERNEL32.dll
        DLL Name: VCRUNTIME140_1.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-stdio-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll

Found by installing Python in Windows Sandbox and trying to import matplotlib. Matplotlib (3.9.2) does not depend on these external dependencies

Matplotlib deps 
$ find matplotlib -name '*.pyd' -print -exec sh -c 'objdump -p {} | grep DLL' \;
matplotlib/backends/_backend_agg.cp312-win_amd64.pyd
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: python312.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: VCRUNTIME140_1.dll
        DLL Name: KERNEL32.dll
matplotlib/backends/_tkagg.cp312-win_amd64.pyd
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: COMCTL32.dll
        DLL Name: PSAPI.DLL
        DLL Name: python312.dll
        DLL Name: KERNEL32.dll
        DLL Name: VCRUNTIME140_1.dll
matplotlib/ft2font.cp312-win_amd64.pyd
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: api-ms-win-crt-stdio-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-locale-l1-1-0.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: python312.dll
        DLL Name: api-ms-win-crt-utility-l1-1-0.dll
        DLL Name: api-ms-win-crt-convert-l1-1-0.dll
        DLL Name: VCRUNTIME140_1.dll
        DLL Name: KERNEL32.dll
matplotlib/_c_internal_utils.cp312-win_amd64.pyd
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: ole32.dll
        DLL Name: SHELL32.dll
        DLL Name: USER32.dll
        DLL Name: python312.dll
        DLL Name: KERNEL32.dll
        DLL Name: VCRUNTIME140_1.dll
matplotlib/_image.cp312-win_amd64.pyd
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: python312.dll
        DLL Name: VCRUNTIME140_1.dll
        DLL Name: KERNEL32.dll
matplotlib/_path.cp312-win_amd64.pyd
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: python312.dll
        DLL Name: VCRUNTIME140_1.dll
        DLL Name: KERNEL32.dll
matplotlib/_qhull.cp312-win_amd64.pyd
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-stdio-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: python312.dll
        DLL Name: api-ms-win-crt-utility-l1-1-0.dll
        DLL Name: api-ms-win-crt-time-l1-1-0.dll
        DLL Name: api-ms-win-crt-convert-l1-1-0.dll
        DLL Name: VCRUNTIME140_1.dll
        DLL Name: KERNEL32.dll
matplotlib/_tri.cp312-win_amd64.pyd
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-stdio-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: api-ms-win-crt-locale-l1-1-0.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: python312.dll
        DLL Name: api-ms-win-crt-filesystem-l1-1-0.dll
        DLL Name: VCRUNTIME140_1.dll
        DLL Name: KERNEL32.dll
matplotlib/_ttconv.cp312-win_amd64.pyd
        DLL
 vma:            Hint    Time      Forward  DLL       First
        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: python312.dll
        DLL Name: api-ms-win-crt-stdio-l1-1-0.dll
        DLL Name: VCRUNTIME140_1.dll
        DLL Name: KERNEL32.dll

Apologies if this counts as a duplicate of #168 or #181 (or the others), but the messaging in those issues seemed... muddy.
@MatthieuDartiailh
Copy link
Member

Thanks for the separate ticket which makes things clearer on my end. I will do my best to investigate ASAP.

@Qix-
Copy link
Contributor

Qix- commented Nov 20, 2024

Another alternative is statically linking the runtimes. Then none of the redist DLLs would be necessary to begin with, instead of needing to package them alongside the plugin.

@MatthieuDartiailh
Copy link
Member

Python ships VCRUNTIME so there is no problem there but MSVCP is problematic.

@QuLogic I would welcome your insight since I mostly followed your suggestions here. One difference I noticed between kiwi setup and Matplotlib is that in kiwi I re-enabled FH4, do you think it may be related ?

@QuLogic
Copy link
Contributor

QuLogic commented Nov 21, 2024

I see you copied the setup from Matplotlib here:

kiwi/.github/workflows/release.yml

Lines 106 to 114 in 16735f9

# On Windows, we explicitly request MSVC compilers (as GitHub Action runners have
# MinGW on PATH that would be picked otherwise), switch to a static build for
# runtimes, but use dynamic linking for `VCRUNTIME140.dll`, `VCRUNTIME140_1.dll`,
# and the UCRT. This avoids requiring specific versions of `MSVCP140.dll`, while
# keeping shared state with the rest of the Python process/extensions.
CIBW_CONFIG_SETTINGS_WINDOWS: >-
setup-args="--vsenv"
setup-args="-Db_vscrt=mt"
setup-args="-Dcpp_link_args=['ucrt.lib','vcruntime.lib','/nodefaultlib:libucrt.lib','/nodefaultlib:libvcruntime.lib']"

These are settings for Meson(-python) but I just noticed you are using setuptools to build. Those flags should make MSVCP statically linked with Meson, but they actually do nothing with setuptools (or perhaps something else entirely, as it didn't error?) Maybe there was a warning that could be seen if the build was in verbose mode?

From setuptools docs, it looks like you could set some environment variables to override build flags. The equivalent should be something like:

CFLAGS="/MT"
CPPFLAGS="/MT"
LDFLAGS="ucrt.lib vcruntime.lib /nodefaultlib:libucrt.lib /nodefaultlib:libvcruntime.lib"

@MatthieuDartiailh
Copy link
Member

Thanks @QuLogic I missed this point. I considered switching to meson but never made the jump.

@MatthieuDartiailh
Copy link
Member

MatthieuDartiailh commented Nov 21, 2024 •
edited
Loading

I will have to try harder since my first attempt did not produce the expected result (https://github.com/nucleic/kiwi/actions/runs/11949548832).

@QuLogic
Copy link
Contributor

QuLogic commented Nov 23, 2024

Maybe try CIBW_ENVIRONMENT_WINDOWS

@MatthieuDartiailh
Copy link
Member

Thanks for the suggestion but it does not seem to do the trick... (https://github.com/nucleic/kiwi/actions/runs/11999734728/job/33447988060)
The logs looks like the env vars are set but still something is not behaving properly. I will investigate directly patching setuptools in cppy. I will have the added benefit of applying to all the projects that uses it.

@MatthieuDartiailh
Copy link
Member

Building locally after patching cppy, I obtain the following dependencies, are those fine;

        DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
        DLL Name: api-ms-win-crt-heap-l1-1-0.dll
        DLL Name: api-ms-win-crt-string-l1-1-0.dll
        DLL Name: api-ms-win-crt-stdio-l1-1-0.dll
        DLL Name: api-ms-win-crt-locale-l1-1-0.dll
        DLL Name: api-ms-win-crt-math-l1-1-0.dll
        DLL Name: VCRUNTIME140_1.dll
        DLL Name: VCRUNTIME140.dll
        DLL Name: python313.dll
        DLL Name: KERNEL32.dll

@MatthieuDartiailh
Copy link
Member

Based on the first post of this issue I think so but I would be happy if somebody can confirm.

@LordAro
Copy link
Author

LordAro commented Nov 24, 2024

Yup, vcruntime*.dll is distributed with cpython, kernel32.dll & api-*.dll are internal windows libraries, and python313 is... python :)

@QuLogic
Copy link
Contributor

QuLogic commented Nov 26, 2024

This is consistent with what we get: matplotlib/matplotlib#28687 (comment)

@MatthieuDartiailh
Copy link
Member

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@QuLogic @Qix- @LordAro @MatthieuDartiailh