Error displayed when running python if pywin32 is installed in pdm env (pep582+install_cache true)

[hcoohb]

• I have searched the issue tracker and believe that this is not a duplicate.

An error message is displayed before running any python program if pywin32 has been installed in a pdm env with pep582 and feature.install_cache = True.
There is no issue detected if pywin32 is installed and feature.install_cache = False.

Steps to reproduce

• pdm installed on windows with pep582 enabled.
• pdm config feature.install_cache on
• Initialize pdm package pdm init and pdm add pycowsay
• pdm add pywin32 (everything looks good when using -v)
• run a python script using this pdm env: python test.py (or even a pdm action such as pdm list)

Actual behavior

Before the python command is run, errors around pywin32 are being printed in terminal:

Error processing line 7 of C:\Users\Me\Devel\test\__pypackages__\3.8\lib\pywin32.pth:

  Traceback (most recent call last):
    File "C:\Users\Me\AppData\Local\Programs\Python\Python38\lib\site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
  ModuleNotFoundError: No module named 'pywin32_bootstrap'

Remainder of file ignored

As a reference, the file C:\Users\Me\Devel\test\__pypackages__\3.8\lib\pywin32.pth is:

# .pth file for the PyWin32 extensions
win32
win32\lib
Pythonwin
# And some hackery to deal with environments where the post_install script
# isn't run.
import pywin32_bootstrap

So it seems it is the line import pywin32_bootstrap that is creating the issue.

Expected behavior

No error output being displayed for every python run when pywin32 package is installed in the pdm env.

Environment Information

pdm info:
PDM version:        1.12.6
Python Interpreter: C:/Users/Me/AppData/Local/Programs/Python/Python38/python.EXE (3.8)
Project Root:       C:/Users/Me/Devel/test
Project Packages:   C:\Users\Me\Devel\test\__pypackages__\3.8

pdm info --env:
{
  "implementation_name": "cpython",
  "implementation_version": "3.8.10",
  "os_name": "nt",
  "platform_machine": "AMD64",
  "platform_release": "10",
  "platform_system": "Windows",
  "platform_version": "10.0.19042",
  "python_full_version": "3.8.10",
  "platform_python_implementation": "CPython",
  "python_version": "3.8",
  "sys_platform": "win32"
}

[frostming]

pywin32 may not be able to support install cache, turn it off to make it work.

[frostming]

install_cache is an opt-in feature that does not work for all cases, closing it now.

[mro-rhansen2]

install_cache is an opt-in feature that do not work for all cases, closing it now.

Can you explain how you came to this determination? I spent a great deal of effort today trying to get to the bottom of this only to realize that it is fundamentally related to another issue that was reported and promptly closed. I see absolutely no evidence that this was ever related to archaisms in pywin32 that would make it incompatible with the install cache.

#1349

What I found is that all of these errors go away if you simply install pywin32 into the venv prior to executing the pdm install. I was able to accomplish this by first removing the default venv using pdm venv remove in-project -y and then setting up a new one with pdm venv create --with-pip. From there I was able to perform pdm run pip install pywin32. Afterwards I was able to perform pdm install without any issues when using the install cache as well as without it. The real kicker here is that the errors are all just noise - everything installs properly even if I don't perform these steps. It simply doesn't instill a sense of confidence in the installation when the package manager is throwing a fit and then telling us that everything is fine.

The fact of the matter is that the default venv is barren and is not configured to support the documentation for this project. It is obvious that things must be done to manually configure the venv in spite of the marketing material that I've seen so far. It seems like the tooling could easily be extended to address these issues (ex: install pywin32 in the venv on Windows machines, install setuptools in the venv if the project currently has a setup.py, etc...). I'd be happy to open another issue to discuss further, but it seems like these things are shut down rather quickly so I didn't see a reason in polluting the board with issues that will seemingly be dismissed without investigating the complaint.

[frostming]

Can you explain how you came to this determination?

If you dig deep into how pywin32 is imported(it has many custom import hooks), you will know why I said that. But if you wouldn't like to or haven't got a clue that's okay and just accept the conclusion that pywin32 doesn't work with PDM's install cache mechanism.

It is rather normal because the maintainers of pywin32 may have never heard of PDM and never expected it to be installed in a centralized cache and couldn't have known how that works. Does it make some sense now? @mro-rhansen2

It simply doesn't instill a sense of confidence in the installation when the package manager is throwing a fit and then telling us that everything is fine.

Yes, those import hooks are executed in runtime and an installer has no means to know it will fail. You wouldn't expect an installer to suddenly run your code and cause any possible side-effects

The fact of the matter is that the default venv is barren and is not configured to support the documentation for this project.

Thanks, the docs are updated to mention that

I'd be happy to open another issue to discuss further, but it seems like these things are shut down rather quickly so I didn't see a reason in polluting the board with issues that will seemingly be dismissed without investigating the complaint.

All those are closed with an explanation or a workaround. I am maintaining this project on my own in my spare time, together with a discord and other discussion channels. Lots of people said I am very responsive as a maintainer. Few are helping with triaging issues so I have to keep the number of open issues at a low level to not burn out myself.

[mro-rhansen2]

I'm really not following you on the issues with pywin32 and the pdm installer. The only time I saw this problem was when I switched from PEP 582 over to letting pdm create a venv. I was using the install cache the entire time. I went back and tried again without the install cache after I found this issue but ended up with the same result.

I just went back and retested my scenarios to make sure that I didn't misrepresent my findings (because I spent many hours digging into this on Friday and could have easily mixed up the results in my head). The only time I see a problem is when I am using the pdm venv. That simply does not work regardless of whether or not the install cache is enabled. The only feasible workaround as far as I can tell would be to install pywin32 manually into the pdm venv.

Everything works just fine when using PEP 582. I originally thought that was working because I have pywin32 installed directly into the particular pyenv environment that I was basing from. However, I just want back and tried again on a fresh pyenv environment and pdm successfully installs pywin32 even with the installation cache enabled whenever I use the PEP 582 featureset.

I get that you've crawled through the pywin32 source and feel like there are things happening there that would prevent it from ever working with the pdm installation cache but I can assure you that practical application tells me otherwise. Everything that I have seen so far points to a problem on the virtual environment installation pathway alone. Perhaps that does somehow relate back to pywin32 implementation details. If that is the case, then so be it - I just felt it was prudent to challenge the assertion given that it wasn't aligning with my own findings.

In all fairness, I didn't even catch that this particular post relates to pep582. I'd spent around 14 hours working with different python package managers to see what will work best for my team and missed that detail whenever I was triaging my encounter with this product. My apologies for conflating the issues.

I've determined that pdm isn't the right product for us at this time largely due to a lack for rich IDE integration - but issues like this one are also a factor in that decision. That bums me out because I do think that pdm is far superior in many important ways but it just isn't "there" yet for me to endorse it in a corporate environment with a bunch of junior developers scurrying around. The reason I engaged in this dialogue; however, was to try and offer insights that could help it reach the level of maturity that would enable me to make the switch.

I am always willing to contribute my own free time to open source projects that I am keen on (most of the time I spent on this was during off hours on Friday evening). I'm not helping by just flinging PR's at you though without understanding your thought process for handling edge cases when they crop up.

[frostming]

I originally thought that was working because I have pywin32 installed directly into the particular pyenv environment that I was basing from

Yes, it absolutely makes a difference. There are several conditions to reproduce the issue:

1. You are on Windows, where the install caches are made available via .pth files rather than symlinks, which is the case for Unix systems
2. Install cache on.
3. You are not installing pywin32 into your base Python environment when using PEP 582 mode.

[mro-rhansen2]

I confirmed that wasn't the case by testing a clean pyenv base that didn't contain pywin32 in addition to a base that did contain pywin32. I tested with the install cache enabled as well as disabled. The installation always works when using PEP 582 and always fails when using virtualenv. The only time virtualenv works is when I use pdm run install pywin32 before the install.

I am on a Windows machine using symlinks.

The screengrab below are the results when I am using PEP 582.

[frostming]

I tested with the install cache enabled as well as disabled. The installation always works when using PEP 582 and always fails when using virtualenv.

I thought we are talking about the runtime failure aren't we? the installation always succeeded from my testing. Or please paste the install error if otherwise.

I am on a Windows machine using symlinks.

Not actually, PDM will fallback to .pth when symlinks are not available. You'd better look at the file list under the library path to verify it.

---

UPDATE

Oh, here is my new observation, when using venv mode, with install.cache on, the installation will throw error messages but installation succeeds, I think it is what you suggests.

Look, whenever a pywin32.pth(from pywin32 package) exists in the site-packages, and the real package is linked to the install cache via .pth files, which you can see in a file named aaa_pywin32.pth(generated by PDM install caching), the error occurs. This is because the python interpreter will be invoked several times to get some interpreter information during the installation, and due to the existence of pywin32.pth, it will try to load the pywin32 modules which at that point aren't visible to the interpreter. However, the assumption will of course be broken if you install a real pywin32 into your venv prior to that.

In PEP 582 mode it works fine because the interpreter won't load the __pypackages__ and those shitting .pth files when being invoked.

With the above said, the conclusion keeps the same: when you encounter errors when using install caches, just disable it.

[mro-rhansen2]

Yes, I see the pth files now. I hadn't noticed the extensions.

I don't get any errors at runtime or during installation when using the install cache and PEP 582. I don't even have pywin32 installed in the base. Everything works as expected. I didn't have problems until trying to switch to venv - which I did because IDE's just aren't ready for PEP 582 and I wanted to see if the venv option was workable until IDE support catches up.

I do think I was just able to reproduce the failure that you were referring to though. If I have pywin32 in the base that the venv is cut from, then I only get errors during installation of some of the dependencies. I do not get errors installing the root package nor do I get errors when running the code.

However, if I remove pywin32 from the base, then I get errors during installation and when running the code barking about a missing pywin32_bootstrap module.

I can then disable the install cache and see that the root package installs and executes without errors. However, I am seeing installation errors from dependencies even with the install cache disabled when installing into a venv. Is that just noise?

[frostming]

Hmm, after revisiting the issue and debugging I think we can fix it(for pywin32, but still not guaranteed to work for other packages).

[mro-rhansen2]

This is an awesome project if I hadn't made myself clear. It's a serious contender to be the standard at some point. I'm just pouting because it won't quite work for my shop just yet.

Let me know how I can help. This seems like it's a labor of love on your part though so I can understand if you're wanting to keep it close for the time being. Maybe I'll find some spare time of my own and look into a pdm plugin for Intellij - who knows?