-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue resolving generated Python module with source at root #17531
Comments
As @dlsmith intuited, this is because The sandbox dir contains the sources (authored and generated) and is added to the sys.path. And that's where we want them to be loaded from. But if the cwd is also the source root, then Python will see the top-level package in the source tree first, load it from there, and because it's not a namespace package, it will look no further for any imports under that package. And the generated code is not in the source tree, so it won't find it. This does not happen with Pants processes in general, because they are fully sandboxed. It's just |
Solution is to override that sys.path[0] setting somehow, looking into it |
I think this requires a Pex feature. It's the one non-hermetic affordance Pex makes, and that's due to competing goals:
Here Pex chooses the Python interpreter behavior of pre-pending . to the sys.path (2) over hermeticity (1). I think this could be both a Pex build-time option and PEX_* runtime override. |
@dlsmith how disruptive is this for you? |
@benjyw Not at all disruptive. I'm happy to adjust the structure of my repo to work around the issue. Though maybe it would be worth adding a warning in the documentation? |
Yeah that makes sense. The docs are in the repo (under docs/) if you feel like cooking up a quick PR? :) Long term I'd like to fix this in pex, but the priority is not high if this isn't a big issue for you, and no one else appears to have hit this. |
This upgrade brings in both a `--venv` `sys.path` fix and a fix relevant to awslambda and GCF amongst other fixes. The relevant changelogs are: + https://github.com/pantsbuild/pex/releases/tag/v2.1.114 + https://github.com/pantsbuild/pex/releases/tag/v2.1.115 + https://github.com/pantsbuild/pex/releases/tag/v2.1.116 Fixes pantsbuild#17621 Fixes pantsbuild#17531 Fixes pantsbuild#17372
Describe the bug
Python failed to resolve a generated module (in this case, from protobuf) when the project was structured with the Python source at the root of the package. My hypothesis is that this is due to the way Python resolves modules: it looks first in the execution directory, then to
PYTHONPATH
. Pants links the generated code from the sandbox via thePYTHONPATH
, but if it replicates/preserves the semantics of Python, the interpreter may first look in a file tree with the parent module structure but without the generated files, leading to an import error.I've included a minimal example below that demonstrates the issue and works around it by nesting the source root under a
src/
prefix.Pants version
2.14.0
OS
MacOS
Additional info
pants-codegen-repro.tar.gz
The text was updated successfully, but these errors were encountered: