-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
python_awslambda can include distributions incompatible with explicit complete_platforms #17621
Comments
The underlying issue is tracked here: pex-tool/pex#1899 |
Ah, thanks 👍 |
Is there a work-around? And/or is it worth allowing pants' |
There is no workaround besides the obvious build on a compatible image. The path forward is definitely to just fix this in Pex. |
…1991) This makes --resolve-local-platforms work better with --complete-platforms: if there's an external complete platform specified for a platform that matches a local interpreter, the local interpreter's complete platform is checked against that external complete platform. If the local interpreter supports any additional tags, the local interpreter is not used. If the local interpreter/environment is newer (supports more tags), the old behaviour would risk including wheels that aren't compatible with the external complete platform. That is, execution will fail, see pantsbuild/pants#17621 for an example. Fixes #1899
I've assigned this to @huonw since he actually did all the work. |
@huonw this is now fixed with your change released in Pex 2.1.116. You can activate this fix with: [pex-cli]
version = "v2.1.116"
known_versions = [
"v2.1.116|macos_arm64|3158c6ab70272f836daade412974e791a069af2eb1d43aee9c10cfdf63154385|4068322",
"v2.1.116|macos_x86_64|3158c6ab70272f836daade412974e791a069af2eb1d43aee9c10cfdf63154385|4068322",
"v2.1.116|linux_x86_64|3158c6ab70272f836daade412974e791a069af2eb1d43aee9c10cfdf63154385|4068322",
"v2.1.116|linux_arm64|3158c6ab70272f836daade412974e791a069af2eb1d43aee9c10cfdf63154385|4068322"
] Otherwise this will roll out with 2.16.x as things stand. |
This upgrade brings in both a `--venv` `sys.path` fix and a fix relevant to awslambda and GCF amongst other fixes. The relevant changelogs are: + https://github.com/pantsbuild/pex/releases/tag/v2.1.114 + https://github.com/pantsbuild/pex/releases/tag/v2.1.115 + https://github.com/pantsbuild/pex/releases/tag/v2.1.116 Fixes pantsbuild#17621 Fixes pantsbuild#17531 Fixes pantsbuild#17372
This upgrade brings in both a `--venv` `sys.path` fix and a fix relevant to awslambda and GCF amongst other fixes. The relevant changelogs are: + https://github.com/pantsbuild/pex/releases/tag/v2.1.114 + https://github.com/pantsbuild/pex/releases/tag/v2.1.115 + https://github.com/pantsbuild/pex/releases/tag/v2.1.116 Fixes #17621 Fixes #17531 Fixes #17372
Describe the bug
We have a Python lambda built via
python_awslambda
(very convenient, thanks!) plus a complete platform JSON that we got by runningpex3 interpreter inspect --markers --tags
from within the Lambda environment. This environment uses tags up to"cp39-cp39-manylinux_2_26_x86_64"
.When building on certain Linux systems, it seems that pants will happily include a
manylinux_2_28
wheel (NB. 28 vs 26), which isn't compatible with the lambda environment. Pex thus fails at start-up:I've tried to create a reproducer in https://gist.github.com/huonw/504ef2a5d44efdc2c2df290c0ce5df16. This uses a complete platform from the Python 3.9 lambda environment, with a dependency that doesn't have any matching wheels. (Unfortunately due to https://pantsbuild.slack.com/archives/C046T6T9U/p1669150590877779, I cannot offer scripted commands that definitely reproduce it, because I cannot run pants under docker on my machine at the moment.)
The following shell snippet loads the repo, and packages the lambda and checks what is included:
The
distributions
object should include acryptography
distribution that's compatible with the complete platforms, e.g.cryptography-38.0.3-cp36-abi3-manylinux_2_24_x86_64.whl
.cryptography-38.0.3-cp36-abi3-manylinux_2_28_x86_64.whl
). The specific environment is Github Actionsubuntu-latest
withactions/setup-python@v4
used withpython-version: 3.9
. Complete platform in "Additional info" below.I suspect the general behaviour here is that if the host system is close enough to the lambda environment, then
pants/src/python/pants/backend/awslambda/python/rules.py
Lines 107 to 109 in a21d8cb
complete_platform
JSON files, and use the host system's one. If the host system has newer glibc versions, this might select an incorrect wheel. In this case, the lambda complete platform is CPython 3.9, and thus running with the same Python version hits this issue.Pants version
2.14.0
OS
Linux
Additional info
Complete platform of failing environment as reported by pex:
The text was updated successfully, but these errors were encountered: