python_awslambda can include distributions incompatible with explicit complete_platforms #17621
Assignees
Labels
Comments
|
The underlying issue is tracked here: pex-tool/pex#1899 |
|
Ah, thanks 👍 |
|
Is there a work-around? And/or is it worth allowing pants' |
|
There is no workaround besides the obvious build on a compatible image. The path forward is definitely to just fix this in Pex. |
jsirois
pushed a commit
to pex-tool/pex
that referenced
this issue
Nov 25, 2022
…1991) This makes --resolve-local-platforms work better with --complete-platforms: if there's an external complete platform specified for a platform that matches a local interpreter, the local interpreter's complete platform is checked against that external complete platform. If the local interpreter supports any additional tags, the local interpreter is not used. If the local interpreter/environment is newer (supports more tags), the old behaviour would risk including wheels that aren't compatible with the external complete platform. That is, execution will fail, see pantsbuild/pants#17621 for an example. Fixes #1899
|
I've assigned this to @huonw since he actually did all the work. |
|
@huonw this is now fixed with your change released in Pex 2.1.116. You can activate this fix with: [pex-cli]
version = "v2.1.116"
known_versions = [
"v2.1.116|macos_arm64|3158c6ab70272f836daade412974e791a069af2eb1d43aee9c10cfdf63154385|4068322",
"v2.1.116|macos_x86_64|3158c6ab70272f836daade412974e791a069af2eb1d43aee9c10cfdf63154385|4068322",
"v2.1.116|linux_x86_64|3158c6ab70272f836daade412974e791a069af2eb1d43aee9c10cfdf63154385|4068322",
"v2.1.116|linux_arm64|3158c6ab70272f836daade412974e791a069af2eb1d43aee9c10cfdf63154385|4068322"
]Otherwise this will roll out with 2.16.x as things stand. |
jsirois
added a commit
to jsirois/pants
that referenced
this issue
Nov 25, 2022
This upgrade brings in both a `--venv` `sys.path` fix and a fix relevant to awslambda and GCF amongst other fixes. The relevant changelogs are: + https://github.com/pantsbuild/pex/releases/tag/v2.1.114 + https://github.com/pantsbuild/pex/releases/tag/v2.1.115 + https://github.com/pantsbuild/pex/releases/tag/v2.1.116 Fixes pantsbuild#17621 Fixes pantsbuild#17531 Fixes pantsbuild#17372
jsirois
added a commit
that referenced
this issue
Nov 26, 2022
This upgrade brings in both a `--venv` `sys.path` fix and a fix relevant to awslambda and GCF amongst other fixes. The relevant changelogs are: + https://github.com/pantsbuild/pex/releases/tag/v2.1.114 + https://github.com/pantsbuild/pex/releases/tag/v2.1.115 + https://github.com/pantsbuild/pex/releases/tag/v2.1.116 Fixes #17621 Fixes #17531 Fixes #17372
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
We have a Python lambda built via
python_awslambda(very convenient, thanks!) plus a complete platform JSON that we got by runningpex3 interpreter inspect --markers --tagsfrom within the Lambda environment. This environment uses tags up to"cp39-cp39-manylinux_2_26_x86_64".When building on certain Linux systems, it seems that pants will happily include a
manylinux_2_28wheel (NB. 28 vs 26), which isn't compatible with the lambda environment. Pex thus fails at start-up:I've tried to create a reproducer in https://gist.github.com/huonw/504ef2a5d44efdc2c2df290c0ce5df16. This uses a complete platform from the Python 3.9 lambda environment, with a dependency that doesn't have any matching wheels. (Unfortunately due to https://pantsbuild.slack.com/archives/C046T6T9U/p1669150590877779, I cannot offer scripted commands that definitely reproduce it, because I cannot run pants under docker on my machine at the moment.)
The following shell snippet loads the repo, and packages the lambda and checks what is included:
The
distributionsobject should include acryptographydistribution that's compatible with the complete platforms, e.g.cryptography-38.0.3-cp36-abi3-manylinux_2_24_x86_64.whl.cryptography-38.0.3-cp36-abi3-manylinux_2_28_x86_64.whl). The specific environment is Github Actionsubuntu-latestwithactions/setup-python@v4used withpython-version: 3.9. Complete platform in "Additional info" below.I suspect the general behaviour here is that if the host system is close enough to the lambda environment, then
pants/src/python/pants/backend/awslambda/python/rules.py
Lines 107 to 109 in a21d8cb
complete_platformJSON files, and use the host system's one. If the host system has newer glibc versions, this might select an incorrect wheel. In this case, the lambda complete platform is CPython 3.9, and thus running with the same Python version hits this issue.Pants version
2.14.0
OS
Linux
Additional info
Complete platform of failing environment as reported by pex:
The text was updated successfully, but these errors were encountered: