Pants 3rdparty non-repo-local plugins with non-wheel deps cannot be used. #71
Comments
|
Another way to phrase the problem at a higher level is that if Pants ships as a binary, plugins must too. That is a new requirement on Pants plugin authors and translates to ensuring their complete dep set is available as pre-built wheels for all the platforms Pants supports. |
Although horrible, I think that this might involve a one line change like: --- a/src/python/pants/init/plugin_resolver.py
+++ b/src/python/pants/init/plugin_resolver.py
@@ -71,3 +71,3 @@ async def resolve_plugins(
PythonExecutable.fingerprinted(
- sys.executable, ".".join(map(str, sys.version_info[:3])).encode("utf8")
+ sys.executable, ".".join(map(str, sys.version_info[:2])).encode("utf8")
),
|
|
Yeah, seems likely. It truly is horrible though. I think the best path forward is to squash this right away - IIUC there are 0 known 3rdparty plugins today with sdist-only deps for platforms Pants supports - and allow this back in later when we can support it in a non-horrible way. I don't think that day will come, but either way, restrict early, expand later is ~always a good safe bet. |
Wouldn't the transitive deps of in-repo plugins also be affected? |
|
Not really. For an in-repo plugin its always easy to arrange find-links with pre-built wheels. That is slightly clunky but not horrible. For a public 3rdparty repo you now ask someone to support hosting for the world. Of course in-repo assumes in-repo and private. You have the same problem again if the plugin is in-repo and public. |
|
Another left field angle that probably doesn't work is to have the PBS Python re-write its sysconfig in an install step. I'm not sure that even works though - I have not experimented. I think that the bad paths are all in there though. If this did work it would likely be complicated / titchy code. You have to be ~/.configure and go find these paths on the current machine, if they exist (users would need gcc / clang / etc and you'd need to detect where they all were. Basically, if you could get that to work, it would properly belong in PBS or its suite. A tool to localize a PBS Python when possible. |
|
Option #18 is to have scie-pants grow the ability to:
This 18th option would have to be opt-in somehow, since it will fail unless pre-reqs for pyenv are met; so you'd only want to use it if you were forced to by plugins with sdist-only deps. |
|
Modern PBS seems to work with everything I throw at it, including psutil, hdrhistogram 0.10.1 (see #69), numpy, PyYAML and pandas, using Scie pants probably wants to check that out: https://github.com/indygreg/python-build-standalone/releases/tag/20240107 |
jsirois
added
the
in progress
|
I'm not sure if the gedanken issue is already fixed over in Pants, but regardless, #52 indicates an PBS upgrade is warranted. |
Ok, that turned out to be because I happened to have clang installed. With just gcc, those two fail under the last 2 years worth of PBS releases, and the hard-coded clang from the PBS build process is not one of those issues solved in the 20240107 release. |
jsirois
removed
the
in progress
|
And to confirm the gedanken bug is real, in the scie-pants repo itself, without clang installed, just gcc, add an sdist-only platform specific requirement: diff --git a/pants.toml b/pants.toml
index 5d8d7f7..cb68c18 100644
--- a/pants.toml
+++ b/pants.toml
@@ -8,6 +8,10 @@ backend_packages = [
"pants.backend.python.typecheck.mypy",
]
+plugins = [
+ "hdrhistogram==0.10.1",
+]
+
[anonymous-telemetry]
enabled = true
repo_id = "e0b99427-9bc2-4f6a-b197-f5f378849b15"And try to run Pants: jsirois@Gill-Windows:~/dev/pantsbuild/scie-pants (main *) $ pants -V
Bootstrapping Pants 2.18.0
Installing pantsbuild.pants==2.18.0 into a virtual environment at /home/jsirois/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0
Found existing installation: setuptools 58.1.0
Uninstalling setuptools-58.1.0:
Successfully uninstalled setuptools-58.1.0
New virtual environment successfully created at /home/jsirois/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0.
14:41:44.88 [INFO] Starting: Resolving plugins: hdrhistogram==0.10.1
14:41:52.26 [INFO] Completed: Resolving plugins: hdrhistogram==0.10.1
14:41:52.26 [ERROR] 1 Exception encountered:
ProcessExecutionFailure: Process 'Resolving plugins: hdrhistogram==0.10.1' failed with exit code 1.
stdout:
stderr:
pid 1485817 -> /home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/bin/python -sE /home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /home/jsirois/.cache/pants/named_caches/pex_root/built_wheels/sdists/hdrhistogram-0.10.1.tar.gz/542a10f166ea07229cd85a6c09229936749ed7840018993537340fa9ada78566/cp39-cp39-manylinux_2_35_x86_64.21d0e2c1bb5e417b8c85a39d5186aca6.work /home/jsirois/.cache/pants/named_caches/pex_root/downloads/resolver_download.uew9dcp9/home.jsirois..cache.nce.3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e.bindings.venvs.2.18.0.bin.python3.9/hdrhistogram-0.10.1.tar.gz --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [152 lines of output]
...
[pbr] Reusing existing SOURCES.txt
running build_ext
building 'pyhdrh' extension
creating build/temp.linux-x86_64-cpython-39
creating build/temp.linux-x86_64-cpython-39/src
clang -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fdebug-default-version=4 -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -fPIC -I/home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/include -I/home/jsirois/.cache/nce/f3ff38b1ccae7dcebd8bbf2e533c9a984fac881de0ffd1636fbb61842bd924de/cpython-3.9.18+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz/python/include/python3.9 -c src/python-codec.c -o build/temp.linux-x86_64-cpython-39/src/python-codec.o
error: command 'clang' failed: No such file or directory
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for hdrhistogram
ERROR: Failed to build one or more wheels
Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
Use --print-stacktrace for more error details and/or -ldebug for more logs.
See https://www.pantsbuild.org/v2.18/docs/troubleshooting for common issues.
Consider reaching out for help: https://www.pantsbuild.org/v2.18/docs/getting-helpMaybe a ~simple solution - Have scie-pants figure out CC and then plumb it via env vars? Here I try that manually: diff --git a/pants.toml b/pants.toml
index 5d8d7f7..d4653c7 100644
--- a/pants.toml
+++ b/pants.toml
@@ -8,6 +8,10 @@ backend_packages = [
"pants.backend.python.typecheck.mypy",
]
+plugins = [
+ "hdrhistogram==0.10.1",
+]
+
[anonymous-telemetry]
enabled = true
repo_id = "e0b99427-9bc2-4f6a-b197-f5f378849b15"
@@ -39,7 +43,8 @@ known_versions = [
[subprocess-environment]
env_vars = [
- "BUILDROOT=%(buildroot)s"
+ "BUILDROOT=%(buildroot)s",
+ "CC"
]
[source]$ CC=gcc pants -V
14:47:30.51 [INFO] Starting: Resolving plugins: hdrhistogram==0.10.1
14:47:33.35 [INFO] Completed: Resolving plugins: hdrhistogram==0.10.1
14:47:33.35 [ERROR] 1 Exception encountered:
ProcessExecutionFailure: Process 'Resolving plugins: hdrhistogram==0.10.1' failed with exit code 1.
stdout:
stderr:
pid 1487232 -> /home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/bin/python -sE /home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /home/jsirois/.cache/pants/named_caches/pex_root/built_wheels/sdists/hdrhistogram-0.10.1.tar.gz/542a10f166ea07229cd85a6c09229936749ed7840018993537340fa9ada78566/cp39-cp39-manylinux_2_35_x86_64.59d589e13bda458ca8616eeffc6d81a6.work /home/jsirois/.cache/pants/named_caches/pex_root/downloads/resolver_download.nzzxlou4/home.jsirois..cache.nce.3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e.bindings.venvs.2.18.0.bin.python3.9/hdrhistogram-0.10.1.tar.gz --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [153 lines of output]
...
[pbr] Reusing existing SOURCES.txt
running build_ext
building 'pyhdrh' extension
creating build/temp.linux-x86_64-cpython-39
creating build/temp.linux-x86_64-cpython-39/src
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fdebug-default-version=4 -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -fPIC -I/home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/include -I/home/jsirois/.cache/nce/f3ff38b1ccae7dcebd8bbf2e533c9a984fac881de0ffd1636fbb61842bd924de/cpython-3.9.18+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz/python/include/python3.9 -c src/python-codec.c -o build/temp.linux-x86_64-cpython-39/src/python-codec.o
gcc: error: unrecognized command-line option ‘-fdebug-default-version=4’
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for hdrhistogram
ERROR: Failed to build one or more wheels
Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
Use --print-stacktrace for more error details and/or -ldebug for more logs.
See https://www.pantsbuild.org/v2.18/docs/troubleshooting for common issues.
Consider reaching out for help: https://www.pantsbuild.org/v2.18/docs/getting-helpSo, apparently no easy way out. |
This is a gedanken issue and I've labelled it a bug, but its unclear how to treat this.
The problem:
If a Pants user has a
[BOOTSTRAP] plugins = [ ... ]entry that requires compilation of an sdist, they cannot use scie-pants, since scie-pants uses Python Build Standalone interpreters and these have an unuseable sysconfig / cannot build C extensions.To work around this it seems you'd either:
The latter is a horrible solution since it requires you have a local interpreter exactly matching the "hidden" Pants interpreter major / minor version (Generally 3.9 currently). This is strictly worse than the status quo of the
./pantsscript today, which has the same requirements, but with more lenient Python major / minor version strictures, accepting any of 3.{7,8,9}.The text was updated successfully, but these errors were encountered: