Handle all output file names when building scies. #2484
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, invoking `pex pex -c pex --venv --scie eager -o pex` to build the Pex PEX scie would error deep inside the science scie build process due to the name collision. Since the user has no control over the lift manifest, this was not able to be corrected.
jsirois
requested review from
zmanji,
cburroughs,
kaos,
benjyw,
huonw and
sureshjoshi
jsirois
commented
Jul 29, 2024
|
|
||
| pex = os.path.join(str(tmpdir), "pex") | ||
| run_pex_command( | ||
| args=[pex_project_dir, "-c", "pex", "--scie", "lazy", "-o", pex] |
There was a problem hiding this comment.
Previously:
:; pex pex -c pex --scie lazy -o pex
Failed to build 1 scie:
1. For CPython 3.11 on linux-x86_64: Command `/home/jsirois/.pex/scies/science/0.3.0/bin/science --cache-dir /home/jsirois/.pex/scies/science/0.3.0/cache lift --file pex=pex --file configure-binding.py=/home/jsirois/.local/bin/tools.venv/lib/python3.11/site-packages/pex/scie/configure-binding.py build --dest-dir /tmp pex-lift-linux-x86_64.toml` failed with exit code 1 (saved lift manifest to pex-lift-linux-x86_64.toml for inspection):
warning: Failed to gather git state for provenance.
Error: Failed to open binary /tmp/tmpht0jwj9r/linux-x86_64/pex for writing Lift { name: "pex", description: None, base: None, load_dotenv: false, size: 2028, hash: "c8df8f64b6b3bc727700f8274fea40dac22512ab5d9b37f4dd4f1a938824cc2f", boot: Boot { commands: {"": Cmd { exe: "{scie.bindings.configure:PYTHON}", args: ["{scie.bindings.configure:PEX}"], env: {Default("PEX_ROOT"): Some("{scie.bindings}/pex_root")}, description: None }}, bindings: {"configure": Cmd { exe: "{cpython}/python/bin/python3.11", args: ["{pex}", "{configure-binding.py}"], env: {Replace("PATH"): None, Replace("PEX_EMIT_WARNINGS"): Some("0"), Replace("PEX_INTERPRETER"): Some("1"), Replace("_PEX_SCIE_INSTALLED_PEX_DIR"): Some("{scie.bindings}/pex_root/unzipped_pexes/562e932f26691cd0f377821c182589bb8ca9f50b"), Default("PEX_.*"): None, Default("PEX_ROOT"): Some("{scie.bindings}/pex_root")}, description: None }, "fetch": Cmd { exe: "{ptex}", args: ["{scie.env.PEX_BOOTSTRAP_URLS={scie.lift}}"], env: {}, description: Some("Fetch files not present in the scie") }} }, files: [File { name: "ptex-linux-x86_64", key: Some("ptex"), size: 6464608, hash: "6ad0b354ee1c1cea32e0fe06e8c3e86fa67722ba44ed70f222aa708465c90c33", file_type: Blob, executable: Some(true), eager_extract: false, source: Scie }, File { name: "cpython-3.11.9+20240726-x86_64-unknown-linux-gnu-install_only.tar.gz", key: Some("cpython"), size: 29713170, hash: "f6e955dc9ddfcad74e77abe6f439dac48ebca14b101ed7c85a5bf3206ed2c53d", file_type: Archive(CompressedTar(Gzip)), executable: None, eager_extract: false, source: LoadBinding("fetch") }, File { name: "configure-binding.py", key: None, size: 837, hash: "3b18b265f8aa2c2f2042f5c1e83b41b2f57eaecaf7ea437455ba7b82b71b61cf", file_type: Blob, executable: None, eager_extract: false, source: Scie }, File { name: "pex", key: None, size: 4170089, hash: "80ff27fd894824feed1eb9d59cbde67afa60b90998b4a127bb0509e138527f11", file_type: Blob, executable: Some(true), eager_extract: false, source: Scie }], other: Some(Other { other: {"ptex": Object {"cpython-3.11.9+20240726-x86_64-unknown-linux-gnu-install_only.tar.gz": String("https://github.com/indygreg/python-build-standalone/releases/download/20240726/cpython-3.11.9%2B20240726-x86_64-unknown-linux-gnu-install_only.tar.gz")}} }) }: File exists (os error 17)
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/jsirois/.cache/nce/c089791c1ddc701f58f6fbb903c625c24349d5c53d84c7db6eab3141abcf741e/science.pyz/__main__.py", line 3, in <module>
_bootstrap.bootstrap()
File "/home/jsirois/.cache/nce/c089791c1ddc701f58f6fbb903c625c24349d5c53d84c7db6eab3141abcf741e/science.pyz/_bootstrap/__init__.py", line 253, in bootstrap
run(import_string(env.entry_point))
File "/home/jsirois/.cache/nce/c089791c1ddc701f58f6fbb903c625c24349d5c53d84c7db6eab3141abcf741e/science.pyz/_bootstrap/__init__.py", line 38, in run
sys.exit(module())
^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/science/exe.py", line 736, in main
_main(prog_name=SCIE_ARGV0)
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/click/decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/science/exe.py", line 700, in _build
assembly_info = build.assemble_scies(
^^^^^^^^^^^^^^^^^^^^^
File "/home/jsirois/.cache/nce/f2a1a0db8bd8fd9a6bafeec0e3b2056757f1e9b4dc5eb677c909667dfc0737b4/bindings/shiv_root/science.pyz_ad030480c8266efca619353ff049b911fee04d2192a1b414c01bf0ca2f16f69e/site-packages/science/commands/build.py", line 57, in assemble_scies
subprocess.run(
File "/home/jsirois/.cache/nce/eb5f83bfcd16391a5c010356e82708a0f1e6b7d4d8106f21e545a858f7d5f409/cpython-3.12.4+20240713-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/jsirois/.pex/scies/science/0.3.0/cache/downloads/77ee2f754692233411ed53545c57b90a1c173e6dda951aae02fe34018964e4be', '-sj', '/home/jsirois/.pex/scies/science/0.3.0/cache/downloads/e2ac47ed97a3c09555d34f186a8fbece015b1f6b8ed2ba0e9484211c7b77151f', '/tmp/tmpht0jwj9r/linux-x86_64/lift.json']' returned non-zero exit status 1.
|
Reviewers - this is good to go for a review / fix release. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, invoking
pex pex -c pex --venv --scie eager -o pextobuild the Pex PEX scie would error deep inside the science scie build
process due to the name collision. Since the user has no control over
the lift manifest, this was not able to be corrected.