Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Makes breaks when provenances are too large #1128

Closed
dsalaza4 opened this issue Aug 10, 2023 · 0 comments
Closed

Makes breaks when provenances are too large #1128

dsalaza4 opened this issue Aug 10, 2023 · 0 comments
Assignees

Comments

@dsalaza4
Copy link
Contributor

dsalaza4 commented Aug 10, 2023

When running jobs that have huge provenance files, the CLI function _nix_hashes receives too many arguments and breaks due to ARG_MAX limitations (depending on the OS).

Here is where the bug happens

makes/src/cli/main/cli.py

Lines 329 to 340 in 2244b2b

def _nix_hashes(*paths: str) -> List[str]:
cmd = [
f"{__NIX_STABLE__}/bin/nix-store",
"--query",
"--hash",
*paths,
]
out, stdout, _ = _run_outputs(cmd, stderr=None)
if out != 0:
raise SystemExit(out)
return stdout.decode().splitlines()

Specifically in the *paths argument provided to the shell (line 334).

Let's make _nix_hashes receive bytes from stdin and then call nix-store using xargs so it does not hit the ARG_MAX limit.

I found this bug by running m gitlab:fluidattacks/universe@trunk /docs/generate/criteria, as it currently depends on several hundred derivations for building the entire criteria section of the website.

@dsalaza4 dsalaza4 self-assigned this Aug 10, 2023
dsalaza4 added a commit to dsalaza4/makes that referenced this issue Aug 10, 2023
- Make _nix_hashes receive bytes so it is compatible with stdin
- Make _nix_hashes process provided paths using xargs to avoid
hitting ARG_MAX limit
- Add findutils to runtime so xargs is available
dsalaza4 added a commit to dsalaza4/makes that referenced this issue Aug 10, 2023
- Make _nix_hashes receive bytes so it is compatible with stdin
- Make _nix_hashes process provided paths using xargs to avoid
hitting ARG_MAX limit
- Add findutils to runtime so xargs is available

Signed-off-by: Daniel Salazar <podany270895@gmail.com>
acuberosatfluid added a commit that referenced this issue Aug 10, 2023
feat(back): #1128 avoid hitting max arg limit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant