Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(possible) bug with retrying tool downloads #17294

Closed
ryanking opened this issue Oct 20, 2022 · 1 comment · Fixed by #17298
Closed

(possible) bug with retrying tool downloads #17294

ryanking opened this issue Oct 20, 2022 · 1 comment · Fixed by #17298
Assignees
Labels
Milestone

Comments

@ryanking
Copy link
Contributor

Describe the bug

It appears that certain types of failures (tcp rather than http) when downloading tools will not trigger retries and instead lead to a process crash.

Pants version
2.14.0rc1+gitf06bf22b

OS
linux

Additional info

This appears to be happening in the bootstrap phase and here is the entirety of the log output:

16:27:01.19 [WARN] DEPRECATED: option 'repos' in scope 'python-repos' is scheduled to be removed in version 3.0.0.dev0.

A deprecated alias for `[python-repos].find_links`.
16:27:01.31 [INFO] Starting: Resolving plugins: toolchain.pants.plugin==0.22.0
16:27:15.75 [INFO] Completed: Resolving plugins: toolchain.pants.plugin==0.22.0
16:27:16.33 [INFO] [auth-plugin] Successfully loaded Toolchain token from env var 'TOOLCHAIN_AUTH_TOKEN', expiration: 2022-12-20T20:52:09+00:00.
Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 112, in main
    PantsLoader.main()
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 108, in main
    cls.run_default_entrypoint()
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 90, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/pants_runner.py", line 99, in run
    runner = LocalPantsRunner.create(
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 158, in create
    specs = calculate_specs(
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/init/specs_calculator.py", line 72, in calculate_specs
    (changed_addresses,) = session.product_request(
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 577, in product_request
    return self.execute(request)
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
    self._raise_on_error([t for _, t in throws])
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 498, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.vcs.changed.find_changed_owners
  in pants.backend.project_info.dependees.find_dependees
  in pants.backend.project_info.dependees.map_addresses_to_dependees
  in pants.engine.internals.graph.resolve_dependencies (src/infra/pipeline/tools/vcfdiff.sh:shell_sources)
  in pants.backend.shell.dependency_inference.infer_shell_dependencies
  in pants.backend.shell.dependency_inference.parse_shell_imports
  in pants.core.util_rules.external_tool.download_external_tool
  in downloaded_file
Traceback (no traceback):
  <pants native internals>
Exception: Error reading URL fetch response: request or response body error: error reading a body from connection: connection reset
NoneType: None
FATAL: exception not rethrown
build/pants/ci/common.sh: line 14:   543 Aborted                 (core dumped) ./pants "${pants_args[@]}" 2>&1
       544 Done                    | tee "${PANTS_LOG}"
Error: Tests failed ([47](https://github.com/color/color/actions/runs/3291293838/jobs/5425264218#step:9:48)s)
See https://www.pantsbuild.org/docs/python-test-goal for info on how pants runs python tests.
Run './pants test <path...>' to test file(s) with pants locally.
Error: Process completed with exit code 1.
@ryanking ryanking added the bug label Oct 20, 2022
@stuhood
Copy link
Member

stuhood commented Oct 20, 2022

Ah, oops. It looks like the retry was only added to the connection initialization, but not to the actual streaming of the body:

while let Some(next_chunk) = response_stream.next().await {
let chunk =
next_chunk.map_err(|err| format!("Error reading URL fetch response: {}", err))?;
hasher
.write_all(&chunk)
.map_err(|err| format!("Error hashing/capturing URL fetch response: {}", err))?;
}

I'll take a look at this.

@stuhood stuhood self-assigned this Oct 20, 2022
@stuhood stuhood added this to the 2.14.x milestone Oct 20, 2022
stuhood added a commit that referenced this issue Oct 20, 2022
As reported in #17294, if an HTTP stream is interrupted after it has opened, the retry that was added in #16798 won't kick in.

This change moves the retry up a level to wrap the entire download attempt, and adds a test of recovering from "post-header" errors.

Fixes #17294.
stuhood added a commit to stuhood/pants that referenced this issue Oct 21, 2022
…sbuild#17298)

As reported in pantsbuild#17294, if an HTTP stream is interrupted after it has opened, the retry that was added in pantsbuild#16798 won't kick in.

This change moves the retry up a level to wrap the entire download attempt, and adds a test of recovering from "post-header" errors.

Fixes pantsbuild#17294.

[ci skip-build-wheels]
stuhood added a commit to stuhood/pants that referenced this issue Oct 21, 2022
…sbuild#17298)

As reported in pantsbuild#17294, if an HTTP stream is interrupted after it has opened, the retry that was added in pantsbuild#16798 won't kick in.

This change moves the retry up a level to wrap the entire download attempt, and adds a test of recovering from "post-header" errors.

Fixes pantsbuild#17294.

[ci skip-build-wheels]
stuhood added a commit to stuhood/pants that referenced this issue Oct 21, 2022
…sbuild#17298)

As reported in pantsbuild#17294, if an HTTP stream is interrupted after it has opened, the retry that was added in pantsbuild#16798 won't kick in.

This change moves the retry up a level to wrap the entire download attempt, and adds a test of recovering from "post-header" errors.

Fixes pantsbuild#17294.

[ci skip-build-wheels]
stuhood added a commit to stuhood/pants that referenced this issue Oct 21, 2022
…sbuild#17298)

As reported in pantsbuild#17294, if an HTTP stream is interrupted after it has opened, the retry that was added in pantsbuild#16798 won't kick in.

This change moves the retry up a level to wrap the entire download attempt, and adds a test of recovering from "post-header" errors.

Fixes pantsbuild#17294.

[ci skip-build-wheels]
stuhood added a commit that referenced this issue Oct 21, 2022
…ry-pick of #17298) (#17302)

As reported in #17294, if an HTTP stream is interrupted after it has opened, the retry that was added in #16798 won't kick in.

This change moves the retry up a level to wrap the entire download attempt, and adds a test of recovering from "post-header" errors.

Fixes #17294.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants