(possible) bug with retrying tool downloads

[ryanking]

Describe the bug

It appears that certain types of failures (tcp rather than http) when downloading tools will not trigger retries and instead lead to a process crash.

Pants version
2.14.0rc1+gitf06bf22b

OS
linux

Additional info

This appears to be happening in the bootstrap phase and here is the entirety of the log output:

16:27:01.19 [WARN] DEPRECATED: option 'repos' in scope 'python-repos' is scheduled to be removed in version 3.0.0.dev0.

A deprecated alias for `[python-repos].find_links`.
16:27:01.31 [INFO] Starting: Resolving plugins: toolchain.pants.plugin==0.22.0
16:27:15.75 [INFO] Completed: Resolving plugins: toolchain.pants.plugin==0.22.0
16:27:16.33 [INFO] [auth-plugin] Successfully loaded Toolchain token from env var 'TOOLCHAIN_AUTH_TOKEN', expiration: 2022-12-20T20:52:09+00:00.
Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 112, in main
    PantsLoader.main()
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 108, in main
    cls.run_default_entrypoint()
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 90, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/pants_runner.py", line 99, in run
    runner = LocalPantsRunner.create(
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 158, in create
    specs = calculate_specs(
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/init/specs_calculator.py", line 72, in calculate_specs
    (changed_addresses,) = session.product_request(
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 577, in product_request
    return self.execute(request)
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
    self._raise_on_error([t for _, t in throws])
  File "/home/runner/.pants/setup/bootstrap-Linux-x86_64/2.14.0rc1+gitf06bf22b_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 498, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.vcs.changed.find_changed_owners
  in pants.backend.project_info.dependees.find_dependees
  in pants.backend.project_info.dependees.map_addresses_to_dependees
  in pants.engine.internals.graph.resolve_dependencies (src/infra/pipeline/tools/vcfdiff.sh:shell_sources)
  in pants.backend.shell.dependency_inference.infer_shell_dependencies
  in pants.backend.shell.dependency_inference.parse_shell_imports
  in pants.core.util_rules.external_tool.download_external_tool
  in downloaded_file
Traceback (no traceback):
  <pants native internals>
Exception: Error reading URL fetch response: request or response body error: error reading a body from connection: connection reset
NoneType: None
FATAL: exception not rethrown
build/pants/ci/common.sh: line 14:   543 Aborted                 (core dumped) ./pants "${pants_args[@]}" 2>&1
       544 Done                    | tee "${PANTS_LOG}"
Error: Tests failed ([47](https://github.com/color/color/actions/runs/3291293838/jobs/5425264218#step:9:48)s)
See https://www.pantsbuild.org/docs/python-test-goal for info on how pants runs python tests.
Run './pants test <path...>' to test file(s) with pants locally.
Error: Process completed with exit code 1.

[stuhood]

Ah, oops. It looks like the retry was only added to the connection initialization, but not to the actual streaming of the body:

pants/src/rust/engine/src/downloads.rs

Lines 184 to 190 in f004a8a

	while let Some(next_chunk) = response_stream.next().await {
	let chunk =
	next_chunk.map_err(|err| format!("Error reading URL fetch response: {}", err))?;
	hasher
	.write_all(&chunk)
	.map_err(|err| format!("Error hashing/capturing URL fetch response: {}", err))?;
	}

I'll take a look at this.