missing digest on main and 2.13.x branch

[tdyas]

Observed a missing digest on main even with backtracking changes in place:

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.pytest_runner.run_python_test (src/python/pants/jvm/compile_test.py:tests)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
  in pants.backend.python.util_rules.pex.create_venv_pex (pytest_runner.pex)
  in pants.backend.python.util_rules.pex.build_pex (pytest_runner.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
  in process
Traceback (no traceback):
  <pants native internals>
Exception: Could not recover from missing digest: Was not present in either the local or remote store: Digest { hash: Fingerprint<217f1d5845a05a89384e5d1e58b23372731b447685ef36c4b204845c2d6ae231>, size_bytes: 2768 }
Error: Process completed with exit code 1.

https://github.com/pantsbuild/pants/runs/7113601869?check_suite_focus=true#step:13:424

[tdyas]

cc @stuhood

[tdyas]

Another example (but on 2.13.x branch):

Engine traceback:
  in select
  in pants.core.goals.lint.lint
  in pants.backend.scala.lint.scalafmt.rules.scalafmt_fmt (scalafmt)
  in pants.engine.process.fallible_to_exec_result_or_raise
  in process
Traceback (no traceback):
  <pants native internals>
Exception: Could not recover from missing digest: Was not present in either the local or remote store: Digest { hash: Fingerprint<9484ac95856510459d1bd52a77a6b93cdd641560decdf9910395ee4d17e88163>, size_bytes: 121417 }
Error: Process completed with exit code 1.

https://github.com/pantsbuild/pants/runs/7116113010?check_suite_focus=true#step:9:279

[tdyas]

Same build failed again but only with local store this time:

Engine traceback:
  in select
  in pants.core.goals.lint.lint
  in pants.backend.python.lint.black.rules.black_fmt (black)
  in pants.engine.process.fallible_to_exec_result_or_raise
  in process
Traceback (no traceback):
  <pants native internals>
Exception: Could not recover from missing digest: Was not present in the local store: Digest { hash: Fingerprint<740528d33c19eca061f11cbee7fd7b030928a8b7b53a5023da6e6168236cc0b9>, size_bytes: 1817 }
Error: Process completed with exit code 1.

https://github.com/pantsbuild/pants/runs/7116545174?check_suite_focus=true#step:9:242

[tdyas]

Would cherry-picking #15979 help?

[stuhood]

Would cherry-picking #15979 help?

No, it won't: this is an async method failing rather than a sync method failing. Thanks for the cherry-pick though anyway!

I'll need to re-audit whether the MissingDigest error is being properly preserved when invoking a process.

[stuhood]

Same build failed again but only with local store this time:

This is apparently because that build failed to authenticate to Toolchain:

16:43:31.72 [WARN] Auth failed - BuildSense plugin is disabled.

...and so remote caching was disabled. How it then encountered an unrecoverable MissingDigest is a mystery though.

[stuhood]

#16008 should slightly adjust the output for this error. I'll be on the lookout for additional cases, but please report them if you see them.

[stuhood]

https://github.com/pantsbuild/pants/runs/7122588680?check_suite_focus=true

[jsirois]

https://github.com/pantsbuild/pants/runs/7163844276?check_suite_focus=true#step:13:468

[jsirois]

https://github.com/pantsbuild/pants/runs/7163948016?check_suite_focus=true#step:13:409

[jsirois]

https://github.com/pantsbuild/pants/runs/7166051510?check_suite_focus=true#step:13:724

[jsirois]

https://github.com/pantsbuild/pants/runs/7166173388?check_suite_focus=true#step:13:708

[jsirois]

https://github.com/pantsbuild/pants/runs/7166187778?check_suite_focus=true#step:13:684

[jsirois]

Noting these last 3 above were all from the same PR / were re-runs. They all failed on the same missing digest. The 4th run succeeded apparently only because of a failure acquiring BuildSense auth.

[stuhood]

If this is causing folks enough trouble, we can re-enable eager_fetch until it is resolved:

pants/pants.toml

Lines 71 to 72 in 8efc396

	# See https://github.com/pantsbuild/pants/issues/11331.
	remote_cache_eager_fetch = false

[jsirois]

I'm prepping that PR presently - red main should not be tolerated / this is in fact causing pain.