BinaryPaths should fingerprint found paths.

[jsirois]

BinaryPaths will often find binaries in standard search paths like /bin and /usr/bin. These paths are subject to upgrades on user machines where a given path like /usr/bin/python or /usr/bin/bash will stay constant over time but its contents will change in ways that can affect program output when upgraded or downgraded. Hashing the contents of the discovered binary paths will automatically invalidate downstream results in the large majority of cases which is what we want.

See #10768 (comment) for motivating discussion of this.

[stuhood]

As discussed in slack, this should likely use an absolute file watching facility in the engine, possibly by exposing intrinsics like PathGlobsAndRoot->Snapshot (both existing types, but making PathGlobsAndRoot a Node in the Graph to memoize/watch it would take some refactoring).

I'm not sure whether it should be exposed to @rules though, or whether it should only be consumed by the CommandRunners as an implementation detail (as described in point two of #10526).

[jsirois]

Point two of #10526 will need to support something like the tests shown here if we're to do this in CommandRunners:

1. The name of the binary may not be enough to find a good one:
   Python needs the binary discovery process to support a test:

   pants/src/python/pants/engine/process.py

   Lines 316 to 332 in 895b914

   	@frozen_after_init
   	@dataclass(unsafe_hash=True)
   	class BinaryPathRequest:
   	"""Request to find a binary of a given name.
   	If `test_args` are specified all binaries that are found will be executed with these args and
   	only those binaries whose test executions exit with return code 0 will be retained.
   	Additionally, if test execution includes stdout content, that will be used to fingerprint the
   	binary path so that upgrades and downgrades can be detected. A reasonable test for many programs
   	might be to pass `["--version"]` for `test_args` since it will both ensure the program runs and
   	also produce stdout text that changes upon upgrade or downgrade of the binary at the discovered
   	path.
   	"""
   	search_path: Tuple[str, ...]
   	binary_name: str
   	test_args: Optional[Tuple[str, ...]]

   
   So it can find a valid bootstrap python in the face of pythonX.Y binaries not being installed to carry the needed info in the binary name:

   pants/src/python/pants/backend/python/util_rules/pex_environment.py

   Lines 136 to 182 in 0f43baa

   	# PEX files are compatible with bootstrapping via Python 2.7 or Python 3.5+. The bootstrap
   	# code will then re-exec itself if the underlying PEX user code needs a more specific python
   	# interpreter. As such, we look for many Pythons usable by the PEX bootstrap code here for
   	# maximum flexibility.
   	all_python_binary_paths = await MultiGet(
   	[
   	Get(
   	BinaryPaths,
   	BinaryPathRequest(
   	search_path=python_setup.interpreter_search_paths,
   	binary_name=binary_name,
   	test_args=[
   	"-c",
   	# N.B.: The following code snippet must be compatible with Python 2.7 and
   	# Python 3.5+.
   	dedent(
   	"""\
   	import sys
   	major, minor = sys.version_info[:2]
   	if (major, minor) == (2, 7) or (major == 3 and minor >= 5):
   	# Here we hash the underlying python interpreter executable to
   	# ensure we detect changes in the real interpreter that might
   	# otherwise be masked by pyenv shim scripts found on the search
   	# path. Naively, just printing out the full version_info would be
   	# enough, but that does not account for supported abi changes (e.g.:
   	# a pyenv switch from a py27mu interpreter to a py27m interpreter.
   	import hashlib
   	hasher = hashlib.sha256()
   	with open(sys.executable, "rb") as fp:
   	# We pick 8192 for efficiency of reads and fingerprint updates
   	# (writes) since it's a common OS buffer size and an even
   	# multiple of the hash block size.
   	for chunk in iter(lambda: fp.read(8192), b""):
   	hasher.update(chunk)
   	sys.stdout.write(hasher.hexdigest())
   	sys.exit(0)
   	else:
   	sys.exit(1)
   	"""
   	),
   	],
   	),
   	)
   	for binary_name in pex_runtime_env.bootstrap_interpreter_names
   	]
   	)

2. More complex search logic may need to be triggered than can easily be expressed in a BinaryPathsRequest test.
   This example could probably re-use BinaryPathsRequest if it accepted an input digest:

   pants/src/python/pants/backend/python/util_rules/pex.py

   Lines 393 to 433 in 0e809db

   	@rule
   	async def find_interpreter(interpreter_constraints: PexInterpreterConstraints) -> PythonExecutable:
   	process = await Get(
   	Process,
   	PexCliProcess(
   	description=f"Find interpreter for constraints: {interpreter_constraints}",
   	# Here we run the Pex CLI with no requirements which just selects an interpreter and
   	# normally starts an isolated repl. By passing `--` we force the repl to instead act as
   	# an interpreter (the selected one) and tell us about itself. The upshot is we run the
   	# Pex interpreter selection logic unperturbed but without resolving any distributions.
   	argv=(
   	*interpreter_constraints.generate_pex_arg_list(),
   	"--",
   	"-c",
   	# N.B.: The following code snippet must be compatible with Python 2.7 and
   	# Python 3.5+.
   	dedent(
   	"""\
   	import hashlib
   	import os
   	import sys
   	python = os.path.realpath(sys.executable)
   	print(python)
   	hasher = hashlib.sha256()
   	with open(python, "rb") as fp:
   	# We pick 8192 for efficiency of reads and fingerprint updates
   	# (writes) since it's a common OS buffer size and an even
   	# multiple of the hash block size.
   	for chunk in iter(lambda: fp.read(8192), b""):
   	hasher.update(chunk)
   	print(hasher.hexdigest())
   	"""
   	),
   	),
   	),
   	)
   	result = await Get(ProcessResult, UncacheableProcess(process))
   	path, fingerprint = result.stdout.decode().strip().splitlines()
   	return PythonExecutable(path=path, fingerprint=fingerprint)

[stuhood]

The name of the binary may not be enough to find a good one:

@jsirois : That makes sense, but it feels like it couples "discovering all entries on a PATH" with "filtering whether a particular discovered PATH entry is valid". The latter thing can easily be implemented in "user space" by @rules... the former thing is the thing that I think maybe needs explicit support from the CommandRunner.

[jsirois]

The latter thing can easily be implemented in "user space" by @rules...

Sure, but if the filtering is separate and this is not an intrinsic but built into the CommandRunner as you suggested then the sequence becomes:

1. Use CommandRunner just to return binary paths.
2. Filter those paths in a rule by running another Process(es).
3. Finally execute a Process with the filtered binary as argv0.

The current userspace implementation does:

1. Use CommandRunner to return binary paths by executing a script :/.
2. Filter those paths in a rule by running a Process per each.
3. Finally execute a Process with the filtered binary as argv0.

I think we need to replace 1 with a native binary for OSX and a native binary for Linux (to replace the script + hashing of the binaries found pre-filter) since we need to support doing all this over remoting. IOW, even though the local CommandRunner or a local intrinsic could search the search path and fingerprint all in-process, the remoting implementation cannot. For that we need the aformentioned platform specific binaries to push into the CAS for remoting to execute. Since we need that, we might as well just head there 1st. Doing it in-process in the local CommandRunner becomes a performance optimization only.

[stuhood]

I think we need to replace 1 with a native binary for OSX and a native binary for Linux (to replace the script + hashing of the binaries found pre-filter) since we need to support doing all this over remoting. IOW, even though the local CommandRunner or a local intrinsic could search the search path and fingerprint all in-process, the remoting implementation cannot. For that we need the aformentioned platform specific binaries to push into the CAS for remoting to execute. Since we need that, we might as well just head there 1st. Doing it in-process in the local CommandRunner becomes a performance optimization only.

I'm less sure about this. Yes, the uniformity between local and remote is good, but not being able to use our file watching facilities because the files that are accessed are hidden from the engine in a forked external binary will mean that we need to fork that binary on every run and have it re-fingerprint things repeatedly.

The reason I mention PathGlobsAndRoot (have opened #10842 about this) is that I think that it is important that the facility we use locally uses file watching to avoid re-fingerprinting on every run. As you mentioned, that isn't something that we can avoid for remote execution. So rather than "always forking a binary", I think that this might be a library implemented generically atop the VFS abstraction (which the engine implements in terms of file watching), which the local CommandRunner uses in-process with file-watching, and the remote CommandRunner runs as a binary "once per docker image digest" or something.

[jsirois]

I'm less sure about this. Yes, the uniformity between local and remote is good, but not being able to use our file watching facilities because the files that are accessed are hidden from the engine in a forked external binary will mean that we need to fork that binary on every run and have it re-fingerprint things repeatedly.

We appear to violently agree. As I said, a native binary is needed for remoting so tactically it seemed to make sense to do this 1st. All the latter will be a performance optimization for the local use case.

[stuhood]

Got it, yes. I think that I misread the "native binary for OSX and a native binary for Linux " as "local binary and remote binary", which is clearly not what you meant on re-read. Thanks!