Include/exclude and heuristic-based immutable_inputs

[stuhood]

Process.immutable_inputs is currently configured as a mapping from a file path to a Digest to materialize at that path. It is merged into the input_digest of a Process to form a "complete" Digest of inputs.

But this separation is a bit unfortunate, because it means that using immutable_inputs (which should be fairly widely used: see discussion on #14070) means a lot of change to how your Process is constructed.

---

From an API-changes perspective, we should:

1. remove the guarantee that inputs are mutable by default, to allow us to automatically choose the strategy that we use for each path, and symlink wherever we choose to.
   • Then, by default, we should automatically use symlinking for files or directories over a certain size, etc.
2. convert the existing immutable_inputs list into a flat list of paths which should be forced/includelisted to be materialized as immutable (regardless of the heuristic from above), but which must already exist in the input_digest (maybe "force" should go in the name? force_immutable_paths= ...?).
   • This will mean that unlike today (where you need to change how your Process is constructed to stop adding an input to the input_digests and instead add it to the immutable_inputs as a dict), you would only need to add a path to the immutable_inputs, without changing your input_digest.
3. add an mutable_inputs (or force_mutable_paths=...?) flat list of paths to act as an excludelist that prevents a path from being materialized as a symlink.

---

From an implementation perspective, this would look like moving the support for creating the symlinks for ImmutableInputs from an explicit step before running a process:

pants/src/rust/engine/process_execution/src/local.rs

Lines 641 to 662 in 08410e5

	// Collect the symlinks to create for immutable inputs or named caches.
	let immutable_inputs_symlinks = {
	let symlinks = immutable_inputs
	.local_paths(&req.input_digests.immutable_inputs)
	.await?;
	match immutable_inputs_prefix {
	Some(prefix) => symlinks
	.into_iter()
	.map(|symlink| WorkdirSymlink {
	src: symlink.src,
	dst: prefix.join(
	symlink
	.dst
	.strip_prefix(immutable_inputs.workdir())
	.unwrap(),
	),
	})
	.collect::<Vec<_>>(),
	None => symlinks,
	}
	};

...to a thing that happens inside fn materialize_directory when an includelisted path is encountered:

pants/src/rust/engine/fs/store/src/lib.rs

Lines 1188 to 1201 in 08410e5

	///
	/// Lays out the directory and all of its contents (files and directories) on disk so that a
	/// process which uses the directory structure can run.
	///
	/// Although `Directory` has internally unique paths, `materialize_directory` can be used with
	/// an existing destination directory, meaning that directory and file creation must be
	/// idempotent.
	///
	pub async fn materialize_directory(
	&self,
	destination: PathBuf,
	digest: DirectoryDigest,
	perms: Permissions,
	) -> Result<(), StoreError> {

materialize_directory(_helper) would take either:

1. a reference to a collection of paths which were includelisted as immutable
2. a trait or function which implemented the heuristic for which directory::Entrys / paths to includelist

...and a reference to ImmutableInputs, and then would create a symlink to the ImmutableInputs whenever it encountered one of the includelisted paths.

[thejcannon]

In the most common case (fixers) couldn't we deduce the mutable paths by the intersection of the input and output paths?

The output paths for any process will either a) not exist yet or b) do exist (from the inputs) and therefore need to be mutable (or else why are you capturing them)

[stuhood]

In the most common case (fixers) couldn't we deduce the mutable paths by the intersection of the input and output paths?

The output paths for any process will either a) not exist yet or b) do exist (from the inputs) and therefore need to be mutable (or else why are you capturing them)

Yea, for sure.