Introduce hash_utils.hash_dir.

[jsirois]

Implement a simple stable hash for the recursive contents of a directory
for use in v1 Tasks. In v2 we have the fs/store crate and intrinsics
that expose its types to @rules; this just tides v1 Tasks over for cases
where we just need a hash and not a Snapshot we'll never materialize
(via self.context._scheduler.<adhoc sync apis>). Performance is
~identical to the self.context._scheduler.capture_snapshots API (a
few percent faster on average).

This change supports #8263.

[Eric-Arellano]

Awesome! Thank you for the type hints and pathlib.Path usage!

[stuhood]

It's not clear to me that "a few percent faster" is worth the extra API (which won't support ignore patterns, etc). But shrug.

[jsirois]

It's not clear to me that "a few percent faster" is worth the extra API

I didn't set out for that reason - the existing "API" was unkown to me. Its also "private" with self.context._scheduler. After finding it after I went to use it, found out it was slower - which was a bit shocking, and then I tried to fully use the api instead of bastardize it to get a fingeprint as a side effect never using the snapshot. Materializing from the snapshot to ensure I had what I fingerprinted, found a 4 minute materialize on a django example project. At that point I returned to what I had not wanting to sink more time into the project.

[stuhood]

Materializing from the snapshot to ensure I had what I fingerprinted, found a 4 minute materialize on a django example project.

Hmm. That's pretty unexpected. While capturing snapshots has some known inefficiencies, materialization should be snappy.

[jsirois]

All the examples I found snapshotted 1-handful of files. This was snapshotting a loose pex like we build in pants for tools which had 5970 files. I suspected this may have had something to do with it but did not dig.