Hook up GOPATH to named_caches?

[Eric-Arellano]

Downloading third-party dependencies is proving really slow with Go. Over two minutes for https://github.com/kubernetes-sigs/external-dns

Our current implementation of third-party packages does not preserve the benefits of downloads across Pants runs. While we cache a particular resolution, if you make one change to go.mod or go.sum, you have to redo the entire download without leveraging already downloaded modules from earlier.

I don't know how to fix this because we probably want to use named_caches, but we do need to populate the chroot with the contents. When populating the chroot with contents, we also would like to symlink a la #12716.

[Eric-Arellano]

but we do need to populate the chroot with the contents

More concretely, we have two uses cases for this:

• Third-party package code needs to extract out the relevant subset so that we can compile the code later:

pants/src/python/pants/backend/go/util_rules/third_party_pkg.py

Lines 185 to 197 in 184d51e

	all_digest_subset_gets.append(
	Get(
	Digest,
	DigestSubset(
	stripped_result_digest,
	PathGlobs(
	[os.path.join(dir_path, "*")],
	glob_match_error_behavior=GlobMatchErrorBehavior.error,
	description_of_origin=f"downloading {import_path}",
	),
	),
	)
	)

• First-party package analysis, which needs to set up $GOPATH but doesn't actually directly operate on the files
  

  pants/src/python/pants/backend/go/util_rules/first_party_pkg.py

  Lines 91 to 96 in 184d51e

  	input_digest = await Get(
  	Digest,
  	MergeDigests(
  	[pkg_sources.snapshot.digest, go_mod_info.digest, all_third_party_packages.digest]
  	),
  	)