Sandbox creation is prohibitively slow for Go first-party package analysis

[Eric-Arellano]

A trace of running ./pants_from_sources --no-process-execution-local-cache dependencies //:root#./ in https://github.com/toolchainlabs/external-dns: pants_run_2021_10_28_14_52_47_122_d2ce1895125d4e4b8519c7445e94e4d0-trace.json.txt

The trace reveals that it is taking 9 seconds to set up the chroot for determining FirstPartyPkgInfo, which is almost certainly because it requires copying in the GOPATH (all downloaded modules) so that go list does not complain about missing third-party packages:

pants/src/python/pants/backend/go/util_rules/first_party_pkg.py

Lines 91 to 96 in 9a4fd98

	input_digest = await Get(
	Digest,
	MergeDigests(
	[pkg_sources.snapshot.digest, go_mod_info.digest, all_third_party_packages.digest]
	),
	)

On my M1, my machine locks up when trying to run dependencies :: because it results in too much contention from copying GOPATH once per each package. That is, each first-party package (directory) suffers from this problem.

Solution 1: our own analysis

Rather than using go list, we can write our own parser to compute this info:

pants/src/python/pants/backend/go/util_rules/first_party_pkg.py

Lines 32 to 53 in 9a4fd98

	@dataclass(frozen=True)
	class FirstPartyPkgInfo:
	"""All the info and digest needed to build a first-party Go package.
	The digest does not strip its source files. You must set `working_dir` appropriately to use the
	`go_first_party_package` target's `subpath` field.
	"""
	digest: Digest
	subpath: str
	import_path: str
	imports: tuple[str, ...]
	test_imports: tuple[str, ...]
	xtest_imports: tuple[str, ...]
	go_files: tuple[str, ...]
	test_files: tuple[str, ...]
	xtest_files: tuple[str, ...]
	s_files: tuple[str, ...]

Unlike go list, our parser would not require third-party deps to be present for analysis. We only need the package's source files.

Over time, it would need to get more complex to handle metadata like this:

        // Cgo directives
        CgoCFLAGS    []string // cgo: flags for C compiler
        CgoCPPFLAGS  []string // cgo: flags for C preprocessor
        CgoCXXFLAGS  []string // cgo: flags for C++ compiler
        CgoFFLAGS    []string // cgo: flags for Fortran compiler
        CgoLDFLAGS   []string // cgo: flags for linker
        CgoPkgConfig []string // cgo: pkg-config names

This approach is what rules_go does.

Solution 2: make sandbox creation faster

For example, use #12716. Probably relates to #13390.

[stuhood]

Although we will definitely be leaning in to improve sandbox creation (likely via immutable caches or FUSE), in this case, option 1 seems like the better choice. As you mentioned: it's not strictly necessary to download the thirdparty code to extract imports.

[stuhood]

Resolved in #13476 (although #12716 will almost certainly be important for the JVM).