go: determine cause(s) of Go plugin performance issues

[tdyas]

When building casload in https://github.com/toolchainlabs/remote-api-tools, a run of ./pants package cmd/casload:bin with an empty cache takes ~11 minutes.

Theories for the performance issue:

1. Analysis for third-party modules is re-downloading all dependent modules since the Go module cache is essentially empty for each invocation of go list ./... to ascertain the packages within a module. I verified that downloads do occur by deleting my Go module cache and running go list -x ./... which displayed all of the download commands.
2. Unpacking GOROOT for each Process invoking the go tool takes lots of time in aggregate. Needs to be confirmed (maybe via adding a work-unit).

[Eric-Arellano]

It would be neat to have metrics for how expensive chroot setup is in general.

[tdyas]

It would be neat to have metrics for how expensive chroot setup is in general.

Looking into it. I want a work-unit for sandbox setup activities.

[tdyas]

#12721 adds a workunit to capture input root materialization time.

[tdyas]

I did a build of Go code in remote-api-tools with an empty cache and captured a trace: https://drive.google.com/file/d/1sV7WMuO1UikQ_byzoZG40_XoR9h7K80Q/view?usp=sharing

Materializing the input root is taking an inordinate amount of time. For example, for a Resolving _go_ext_mod_package_metadata, 37.437s of 52.215s (~71%) was spent for sandbox setup. In another case for same type of task, 35.390s of 57.911s (61%) was spent for sandbox setup. Unclear if the time was just extraction from the archive or was contention on my laptop's SSD with so many tasks trying to extract at once. Either case is an issue that needs to be fixed.

[stuhood]

71% and 61% of the time is a lot, but still leaves a lot of other time unaccounted for... if I did things correctly, a clean build of that binary directly using go took two orders of magnitude less time to complete.

It looks like the $GOCACHE is being placed inside of the sandbox: assuming that it is trustworthy (it defaults to a global location, so it most likely is) that should probably be moved into a append_only_cache instead. Example:

pants/src/python/pants/backend/python/util_rules/pex_environment.py

Lines 129 to 134 in ae647d1

	pex_root = PurePath(".cache") / self._PEX_ROOT_DIRNAME
	return CompletePexEnvironment(
	_pex_environment=self,
	pex_root=pex_root,
	_working_directory=PurePath(working_directory) if working_directory else None,
	append_only_caches=FrozenDict({self._PEX_ROOT_DIRNAME: str(pex_root)}),

(...which uses a cache named pex_root, and asks for a symlink at .cache/pex_root).

[tdyas]

It looks like the $GOCACHE is being placed inside of the sandbox: assuming that it is trustworthy (it defaults to a global location, so it most likely is) that should probably be moved into a append_only_cache instead.

I believe the other time is accounted for by go re-downloading third party dependencies (as mentioned in this issue's first comment). I will attempt to verify that by making the third party deps available in the input root.

The GOCACHE should not be of much value since we store the package archives in the CAS. The plugin invokes go tool compile and go tool link directory, so I do not believe GOCACHE will even be used for those invocations.

[stuhood]

It looks like the $GOCACHE is being placed inside of the sandbox: assuming that it is trustworthy (it defaults to a global location, so it most likely is) that should probably be moved into a append_only_cache instead.

I believe the other time is accounted for by go re-downloading third party dependencies (as mentioned in this issue's first comment). I will attempt to verify that by making the third party deps available in the input root.

I'm a bit confused: is the GOCACHE different from the "go module cache" that you mention in that first comment?

[tdyas]

I'm a bit confused: is the GOCACHE different from the "go module cache" that you mention in that first comment?

Yes. GOCACHE is the "build cache" for built artifacts (i.e., object files, package archives, etc.), while the "module cache" is the downloaded copies of third-party modules generally stored in the GOPATH (and not the GOCACHE).

[Eric-Arellano]

Closing in favor of #12772 and #12771. We can open a new issue if performance continues to be a concern after both are fixed.