Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go: determine cause(s) of Go plugin performance issues #12719

Closed
tdyas opened this issue Sep 1, 2021 · 9 comments
Closed

go: determine cause(s) of Go plugin performance issues #12719

tdyas opened this issue Sep 1, 2021 · 9 comments
Labels
backend: Go Go backend-related issues

Comments

@tdyas
Copy link
Contributor

tdyas commented Sep 1, 2021

When building casload in https://github.com/toolchainlabs/remote-api-tools, a run of ./pants package cmd/casload:bin with an empty cache takes ~11 minutes.

Theories for the performance issue:

  1. Analysis for third-party modules is re-downloading all dependent modules since the Go module cache is essentially empty for each invocation of go list ./... to ascertain the packages within a module. I verified that downloads do occur by deleting my Go module cache and running go list -x ./... which displayed all of the download commands.
  2. Unpacking GOROOT for each Process invoking the go tool takes lots of time in aggregate. Needs to be confirmed (maybe via adding a work-unit).
@tdyas tdyas added the backend: Go Go backend-related issues label Sep 1, 2021
@Eric-Arellano
Copy link
Contributor

It would be neat to have metrics for how expensive chroot setup is in general.

@tdyas
Copy link
Contributor Author

tdyas commented Sep 1, 2021

It would be neat to have metrics for how expensive chroot setup is in general.

Looking into it. I want a work-unit for sandbox setup activities.

@tdyas
Copy link
Contributor Author

tdyas commented Sep 1, 2021

#12721 adds a workunit to capture input root materialization time.

@tdyas
Copy link
Contributor Author

tdyas commented Sep 2, 2021

I did a build of Go code in remote-api-tools with an empty cache and captured a trace: https://drive.google.com/file/d/1sV7WMuO1UikQ_byzoZG40_XoR9h7K80Q/view?usp=sharing

Materializing the input root is taking an inordinate amount of time. For example, for a Resolving _go_ext_mod_package_metadata, 37.437s of 52.215s (~71%) was spent for sandbox setup. In another case for same type of task, 35.390s of 57.911s (61%) was spent for sandbox setup. Unclear if the time was just extraction from the archive or was contention on my laptop's SSD with so many tasks trying to extract at once. Either case is an issue that needs to be fixed.

@stuhood
Copy link
Member

stuhood commented Sep 3, 2021

71% and 61% of the time is a lot, but still leaves a lot of other time unaccounted for... if I did things correctly, a clean build of that binary directly using go took two orders of magnitude less time to complete.

It looks like the $GOCACHE is being placed inside of the sandbox: assuming that it is trustworthy (it defaults to a global location, so it most likely is) that should probably be moved into a append_only_cache instead. Example:

pex_root = PurePath(".cache") / self._PEX_ROOT_DIRNAME
return CompletePexEnvironment(
_pex_environment=self,
pex_root=pex_root,
_working_directory=PurePath(working_directory) if working_directory else None,
append_only_caches=FrozenDict({self._PEX_ROOT_DIRNAME: str(pex_root)}),
(...which uses a cache named pex_root, and asks for a symlink at .cache/pex_root).

@tdyas
Copy link
Contributor Author

tdyas commented Sep 3, 2021

It looks like the $GOCACHE is being placed inside of the sandbox: assuming that it is trustworthy (it defaults to a global location, so it most likely is) that should probably be moved into a append_only_cache instead.

I believe the other time is accounted for by go re-downloading third party dependencies (as mentioned in this issue's first comment). I will attempt to verify that by making the third party deps available in the input root.

The GOCACHE should not be of much value since we store the package archives in the CAS. The plugin invokes go tool compile and go tool link directory, so I do not believe GOCACHE will even be used for those invocations.

@stuhood
Copy link
Member

stuhood commented Sep 3, 2021

It looks like the $GOCACHE is being placed inside of the sandbox: assuming that it is trustworthy (it defaults to a global location, so it most likely is) that should probably be moved into a append_only_cache instead.

I believe the other time is accounted for by go re-downloading third party dependencies (as mentioned in this issue's first comment). I will attempt to verify that by making the third party deps available in the input root.

I'm a bit confused: is the GOCACHE different from the "go module cache" that you mention in that first comment?

@tdyas
Copy link
Contributor Author

tdyas commented Sep 3, 2021

I'm a bit confused: is the GOCACHE different from the "go module cache" that you mention in that first comment?

Yes. GOCACHE is the "build cache" for built artifacts (i.e., object files, package archives, etc.), while the "module cache" is the downloaded copies of third-party modules generally stored in the GOPATH (and not the GOCACHE).

@Eric-Arellano
Copy link
Contributor

Closing in favor of #12772 and #12771. We can open a new issue if performance continues to be a concern after both are fixed.

tdyas pushed a commit that referenced this issue Oct 4, 2021
Closes #12771.

This does not actually improve cache invalidation until we figure out #13093. But it does dramatically reduce the size of each sandbox, which #12719 identified is causing substantial slowdowns to set up.

[ci skip-rust]
[ci skip-build-wheels]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend: Go Go backend-related issues
Projects
None yet
Development

No branches or pull requests

3 participants