You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The trace reveals that it is taking 9 seconds to set up the chroot for determining FirstPartyPkgInfo, which is almost certainly because it requires copying in the GOPATH (all downloaded modules) so that go list does not complain about missing third-party packages:
On my M1, my machine locks up when trying to run dependencies :: because it results in too much contention from copying GOPATH once per each package. That is, each first-party package (directory) suffers from this problem.
Solution 1: our own analysis
Rather than using go list, we can write our own parser to compute this info:
Although we will definitely be leaning in to improve sandbox creation (likely via immutable caches or FUSE), in this case, option 1 seems like the better choice. As you mentioned: it's not strictly necessary to download the thirdparty code to extract imports.
A trace of running
./pants_from_sources --no-process-execution-local-cache dependencies //:root#./
in https://github.com/toolchainlabs/external-dns: pants_run_2021_10_28_14_52_47_122_d2ce1895125d4e4b8519c7445e94e4d0-trace.json.txtThe trace reveals that it is taking 9 seconds to set up the chroot for determining
FirstPartyPkgInfo
, which is almost certainly because it requires copying in theGOPATH
(all downloaded modules) so thatgo list
does not complain about missing third-party packages:pants/src/python/pants/backend/go/util_rules/first_party_pkg.py
Lines 91 to 96 in 9a4fd98
On my M1, my machine locks up when trying to run
dependencies ::
because it results in too much contention from copying GOPATH once per each package. That is, each first-party package (directory) suffers from this problem.Solution 1: our own analysis
Rather than using
go list
, we can write our own parser to compute this info:pants/src/python/pants/backend/go/util_rules/first_party_pkg.py
Lines 32 to 53 in 9a4fd98
Unlike
go list
, our parser would not require third-party deps to be present for analysis. We only need the package's source files.Over time, it would need to get more complex to handle metadata like this:
This approach is what
rules_go
does.Solution 2: make sandbox creation faster
For example, use #12716. Probably relates to #13390.
The text was updated successfully, but these errors were encountered: