-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Garbage collect the v2 engine Graph #7675
Comments
### Problem `pantsd` does not implement garbage collection of the `Graph` (see #7675), but additionally, there are likely a few Python-level reference cycles beyond those that we have already discovered. ### Solution We will eventually implement #7675, and it will need a config value to control how much it collects. But in the meantime, having a configurable built-in cap on total memory usage is generally useful, and can consume the same flag that our eventual collection will. ### Result `pantsd` will restart itself when it uses more than the configured amount of memory (defaulting to 4GB). As mentioned in the test comment, until #8200 is fixed, the message rendered when we restart will not be particularly friendly, so we should likely not cherry-pick this to 1.29.x, which will not receive #8200. This is not a complete fix for #9999, but I'm going to resolve it in favor of tracking followup in #7675. [ci skip-rust-tests] [ci skip-jvm-tests]
### Problem The default max memory usage from #10003 was chosen with larger monorepos in mind, and didn't match the expectation of consumers in smaller repos. ### Solution Very large repos will have folks who are able/willing to adjust limits like this to optimize for their repo size, so adjust the default down in favor of having good defaults. This relates to #7675, but it isn't the time to dive in there. ### Result Fixes #10264. [ci skip-rust-tests]
One place in which we could cheaply start to do more garbage collection would be to convert the interning pants/src/rust/engine/src/interning.rs Lines 11 to 43 in 5aebe76
|
To accomplish garbage collection of
|
Is it worth going down this route? From my naive point of view it looks like you could do this as long as you could create a weakref to the object and implement a remove key callback on the interns struct. |
Okay Rust newb question: it looks like I can't do Also is there are particular reason you're suggesting age as the discriminant? Simplicity of implementation? Seems like access time might be a better discriminant long term, which could turn this into something resembling an LRU cache. But I guess that could be a follow-up |
When you run into cases like this early on, the answer will be to use
The idea behind using a |
Pushing this now to know if this is directionally or generally correct. This is an attempt at solving pantsbuild#7675. I'm aware there needs to be tests on the graphs, but I'm still trying to load the data structure semantics into my head to know how to construct a test. In the meanwhile, I'd like some feedback to see if this is going in the right direction or if I'm totally off base here. Also feel free to correct me or push me to better practices on my Rust.
Pushing this now to know if this is directionally or generally correct. This is an attempt at solving pantsbuild#7675. I'm aware there needs to be tests on the graphs, but I'm still trying to load the data structure semantics into my head to know how to construct a test. In the meanwhile, I'd like some feedback to see if this is going in the right direction or if I'm totally off base here. Also feel free to correct me or push me to better practices on my Rust.
Pushing this now to know if this is directionally or generally correct. This is an attempt at solving pantsbuild#7675. I'm aware there needs to be tests on the graphs, but I'm still trying to load the data structure semantics into my head to know how to construct a test. In the meanwhile, I'd like some feedback to see if this is going in the right direction or if I'm totally off base here. Also feel free to correct me or push me to better practices on my Rust.
Commented on #14676 (comment): actually allowing |
The v2
Graph
(implemented in rust) does not implement garbage collection, although it is definitely feasible.As we fix other issues and
pantsd
instances are able to stay up longer and longer, we should consider implementing garbage collection based on walking from recently requested roots in theSession
.The text was updated successfully, but these errors were encountered: