Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pantsd memory leak #9999

Closed
asherf opened this issue Jun 8, 2020 · 4 comments
Closed

pantsd memory leak #9999

asherf opened this issue Jun 8, 2020 · 4 comments
Assignees
@stuhood

Comments

@asherf
Copy link
Member

asherf commented Jun 8, 2020 

Jun  8 03:12:27 ip-10-1-201-194 kernel: [3411370.047948] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name                                                                                             [19/257]
Jun  8 03:12:27 ip-10-1-201-194 kernel: [3411370.048020] [14990] 34179 14990 462185213   495449 11333632        0             0 pantsd [/data/h
Jun  8 03:12:27 ip-10-1-201-194 kernel: [3411370.048022] [  807] 34179   807 461979768   303115  9003008        0             0 pantsd [/data/h


.....

Jun  8 03:12:27 ip-10-1-201-194 kernel: [3411370.048064] [10873]     0 10873     9054       80   114688        0             0 cron
Jun  8 03:12:27 ip-10-1-201-194 kernel: [3411370.048066] [10881]     0 10881     6305     1252    77824        0          -900 snapd
Jun  8 03:12:27 ip-10-1-201-194 kernel: [3411370.048067] [10882]     0 10882    10796      265   114688        0             0 systemd-udevd
Jun  8 03:12:27 ip-10-1-201-194 kernel: [3411370.048068] Out of memory: Kill process 14990 (pantsd [/data/h) score 525 or sacrifice child
Jun  8 03:12:27 ip-10-1-201-194 kernel: [3411370.052392] Killed process 14990 (pantsd [/data/h) total-vm:1848740852kB, anon-rss:1981796kB, file-rss:0kB, shmem-rss:0kB
Jun  8 03:12:27 ip-10-1-201-194 kernel: [3411370.217615] oom_reaper: reaped process 14990 (pantsd [/data/h), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
@asherf
Copy link
Member Author

asherf commented Jun 8, 2020 •
edited
Loading

@stuhood @gshuflin

@stuhood stuhood added the pantsd label Jun 8, 2020
@stuhood
Copy link
Member

stuhood commented Jun 8, 2020 •
edited
Loading

It is not surprising that pantsd leaks memory... we expect about 10-15% per run. But given that, we should probably have pantsd checking its own memory usage and shutting down if it goes above a limit.

Thanks for the report!

@stuhood
Copy link
Member

stuhood commented Jun 8, 2020

Related: #7675 and #8193.

@stuhood stuhood self-assigned this Jun 8, 2020
@stuhood stuhood mentioned this issue Jun 8, 2020
Merged
stuhood added a commit that referenced this issue Jun 9, 2020
@stuhood
Add a configurable cap on total pantsd memory usage. (#10003)
fa53e8e 
### Problem

`pantsd` does not implement garbage collection of the `Graph` (see #7675), but additionally, there are likely a few Python-level reference cycles beyond those that we have already discovered.

### Solution

We will eventually implement #7675, and it will need a config value to control how much it collects. But in the meantime, having a configurable built-in cap on total memory usage is generally useful, and can consume the same flag that our eventual collection will.

### Result

`pantsd` will restart itself when it uses more than the configured amount of memory (defaulting to 4GB). As mentioned in the test comment, until #8200 is fixed, the message rendered when we restart will not be particularly friendly, so we should likely not cherry-pick this to 1.29.x, which will not receive #8200.

This is not a complete fix for #9999, but I'm going to resolve it in favor of tracking followup in #7675.

[ci skip-rust-tests]
[ci skip-jvm-tests]
@stuhood
Copy link
Member

stuhood commented Jun 9, 2020

Although #10003 was not a complete fix for this, I'm going to resolve it in favor of tracking followup in #7675.

@stuhood stuhood closed this as completed Jun 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stuhood stuhood

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stuhood @asherf