Add a more aggressive GC watchdog to help in bursty memory usage situations #8798
Labels
kind/enhancement
A net-new feature or improvement to an existing feature
P2
Medium: Good to have, but can wait until someone steps up
Comments
Jorropo
added
the
kind/enhancement
|
@Jorropo : cool, sounds good. We look forward to hearing what you learn from experimenting. |
3 tasks
3 tasks
|
Update: go implemented this somewhat recently however it's disabled by default. |
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 5, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Mar 30, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Mar 30, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
May 3, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
May 4, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Jun 30, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checklist
Description
In #8797 it happen often that bitswap use 20+Gigs of ram and will continue even after all requests has been canceled / finished.
However due to how golang GC's work, that lead to an accoutumance to high memory usage, and the GC wont get agressive to reclaim space.
(note the way the GC works, is that the GC tries to run when it is a factor X above the last GC size, so if your last size was 20Gig, golang might as well wait for 40gig of memory usage before gcing)
I belive lotus use https://github.com/raulk/go-watchdog to help in memory bursty conditions, maybe we should try it ? (I'm creating the issue to log any experiments)
The text was updated successfully, but these errors were encountered: