feat: daemon: automatically set GOMEMLIMIT if it is unset #9451
Conversation
Jorropo
force-pushed
the
magic-soft-memory-limit
branch
4 times, most recently
from
d177aeb
to
4c0bec8
Compare
|
Would it perhaps make sense to default this to something (based on) the resource manager's limit? https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgrmaxmemory |
Jorropo
force-pushed
the
magic-soft-memory-limit
branch
from
4c0bec8
to
6b3e242
Compare
There was a problem hiding this comment.
Added some minor comments.
Shall we add some documentation about GOMEMLIMIT and how it is being used on kubo here: https://github.com/ipfs/kubo/blob/master/docs/environment-variables.md ?
|
@Jorropo : I like the idea of this but in practice is this actually going to help with resource manager complaints since the go-libp2p resource manager does its own state tracking for resource usage? (I agree it will help Kubo less system resources.) If it's not going to help with the errors/alarming, I suggest we push this out to next release since we're already trying to squeeze a lot in between now and 2022-12-08 and would rather divert our attention to other items for 0.18. |
|
@BigLep this does not change anything to the error logging, It helps with peoples reporting OOMs. |
Also, ideally, significantly less GC disruptions on high-use nodes. |
|
2023-01-31: we pushed to next iteration because of concerns about getting caught in CPU death spiral and not actually dying. This requires extra discussion and we don't have bandwidth for it now. |
Jorropo
force-pushed
the
magic-soft-memory-limit
branch
2 times, most recently
from
63c4002
to
a3fffe5
Compare
Jorropo
force-pushed
the
magic-soft-memory-limit
branch
from
a3fffe5
to
4d9489a
Compare
Jorropo
force-pushed
the
magic-soft-memory-limit
branch
from
4d9489a
to
819d8e3
Compare
|
❤️ |
|
I checked the CPU death spiral was some other bug. |
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
force-pushed
the
magic-soft-memory-limit
branch
from
819d8e3
to
8011af2
Compare
|
@Jorropo : what are the next steps here? Is this realistically going to get merged the next month? |
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set.
This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories.
Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off.
Closes: #8798