Test and document use and performance of Go 1.19's GC memory limit

[dokterbob]

Checklist

• My issue is specific & actionable.
• I am not suggesting a protocol enhancement.
• I have searched on the issue tracker for my issue.

Description

In addition to the upcoming Resource Manager, as of version 1.19, Golang has a built-in soft memory limit option, which tunes how the garbage collector operates within memory-constrained environments. This option can be set either using the GOMEMLIMIT environment varibale or by using SetMemoryLimit in the runtime/debug package.

The current default configuration of kubo under high load tends to eat all the memory it can gets until it gets until killed by the OOM, causing frequent restarts of nodes and loss of connections and service.

The new resource manager should be able to address some of these issues. However, documenting and testing Go's native soft memory limits could still significantly improve performance and/or memory usage, particularly under memory-constrained environments:

1. In addition to the resource manager's decisions on runtime usage, the limit-aware GC would collaborate to have more memory available for the kubo runtime.
2. Until the resource manager is enabled by default, it could already increase stability and performance.

Note that 2. is written as could. Hence, my request is as follows:

1. Perform manual tests (with and/or without resource manager) with Golang's soft memory limit set and evaluate results.
2. Document results as recommendations for production-scale IPFS deployments.
3. If aforementioned recommendation is positive, add automated testing.
4. Possibly, call SetMemoryLimit when a memory limit is configured in the resource manager.

[dokterbob]

Related: #8936 #8798
Golang's GC guide, section 'Memory limit': https://go.dev/doc/gc-guide

[Jorropo]

AFAIT go's soft memory limit will just run GC no matter what when you reach the limit.

GC more often cost more CPU cycles "wasted".
I don't really see what you want to test.

[fanhai]

The current default configuration of kubo under high load tends to eat all the memory it can gets until it gets until killed by the OOM, causing frequent restarts of nodes and loss of connections and service.

GOMEMLIMIT

Does it work?

[dokterbob]

My question exactly. Might be able to test over the coming weeks.

[dokterbob]

AFAIT go's soft memory limit will just run GC no matter what when you reach the limit.

GC more often cost more CPU cycles "wasted". I don't really see what you want to test.

Whether in production environments (so not in theory)GOMEMLIMIT affects stability or performance. Specifically, whether it reduces OOM kills, whether it leads to more efficient memory use and better scheduling of GC, as the docs suggest it might.

Specifically, what I am hoping for is that Go will GC in time as to prevent OOM kills due to lingering garbage. Some more background: https://weaviate.io/blog/2022/08/GOMEMLIMIT-a-Game-Changer-for-High-Memory-Applications.html

[Jorropo]

Well I've tested, it works very well, you can't magically half kubo's memory usage because as you get closer to GOMEMLIMIT the GC will run exponentially more often until it just run continously and performance is hot garbage.

I've implemented a conservative limit that should still help there: #9451

[dokterbob]

@Jorropo This is awesome! I added the suggestion of maybe setting it based on the resource manager's memory limit.