async backing: potential memory leak

[sandreim]

This has been observed during a load test on Versi @ 300 validators and 60 parachains with async backing enabled.

Configuration:

asyncBackingParams: {
    maxCandidateDepth: 3
    allowedAncestryLen: 2
  }

I suspect statement-distribution or prospective-parachains to be the culprit.

At the same time some nodes seem to stop running with last message: 2023-09-06 12:13:34.377 ERROR tokio-runtime-worker polkadot_overseer: subsystem exited with error subsystem="statement-distribution-subsystem" err=FromOrigin { origin: "statement-distribution", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }

Possibly killed due to OOM, but still investigating.

[sandreim]

Nodes are indeed being evicted due to excessive mem usage, so the subsystem exited with error is explained.

[sandreim]

statement-distribution as a CPU usage champion.

[sandreim]

@rphmeier the new logs you pushed to #1410 show that we are tracking an ever increasing number of leaves.

https://grafana.teleport.parity.io/goto/Et7Z5lzIR?orgId=1

This explains both unbounded memory and CPU usage growth.

[rphmeier]

67d1bc0

[alexggh]

67d1bc0

@rphmeier This fix doesn't seem to have made it into master, I can still see high cpu when testing on versi and it seems that the same thing started happening on rococo once we updated to a runtime supporting async-backing.

Any idea if it wasn't included intentionally or by accident ?

[sandreim]

This seems to be an oversight, fix is part of #1436 which was not merged 🤦🏼

[sandreim]

stale fixed issue