OOM Westmint

[Maharacha]

Version: 0.9.400
Chain: Westmint
Parameters: --chain westmint --pruning archive --ws-external --ws-port 9966 --ws-max-connections 1000 --rpc-external --rpc-port 9933 --rpc-cors all --rpc-methods Safe --prometheus-external

From resync of both Relay and Para DBs, this is the memory consumtion, currently having 64GB available memory.

Log from start to first OOM:
westmint_oom.log

[bkchr]

Did you compile the node yourself? What OS version are you using?

[bkchr]

Pinned block cache limit reached. Evicting value

This is really weird, looks maybe we accumulate notifications somewhere and thus it keeps all the stuff in memory and we die.

CC @skunert

[Maharacha]

Did you compile the node yourself? What OS version are you using?

Using binary from https://github.com/paritytech/cumulus/releases/download/v0.9.400/polkadot-parachain
on

Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy

[bkchr]

Ty! Yeah I think the caching issue is the reason you are running out of memory!

[skunert]

Interesting 🤔

Ty! Yeah I think the caching issue is the reason you are running out of memory!

At first glance, it does not sound like a caching issue but notification hoarding somewhere as you said in your first answer. The cache has a fixed maximum size, after all.

[Maharacha]

I enabled -ldebug on both relay and para if that helps. This is after a restart. It did not OOM so far.
https://drive.google.com/file/d/12mmUhBwxPn-QGdzSvFQN2aOG7jA-ILtN/view?usp=sharing

[skunert]

I have the strong suspicion that this is related to #2495, I will run some more experiments. Also have some ideas on how to fix.

[Maharacha]

It is in sync and has been working well for a few days now. Constantly on ~11G memory usage. But it was a roller coaster to get there 😄