Possibly outdated hardware requirements for running validator nodes #711
Comments
|
This is very likely some memory leak. Please give more information on your setup. Some metrics would also be nice, CPU usage, memory usage around the time it crashes. Please also post the args you are using to run your node. CC @koute |
|
My polkadot current version is 0.9.36. I am running node with systemd: The full typical log looks like this: It usually crashes at this moment - 22/100. When I observe whats happening with server resources through htop I see rising RAM use from around ~700MB that is has "fresh" up to the limit of ~32GB server has. The process killing the node is 'OOM killer'. Let me know what else can I provide in order to help with the issue? |
|
@sealbox Looks like you're running an RPC node and are getting RPC connections. Are those connections yours? Is this a public RPC server? |
|
RPC ports stays behind firewall and allows connection only from my own apps. |
|
Here's the log with RPC port disabled on firewall. Notice that thre are no connections now (because firewall is not allowing them), but the node still crashes the same way: |
|
@sealbox why are you running with |
|
This means that you are basically loading 432000 blocks into memory. This is the reason why your node is getting killed directly after startup. If you really need to run with such a huge pruning window, you should use |
|
paritytech/substrate#13414 this should print some warning in the future to make the user aware. |
claravanstaden
pushed a commit
to Snowfork/polkadot-sdk
that referenced
this issue
Dec 8, 2023
Co-authored-by: David Dunn <26876072+doubledup@users.noreply.github.com> Co-authored-by: Alistair Singh <alistair.singh7@gmail.com>
helin6
pushed a commit
to boolnetwork/polkadot-sdk
that referenced
this issue
Feb 5, 2024
* Move open_frontier_backend to fc_db Signed-off-by: koushiro <koushiro.cqx@gmail.com> * Some nits Signed-off-by: koushiro <koushiro.cqx@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The webpage for Polkadot project states that in order to run validator nodes it is required to have 16 GB RAM. I have been running validator node for more than a year (and updating regularly - currently running v 0.9.36) on dedicated server with 32 GB RAM. Outside of the Polkadot nothing else is running there. Lately my node is crashing all the time. What happens is the node starts, the RAM usage is rising, rising and rising up to ~32GB RAM which is server hardware limit then the node crashes by being killed by kernel:
This repeats every few minutes. Used to work fine before.
My question here is - is there possibility to still run node with 32GB server? Are there any specific settings that I should look into? Or is the documentation outdated and it should state the required value is 64GB currently not, 16GB, and I should do hardware update?
The text was updated successfully, but these errors were encountered: