Fix a storage leak in parachains db #5594

eskimor · 2022-05-25T10:50:47Z

This PR makes sure old votes are pruned from the database in dispute-coordinator.

Should be tested on Versi and with paritydb.

Tested on Rococo - disk usage seems to go down.
Tested on paritydb: Nodes are crashing, but things are working after a db reset:

 WARN tokio-runtime-worker sc_service::builder: The NetworkStart returned as part of `build_network` has been silently dropped    
Error: 
   0: InvalidConfiguration("Column config mismatch for column 4. Expected \"preimage: false, uniform: false, refc: false, compression: 0, ordered: true\", got \"preimage: false, uniform: false, refc: false, compression: 0, ordered: false\"")

burnin on Kusama and check for saved storage space.

Release Notes

Operators of nodes running paritydb will need to delete the parachains database before upgrading.

node/core/dispute-coordinator/src/db/v1.rs

sandreim

LGTM

node/core/dispute-coordinator/src/db/v1.rs

sandreim · 2022-05-31T10:29:38Z

node/core/dispute-coordinator/src/db/v1.rs

+#[cfg(test)]
+const MAX_CLEAN_BATCH_SIZE: u32 = 10;
+#[cfg(not(test))]
+const MAX_CLEAN_BATCH_SIZE: u32 = 300;


What is the impact of doing this many items on each earliest session update? (for nodes who have a lot of dangling storage items to clean)

From what I have seen so far, it seems to be pretty fast - although that might have been mostly empty sessions. Worst thing that could happen is that a valdiator is heavily loaded at a session boundary and fails to do some work. I also don't have any good data yet about how much wasted storage we are actually talking about, if it is tiny we can go with smaller batch sizes as then it does not matter if it takes forever.

burnin on Kusama will tell.

ordian

How did burn-in on Kusama go? What are the expected savings?

node/core/dispute-coordinator/src/db/v1.rs

Co-authored-by: Andronik <write@reusable.software>

eskimor · 2022-06-04T08:04:45Z

burnin on Kusama was not done yet. On Rococo I don't have definite data, but it looks like around 10%. I will add a metric how long cleanup rounds take and will then burnin on Kusama.

…o rk-fix-dispute-storage-leak

eskimor · 2022-06-13T13:39:29Z

burnin looks fine so far. Given that cleanup takes a while, let's include this sooner than later.

* Fix cleanup of old votes. * Cleanup. * Get rid of redundant import * Tests + logging * Fix db key name. * Add some reasoning to batch size. * Add dispute data to indexed columns * Fix fmt * Add helper function. * Fix typos. * Update node/core/dispute-coordinator/src/db/v1.rs Co-authored-by: Andronik <write@reusable.software> * Update node/core/dispute-coordinator/src/db/v1.rs Co-authored-by: Andronik <write@reusable.software> * Add metric for how long cleanup takes. Co-authored-by: Andronik <write@reusable.software>

Generic-Chain · 2022-07-18T17:48:18Z

this has not been resolved, since the release of v0.9.25 (and after a full cleanup of parachains/db folder) it has gradually increased with .sst files left behind since Jul 6 every day. At the moment on a validator that is active it has gotten to 376 .sst files total and 23GB in size of the parachains/db folder.

du -hs .local/share/polkadot/chains/ksmcc3/db/full/parachains/db/
24G .local/share/polkadot/chains/ksmcc3/db/full/parachains/db/

ls -ltr .local/share/polkadot/chains/ksmcc3/db/full/parachains/db/*.sst | wc -l
376

eskimor · 2022-07-22T14:56:48Z

Thanks @Generic-Chain ! Indeed there exists another leak, @vstakhov already identified that one as well and fixed it.

vstakhov · 2022-07-22T15:22:29Z

It is not very likely that the leak I have fixed could lead to 23Gb of storage waste. One thing we could try is to examine the current database to figure out what's happening.

Generic-Chain · 2022-07-26T16:41:24Z

I'm just reporting what I am seeing, I'm not familiar with what exactly was fixed and if there's some other storage leak unrelated to this one - the files are there since the cleanup and upgrade and the size and count keeps increasing

8 days after my post is up to 40GB and 576 .sst files (so +16GB and +200 .sst files)

du -hs .local/share/polkadot/chains/ksmcc3/db/full/parachains/db/
40G .local/share/polkadot/chains/ksmcc3/db/full/parachains/db/

ls -ltr .local/share/polkadot/chains/ksmcc3/db/full/parachains/db/*.sst | wc -l
576

anyone with an active validator can confirm this - I have 2 and both are having the same issue, 2nd one has 40GB and 545 files in the parachains/db folder

Fix cleanup of old votes.

7601247

github-actions bot added the A3-in_progress Pull request is in progress. No review needed at this stage. label May 25, 2022

eskimor added 3 commits May 25, 2022 12:51

Cleanup.

834f308

Get rid of redundant import

a2b43ee

Tests + logging

e31ecbd

eskimor requested a review from sandreim May 25, 2022 13:57

Fix db key name.

85ced82

eskimor marked this pull request as ready for review May 25, 2022 14:07

Add some reasoning to batch size.

b6b0b37

ordian reviewed May 25, 2022

View reviewed changes

node/core/dispute-coordinator/src/db/v1.rs Show resolved Hide resolved

eskimor added 2 commits May 25, 2022 21:43

Add dispute data to indexed columns

2fccece

Fix fmt

c1415da

sandreim approved these changes May 31, 2022

View reviewed changes

eskimor added 3 commits June 3, 2022 16:03

Merge branch 'master' into rk-fix-dispute-storage-leak

37f8a19

Add helper function.

07b0431

Fix typos.

a83b172

eskimor added the B1-releasenotes label Jun 3, 2022

eskimor changed the title ~~Fix cleanup of old votes.~~ Fix a storage leak in parachains db Jun 3, 2022

ordian approved these changes Jun 3, 2022

View reviewed changes

node/core/dispute-coordinator/src/db/v1.rs Outdated Show resolved Hide resolved

node/core/dispute-coordinator/src/db/v1.rs Outdated Show resolved Hide resolved

eskimor and others added 2 commits June 4, 2022 10:00

Update node/core/dispute-coordinator/src/db/v1.rs

5513608

Co-authored-by: Andronik <write@reusable.software>

Update node/core/dispute-coordinator/src/db/v1.rs

1253ab1

Co-authored-by: Andronik <write@reusable.software>

eskimor added 2 commits June 6, 2022 23:20

Add metric for how long cleanup takes.

dfcc91b

Merge remote-tracking branch 'origin/rk-fix-dispute-storage-leak' int…

7077ddb

…o rk-fix-dispute-storage-leak

eskimor removed the B0-silent Changes should not be mentioned in any release notes label Jun 8, 2022

eskimor merged commit 3e4fe06 into master Jun 13, 2022

eskimor deleted the rk-fix-dispute-storage-leak branch June 13, 2022 13:38

librelois mentioned this pull request Jul 19, 2022

Update substrate/polkadot/cumulus from v0.9.23 to v0.9.26 moonbeam-foundation/moonbeam#1691

Closed

sandreim mentioned this pull request Jul 20, 2022

Parachains db column "migration" #5797

Merged

ghzlatarev mentioned this pull request Jul 29, 2022

Update deps from v0.9.22 to v0.9.26 Manta-Network/Manta#720

Merged

3 tasks

eskimor mentioned this pull request Nov 11, 2022

Member request polkadot-fellows/seeding#57

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix a storage leak in parachains db #5594

Fix a storage leak in parachains db #5594

eskimor commented May 25, 2022 •

edited

Loading

sandreim left a comment

sandreim May 31, 2022

eskimor May 31, 2022

eskimor May 31, 2022

ordian left a comment

eskimor commented Jun 4, 2022

eskimor commented Jun 13, 2022

Generic-Chain commented Jul 18, 2022 •

edited

Loading

eskimor commented Jul 22, 2022 •

edited

Loading

vstakhov commented Jul 22, 2022

Generic-Chain commented Jul 26, 2022

Fix a storage leak in parachains db #5594

Fix a storage leak in parachains db #5594

Conversation

eskimor commented May 25, 2022 • edited Loading

Release Notes

sandreim left a comment

Choose a reason for hiding this comment

sandreim May 31, 2022

Choose a reason for hiding this comment

eskimor May 31, 2022

Choose a reason for hiding this comment

eskimor May 31, 2022

Choose a reason for hiding this comment

ordian left a comment

Choose a reason for hiding this comment

eskimor commented Jun 4, 2022

eskimor commented Jun 13, 2022

Generic-Chain commented Jul 18, 2022 • edited Loading

eskimor commented Jul 22, 2022 • edited Loading

vstakhov commented Jul 22, 2022

Generic-Chain commented Jul 26, 2022

eskimor commented May 25, 2022 •

edited

Loading

Generic-Chain commented Jul 18, 2022 •

edited

Loading

eskimor commented Jul 22, 2022 •

edited

Loading