perf(storage, mdbx): set rp augment limit

[shekhirin]

Resolves #5228

Explanation

reth/crates/storage/db/src/implementation/mdbx/mod.rs

Lines 102 to 127 in a3eaa03

	// This parameter sets the maximum size of the "reclaimed list", and the unit of measurement
	// is "pages". Reclaimed list is the list of freed pages that's populated during the
	// lifetime of DB transaction, and through which MDBX searches when it needs to insert new
	// record with overflow pages. The flow is roughly the following:
	// 0. We need to insert a record that requires N number of overflow pages (in consecutive
	// sequence inside the DB file).
	// 1. Get some pages from the freelist, put them into the reclaimed list.
	// 2. Search through the reclaimed list for the sequence of size N.
	// 3. a. If found, return the sequence.
	// 3. b. If not found, repeat steps 1-3. If the reclaimed list size is larger than
	// the `rp augment limit`, stop the search and allocate new pages at the end of the file:
	// https://github.com/paradigmxyz/reth/blob/2a4c78759178f66e30c8976ec5d243b53102fc9a/crates/storage/libmdbx-rs/mdbx-sys/libmdbx/mdbx.c#L11479-L11480.
	//
	// Basically, this parameter controls for how long do we search through the freelist before
	// trying to allocate new pages. Smaller value will make MDBX to fallback to
	// allocation faster, higher value will force MDBX to search through the freelist
	// longer until the sequence of pages is found.
	//
	// The default value of this parameter is set depending on the DB size. The bigger the
	// database, the larger is `rp augment limit`.
	// https://github.com/paradigmxyz/reth/blob/2a4c78759178f66e30c8976ec5d243b53102fc9a/crates/storage/libmdbx-rs/mdbx-sys/libmdbx/mdbx.c#L10018-L10024.
	//
	// Previously, MDBX set this value as `256 * 1024` constant. Let's fallback to this,
	// because we want to prioritize freelist lookup speed over database growth.
	// https://github.com/paradigmxyz/reth/blob/fa2b9b685ed9787636d962f4366caf34a9186e66/crates/storage/libmdbx-rs/mdbx-sys/libmdbx/mdbx.c#L16017.
	inner_env.set_rp_augment_limit(256 * 1024);

Results

Before 12:10 – default rp augment limit value (which is around 137 million)
After 12:25 – custom rp augment limit value set to 256 * 1024

Note: disregard the drop of the Freelist chart, it's due to node having a slight downtime during the restart and many blocks being inserted at once.

[rkrasiuk]

omfg

[mattsse]

hehe

[joshieDo]

send it

[gakonst]

1 loc change, 25 comments, 1 month of effort, gg

this does mean the db will grow faster on heavy freelist-related scenarios?

[shekhirin]

this does mean the db will grow faster on heavy freelist-related scenarios?

Yes, but this effect should be small, because only large insertions will lead to new allocations.