Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(storage, mdbx): set rp augment limit #5614

Merged
merged 3 commits into from
Nov 29, 2023
Merged

perf(storage, mdbx): set rp augment limit #5614

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fd245c3
fix(storage, mdbx): set rp augment limit
shekhirin Nov 28, 2023
f6be42d
add references to mdbx.c (oh. my. god.)
shekhirin Nov 28, 2023
a3eaa03
reference old value
shekhirin Nov 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions crates/storage/db/src/implementation/mdbx/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,32 @@ impl DatabaseEnv {
});
// configure more readers
inner_env.set_max_readers(DEFAULT_MAX_READERS);
// This parameter sets the maximum size of the "reclaimed list", and the unit of measurement
// is "pages". Reclaimed list is the list of freed pages that's populated during the
// lifetime of DB transaction, and through which MDBX searches when it needs to insert new
// record with overflow pages. The flow is roughly the following:
// 0. We need to insert a record that requires N number of overflow pages (in consecutive
// sequence inside the DB file).
// 1. Get some pages from the freelist, put them into the reclaimed list.
// 2. Search through the reclaimed list for the sequence of size N.
// 3. a. If found, return the sequence.
// 3. b. If not found, repeat steps 1-3. If the reclaimed list size is larger than
// the `rp augment limit`, stop the search and allocate new pages at the end of the file:
// https://github.com/paradigmxyz/reth/blob/2a4c78759178f66e30c8976ec5d243b53102fc9a/crates/storage/libmdbx-rs/mdbx-sys/libmdbx/mdbx.c#L11479-L11480.
//
// Basically, this parameter controls for how long do we search through the freelist before
// trying to allocate new pages. Smaller value will make MDBX to fallback to
// allocation faster, higher value will force MDBX to search through the freelist
// longer until the sequence of pages is found.
//
// The default value of this parameter is set depending on the DB size. The bigger the
// database, the larger is `rp augment limit`.
// https://github.com/paradigmxyz/reth/blob/2a4c78759178f66e30c8976ec5d243b53102fc9a/crates/storage/libmdbx-rs/mdbx-sys/libmdbx/mdbx.c#L10018-L10024.
//
// Previously, MDBX set this value as `256 * 1024` constant. Let's fallback to this,
// because we want to prioritize freelist lookup speed over database growth.
// https://github.com/paradigmxyz/reth/blob/fa2b9b685ed9787636d962f4366caf34a9186e66/crates/storage/libmdbx-rs/mdbx-sys/libmdbx/mdbx.c#L16017.
inner_env.set_rp_augment_limit(256 * 1024);

if let Some(log_level) = log_level {
// Levels higher than [LogLevel::Notice] require libmdbx built with `MDBX_DEBUG` option.
Expand Down