-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(exex): write ahead log #10995
base: main
Are you sure you want to change the base?
feat(exex): write ahead log #10995
Conversation
acf5263
to
213bcea
Compare
87f42f2
to
6c2874a
Compare
6c2874a
to
b05bb87
Compare
2db5ebd
to
d80c196
Compare
99b0e65
to
fa17c65
Compare
4a16fd6
to
1b79e58
Compare
0cc1753
to
108981f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unsure I fully understood how the file and cache is kept in sync.
crates/exex/exex/src/wal/mod.rs
Outdated
/// Clears the block cache and fills it with the notifications from the [`Storage`], up to the | ||
/// given offset in bytes, not inclusive. | ||
#[instrument(target = "exex::wal", skip(self))] | ||
fn fill_block_cache(&mut self, to_offset: u64) -> eyre::Result<()> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is basically initialization?
/// The underlying WAL storage backed by a file. | ||
/// | ||
/// Each notification is written without any delimiters and structured as follows: | ||
/// ```text | ||
/// +--------------------------+----------------------------------+ | ||
/// | little endian u32 length | MessagePack-encoded notification | | ||
/// +--------------------------+----------------------------------+ | ||
/// ``` | ||
/// The length is the length of the MessagePack-encoded notification in bytes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not obvious to me what the order is here,
are new notifications appended?
crates/exex/exex/src/wal/storage.rs
Outdated
/// Truncates the underlying file from the given byte offset (inclusive) to the end of the file. | ||
/// | ||
/// 1. Creates a new file and copies all notifications starting from the offset (inclusive) to | ||
/// the end of the original file. | ||
/// 2. Renames the new file to the original file. | ||
/// | ||
/// # Returns | ||
/// | ||
/// The old and new underlying file sizes in bytes | ||
pub(super) fn truncate_from_offset(&mut self, offset: u64) -> eyre::Result<(u64, u64)> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so finalizing a block means we must copy and reinitialize the entire file?
with > 64 blocks (more with reorged blocks) this isn't very cheap I suppose
what happens to the offsets that the cache tracks?
Towards #10489.
Implements the write-ahead log for ExEx notifications but doesn't integrate it yet.
reth/crates/exex/exex/src/wal/mod.rs
Lines 21 to 33 in 0cc1753