Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore merge upstream 3f9816e1c #55

Merged
merged 97 commits into from
Nov 29, 2024
Merged
Changes from 1 commit
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
dcaa06a
feat: make more block types generic (#12812)
klkvr Nov 25, 2024
caac226
chore: make CanonicalInMemoryState generic over sealed header and blo…
syjn99 Nov 25, 2024
04dd005
feat(trie): sparse trie methods for trie task integration (#12720)
shekhirin Nov 25, 2024
863c523
feat: `ChainStorageReader` (#12836)
klkvr Nov 25, 2024
9f37d40
feat: enforce relations between block parts on NodePrimitives (#12846)
klkvr Nov 25, 2024
6b088bd
perf(engine): sparse trie calculation for state root task (#12843)
shekhirin Nov 25, 2024
e2c42ae
test(trie): use proof nodes helper (#12848)
shekhirin Nov 25, 2024
c44e11b
feat: add `Block` AT to `BlockReader` (#12837)
klkvr Nov 25, 2024
8e4a917
refactor: isolate `BlockchainTree` setup in `DefaultEngineLauncher` (…
klkvr Nov 25, 2024
b96c0d9
docs: clarify installation guide NVMe term (#12853)
vcastellm Nov 25, 2024
404f8f8
refactor: unify logic for writing headers (#12858)
klkvr Nov 26, 2024
82eb383
trie: add unit tests for nibbles (#12758)
tcoratger Nov 26, 2024
96d61dd
feat: add pending_transactions_max in trait TransactionPool (#12856)
kien-rise Nov 26, 2024
fae44bf
chore(trie): move prefix sets to `reth_trie_common` (#12860)
rkrasiuk Nov 26, 2024
13cd77c
test: enable op prim arbitrary in e2e test-utils (#12862)
mattsse Nov 26, 2024
9d2e04c
chore: relax mempool impl (#12865)
mattsse Nov 26, 2024
aa0a114
chore(trie): introduce `serde` feature on `reth-trie-common` (#12864)
rkrasiuk Nov 26, 2024
1b4048e
chore: add encodable to pooled recovered (#12866)
mattsse Nov 26, 2024
a28fa24
feat: use Pooled type for get_pooled_transaction_element in pool trai…
mattsse Nov 26, 2024
334ccd0
chore: add helper for encoded (#12868)
mattsse Nov 26, 2024
dab63e6
feat: emit node event when inserted executed block (#12726)
stevencartavia Nov 26, 2024
b34fb78
feat: make PooledTransactionsElementEcRecovered generic over transact…
mattsse Nov 26, 2024
26fc701
feat: `NodePrimitivesProvider` (#12855)
klkvr Nov 26, 2024
2dc9288
chore(trie): move trie updates to `reth-trie-common` (#12863)
rkrasiuk Nov 26, 2024
2b4fa3b
dep(trie): remove `reth-trie` dep from `reth-trie-sparse` (#12872)
rkrasiuk Nov 26, 2024
c19b8e1
chore(trie): remove todo comment (#12875)
rkrasiuk Nov 26, 2024
2776310
feat: use pooled AT for get_pooled_transactions (#12876)
mattsse Nov 26, 2024
2840b6f
chore: make OrderedSealedBlock generic over header and body types (#1…
htiennv Nov 26, 2024
b6d6bf5
deps: bump sysinfo (#12874)
shekhirin Nov 26, 2024
a0f99df
feat(trie): `StorageRootProvider::storage_multiproof` (#12879)
rkrasiuk Nov 26, 2024
dee0b8c
feat: relax constraints for on_get_pooled_txs (#12880)
mattsse Nov 26, 2024
2d6b893
refactor: unify logic for writing receipts (#12878)
klkvr Nov 26, 2024
4dfaa46
chore: unify recover_singer (#12881)
mattsse Nov 26, 2024
6bba5e6
chore(ci): debug logs on kurtosis-op client advance check failure (#1…
fgimenez Nov 26, 2024
ebf837e
chore: unify more tx signed fns (#12883)
mattsse Nov 26, 2024
d51b347
feat: add utility trait methods to Transaction (#12704)
ftupas Nov 26, 2024
2ce7417
feat: relax provider bounds (#12857)
klkvr Nov 26, 2024
5dc914b
chore: make cargo t compile for reth-trie-common (#12884)
mattsse Nov 26, 2024
83af493
Introduce StateCommitment in StateProviders (#12602)
frisitano Nov 26, 2024
38cf6c9
refactor: improve state writing functions for db provider (#12885)
klkvr Nov 26, 2024
e8d63e4
chore: fix unused warning (#12888)
mattsse Nov 26, 2024
02f3427
feat: introduce networkprimitives in transition fetcher (#12889)
mattsse Nov 26, 2024
6b3c7c6
fix(ci): warn instead of failing no tests (#12893)
hai-rise Nov 27, 2024
acfcfbd
perf: minimize clones when saving blocks (#12870)
hai-rise Nov 27, 2024
3b8c661
added generic header (#12897)
PelleKrab Nov 27, 2024
51afa4c
chore(sdk): Add MaybeArbitrary to all ATs on NodePrimitives (#12847)
zitup Nov 27, 2024
b33757f
feat: extends engine validator (#12900)
mattsse Nov 27, 2024
a0e2961
chore: make BeaconConsensusEngineEvent generic over data primitives (…
programskillforverification Nov 27, 2024
42c24f0
refactor: unify code paths for receipts removal (#12887)
klkvr Nov 27, 2024
db9b86a
Upcast trait bound on `BlockBody::Transaction` to `SignedTransaction`…
emhane Nov 27, 2024
b62929c
feat: add engine validator addon (#12905)
mattsse Nov 27, 2024
533b555
feat: add `Receipt` AT to `ReceiptProvider` (#12890)
klkvr Nov 27, 2024
1131bde
feat(engine): proof fetching on state update for StateRootTask (#12458)
fgimenez Nov 27, 2024
7a6a725
feat: add `Receipt` AT to writer traits (#12892)
klkvr Nov 27, 2024
7a60530
fix: small db provider fixes (#12908)
klkvr Nov 27, 2024
26bfe7c
feat: `SparseStateTrie::reveal_multiproof` (#12909)
rkrasiuk Nov 27, 2024
f9ad3f8
chore: add fs-util::open (#12911)
mattsse Nov 27, 2024
2705e3a
feat: add helpers for opcode tracing (#12899)
mattsse Nov 27, 2024
2700db1
feat(trie): `SparseStateTrie::remove_storage_leaf` (#12912)
rkrasiuk Nov 27, 2024
00c5b69
feat: integrate node primitives in engine handler (#12914)
mattsse Nov 27, 2024
4724564
feat(trie): `SparseStateTrie::storage_trie_mut` (#12913)
rkrasiuk Nov 27, 2024
8d70e89
chore: pass generic header to validate_header_base_fee (#12921)
htiennv Nov 27, 2024
2179301
feat: add functions for env creation (#12928)
mattsse Nov 27, 2024
0df02ca
chore: pass generic header and body to validate_shanghai_withdrawals …
htiennv Nov 28, 2024
bb0bd77
chore: make generic data primitives TreeState (#12924)
htiennv Nov 28, 2024
a3eb302
chore: pass generic header to validate_header_gas (#12931)
0xMaze Nov 28, 2024
39e0573
chore: make generic header InvalidHeaderCache (#12940)
htiennv Nov 28, 2024
28ef574
feat(engine): integrate sparse trie into the state root task (#12907)
shekhirin Nov 28, 2024
064f939
perf(tree): reveal multiproof in tree task (#12949)
rkrasiuk Nov 28, 2024
b2e1da3
fix(tree): root task storage leaf removal (#12950)
rkrasiuk Nov 28, 2024
cbcf79a
chore(engine, state root task): remove unused StdReceiverStream struc…
fgimenez Nov 28, 2024
e9a6e45
feat(trie): `SparseStateTrie::update_account` (#12954)
rkrasiuk Nov 28, 2024
c274462
perf: use alloy hash map in trie related code (#12956)
rkrasiuk Nov 28, 2024
1d5bd46
chore: make op-node-testing- leaner (#12958)
mattsse Nov 28, 2024
eac02d9
fix(trie): always create empty storage multiproofs (#12915)
rkrasiuk Nov 28, 2024
8589503
feat(engine): use `SparseStateTrie::update_account` in state root tas…
shekhirin Nov 28, 2024
1210fd9
chore: make ethereum-node deps leaner (#12963)
mattsse Nov 28, 2024
8c86d63
perf(trie): avoid update reallocation & track wiped (#12929)
rkrasiuk Nov 28, 2024
e1b25c8
perf(tree): keep storage trie around for updates (#12971)
rkrasiuk Nov 28, 2024
55b758a
feat: generic data primitives EngineApiEvent (#12964)
hoank101 Nov 28, 2024
cca6372
feat: integrate type specific engine validator in tree service (#12952)
mattsse Nov 28, 2024
9f20ebc
refactor(tx-pool): add enum `InvalidKind` to `mark_invalid` (#12845)
lean-apple Nov 28, 2024
ca72d45
chore(engine): remove uneeded if (#12974)
fgimenez Nov 28, 2024
8e19455
fix: cap request gas limit in eth estimate (#12970)
mattsse Nov 28, 2024
856dc7d
ci: ignore ethereum crates for testing (#12976)
mattsse Nov 28, 2024
29d84e4
fix(tree): account info on state update in root task (#12978)
rkrasiuk Nov 28, 2024
aea5613
chore: lower tmp ban duration for trusted or static peers (#12961)
mattsse Nov 28, 2024
ae395e8
fix(engine): fetched proof targets in state root task (#12983)
shekhirin Nov 28, 2024
da53d76
perf(trie): init hashed storage only on existing **changed** slots (#…
rkrasiuk Nov 28, 2024
793fc23
fix: correctly poll `NetworkState` (#12973)
klkvr Nov 28, 2024
9fe459e
fix: disable state root calculation for eth_simulateV1 (#12993)
klkvr Nov 29, 2024
3f9816e
feat: integrate generic primitives into pruner (#12995)
klkvr Nov 29, 2024
733f60f
chore: merge upstrea
frisitano Nov 29, 2024
a1a9046
fix: add updated fork.yaml
frisitano Nov 29, 2024
928db75
fix: lints
frisitano Nov 29, 2024
1cadff6
fix: feature propogation
frisitano Nov 29, 2024
771056c
fix: lint
frisitano Nov 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor: unify logic for writing headers (paradigmxyz#12858)
klkvr authored Nov 26, 2024

Verified

This commit was signed with the committer’s verified signature.
OMEGARAZER OMEGA_RAZER
commit 404f8f877842efa887a1b39e78de2eb6ec97289d
39 changes: 22 additions & 17 deletions crates/storage/provider/src/providers/database/provider.rs
Original file line number Diff line number Diff line change
@@ -2749,22 +2749,12 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockWrite
fn insert_block(
&self,
block: SealedBlockWithSenders<Self::Block>,
write_transactions_to: StorageLocation,
write_to: StorageLocation,
) -> ProviderResult<StoredBlockBodyIndices> {
let block_number = block.number;

let mut durations_recorder = metrics::DurationsRecorder::default();

self.tx.put::<tables::CanonicalHeaders>(block_number, block.hash())?;
durations_recorder.record_relative(metrics::Action::InsertCanonicalHeaders);

// Put header with canonical hashes.
self.tx.put::<tables::Headers>(block_number, block.header.as_ref().clone())?;
durations_recorder.record_relative(metrics::Action::InsertHeaders);

self.tx.put::<tables::HeaderNumbers>(block.hash(), block_number)?;
durations_recorder.record_relative(metrics::Action::InsertHeaderNumbers);

// total difficulty
let ttd = if block_number == 0 {
block.difficulty
@@ -2775,8 +2765,26 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockWrite
parent_ttd + block.difficulty
};

self.tx.put::<tables::HeaderTerminalDifficulties>(block_number, ttd.into())?;
durations_recorder.record_relative(metrics::Action::InsertHeaderTerminalDifficulties);
if write_to.database() {
self.tx.put::<tables::CanonicalHeaders>(block_number, block.hash())?;
durations_recorder.record_relative(metrics::Action::InsertCanonicalHeaders);

// Put header with canonical hashes.
self.tx.put::<tables::Headers>(block_number, block.header.as_ref().clone())?;
durations_recorder.record_relative(metrics::Action::InsertHeaders);

self.tx.put::<tables::HeaderTerminalDifficulties>(block_number, ttd.into())?;
durations_recorder.record_relative(metrics::Action::InsertHeaderTerminalDifficulties);
}

if write_to.static_files() {
let mut writer =
self.static_file_provider.get_writer(block_number, StaticFileSegment::Headers)?;
writer.append_header(&block.header, ttd, &block.hash())?;
}

self.tx.put::<tables::HeaderNumbers>(block.hash(), block_number)?;
durations_recorder.record_relative(metrics::Action::InsertHeaderNumbers);

let mut next_tx_num = self
.tx
@@ -2805,10 +2813,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockWrite
next_tx_num += 1;
}

self.append_block_bodies(
vec![(block_number, Some(block.block.body))],
write_transactions_to,
)?;
self.append_block_bodies(vec![(block_number, Some(block.block.body))], write_to)?;

debug!(
target: "providers::db",
5 changes: 4 additions & 1 deletion crates/storage/provider/src/traits/block.rs
Original file line number Diff line number Diff line change
@@ -68,10 +68,13 @@ pub trait BlockWriter: Send + Sync {
///
/// Return [StoredBlockBodyIndices] that contains indices of the first and last transactions and
/// transition in the block.
///
/// Accepts [`StorageLocation`] value which specifies where transactions and headers should be
/// written.
fn insert_block(
&self,
block: SealedBlockWithSenders<Self::Block>,
write_transactions_to: StorageLocation,
write_to: StorageLocation,
) -> ProviderResult<StoredBlockBodyIndices>;

/// Appends a batch of block bodies extending the canonical chain. This is invoked during
72 changes: 3 additions & 69 deletions crates/storage/provider/src/writer/mod.rs
Original file line number Diff line number Diff line change
@@ -4,26 +4,22 @@ use crate::{
BlockExecutionWriter, BlockWriter, HistoryWriter, StateChangeWriter, StateWriter,
StaticFileProviderFactory, StorageLocation, TrieWriter,
};
use alloy_consensus::Header;
use alloy_primitives::{BlockNumber, B256, U256};
use alloy_primitives::BlockNumber;
use reth_chain_state::ExecutedBlock;
use reth_db::{
cursor::DbCursorRO,
models::CompactU256,
tables,
transaction::{DbTx, DbTxMut},
};
use reth_errors::{ProviderError, ProviderResult};
use reth_execution_types::ExecutionOutcome;
use reth_primitives::{SealedBlock, StaticFileSegment};
use reth_stages_types::{StageCheckpoint, StageId};
use reth_primitives::StaticFileSegment;
use reth_storage_api::{
DBProvider, HeaderProvider, ReceiptWriter, StageCheckpointWriter, TransactionsProviderExt,
};
use reth_storage_errors::writer::UnifiedStorageWriterError;
use revm::db::OriginalValuesKnown;
use std::{borrow::Borrow, sync::Arc};
use tracing::{debug, instrument};
use tracing::debug;

mod database;
mod static_file;
@@ -196,7 +192,6 @@ where
let sealed_block =
block.block().clone().try_with_senders_unchecked(block.senders().clone()).unwrap();
self.database().insert_block(sealed_block, StorageLocation::Both)?;
self.save_header_and_transactions(block.block.clone())?;

// Write state and changesets to the database.
// Must be written after blocks because of the receipt lookup.
@@ -223,35 +218,6 @@ where
Ok(())
}

/// Writes the header & transactions to static files, and updates their respective checkpoints
/// on database.
#[instrument(level = "trace", skip_all, fields(block = ?block.num_hash()) target = "storage")]
fn save_header_and_transactions(&self, block: Arc<SealedBlock>) -> ProviderResult<()> {
debug!(target: "provider::storage_writer", "Writing headers and transactions.");

{
let header_writer =
self.static_file().get_writer(block.number, StaticFileSegment::Headers)?;
let mut storage_writer = UnifiedStorageWriter::from(self.database(), header_writer);
let td = storage_writer.append_headers_from_blocks(
block.header().number,
std::iter::once(&(block.header(), block.hash())),
)?;

debug!(target: "provider::storage_writer", block_num=block.number, "Updating transaction metadata after writing");
self.database()
.tx_ref()
.put::<tables::HeaderTerminalDifficulties>(block.number, CompactU256(td))?;
self.database()
.save_stage_checkpoint(StageId::Headers, StageCheckpoint::new(block.number))?;
}

self.database()
.save_stage_checkpoint(StageId::Bodies, StageCheckpoint::new(block.number))?;

Ok(())
}

/// Removes all block, transaction and receipt data above the given block number from the
/// database and static files. This is exclusive, i.e., it only removes blocks above
/// `block_number`, and does not remove `block_number`.
@@ -323,38 +289,6 @@ where
None => Err(UnifiedStorageWriterError::MissingStaticFileWriter),
}
}

/// Appends headers to static files, using the
/// [`HeaderTerminalDifficulties`](tables::HeaderTerminalDifficulties) table to determine the
/// total difficulty of the parent block during header insertion.
///
/// NOTE: The static file writer used to construct this [`UnifiedStorageWriter`] MUST be a
/// writer for the Headers segment.
pub fn append_headers_from_blocks<H, I>(
&mut self,
initial_block_number: BlockNumber,
headers: impl Iterator<Item = I>,
) -> ProviderResult<U256>
where
I: Borrow<(H, B256)>,
H: Borrow<Header>,
{
self.ensure_static_file_segment(StaticFileSegment::Headers)?;

let mut td = self
.database()
.header_td_by_number(initial_block_number)?
.ok_or(ProviderError::TotalDifficultyNotFound(initial_block_number))?;

for pair in headers {
let (header, hash) = pair.borrow();
let header = header.borrow();
td += header.difficulty;
self.static_file_mut().append_header(header, td, hash)?;
}

Ok(td)
}
}

impl<ProviderDB>