feat: add ETL and use it on HeaderStage
#6154
Conversation
joshieDo
requested review from
rakita,
onbjerg,
rkrasiuk,
shekhirin and
gakonst
as code owners
| "Buffer is not empty" | ||
| ); | ||
| // Return if stage has already completed the gap | ||
| if self.is_etl_ready { |
There was a problem hiding this comment.
isn't this check the same as if gap.is_closed()?
There was a problem hiding this comment.
added additional context on the doc but no:
gapchecks the DB to see if the gap is closedis_etl_readyis true only when the ETL has all headers that will be inserted onto the DB
crates/stages/src/stages/headers.rs
Outdated
| // iterate them in the reverse order | ||
| for header in headers.into_iter().rev() { | ||
| ) -> Result<BlockNumber, StageError> { | ||
| trace!(target: "sync::stages::headers", len = self.header_collector.len(), "writing headers"); |
There was a problem hiding this comment.
i think we need some kind of info logs here, because we only write them during the download phase, and being silent during the ETL phase
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
| // If we only have the genesis block hash, then we are at first sync, and we can remove it, | ||
| // add it to the collector and use tx.append on all hashes. | ||
| if let Some((hash, block_number)) = cursor_header_numbers.last()? { | ||
| if block_number.value()? == 0 { | ||
| self.hash_collector.insert(hash.key()?, 0); | ||
| cursor_header_numbers.delete_current()?; | ||
| first_sync = true; | ||
| } | ||
| } |
There was a problem hiding this comment.
why remove it if we immediately re-insert it?
There was a problem hiding this comment.
if the hash "sort position" is in the middle of all sorted hashes it forces to tx.insert half of the hashes. This way we can use tx.append on all
| @@ -143,7 +143,7 @@ fn should_use_alt_impl(ftype: &String, segment: &syn::PathSegment) -> bool { | |||
| if let (Some(path), 1) = | |||
| (arg_path.path.segments.first(), arg_path.path.segments.len()) | |||
| { | |||
| if ["B256", "Address", "Address", "Bloom", "TxHash"] | |||
| if ["B256", "Address", "Address", "Bloom", "TxHash", "BlockHash"] | |||
There was a problem hiding this comment.
is this breaking/will it change the codec?
There was a problem hiding this comment.
no, this is actually the first type where use BlockHash with Compact (I'm sure we use B256 in some)
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
This reverts commit e218e52.
PR into
feat/static-files.Built on top of cherry-picked #5812 from @onbjerg . This is a stepping stone into pushing Headers into static files.
This PR adds:
reth-etl: temporary file space where we push unsorted data to disk, that can be processed later on in a sorted manner. s/o to https://github.com/akula-bft/akula from where the implementation was inspired from.HeaderStageinto using ETL.HeaderStageactually receives headers in a reverse order. ETL allows us to store all the missing headers on disk, and later on iterate on them in ascending order.TotalDifficultyStagesince now we iterate the headers in ascending order, and can easily calculate the total difficulty as we append to other tables.HeaderHash -> BlockNumbertable can be populated withtx.appendon first sync. (similar on howTxLookupis going to look like when we use ETL there.)checkpoint.done = false. We always download all headers to fill the gap into the ETL collector before executing the stage. Since ETL uses temporary files, if there is a shutdown all this data is lost and the stage will have to execute from scratch as if nothing had happened.