Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"blockchain_tree: Block hash not found in block indices block_hash=0x743071" #3446

Closed
1 task done
Gaon3 opened this issue Jun 28, 2023 · 11 comments · Fixed by #3588
Closed
1 task done

"blockchain_tree: Block hash not found in block indices block_hash=0x743071" #3446

Gaon3 opened this issue Jun 28, 2023 · 11 comments · Fixed by #3588
Labels
A-engine Related to the engine implementation C-bug An unexpected or incorrect behavior

Comments

@Gaon3
Copy link
Contributor

Gaon3 commented Jun 28, 2023 •
edited
Loading

Describe the bug

My node finished syncing today's morning.
However, it kept giving me this warning:

2023-06-28T07:57:10.514310Z  WARN make_canonical{block_hash=0x743071e9f63b8582a5f6af31a8e995c7b86d41cd9a88716068a676a3061a4e46}: blockchain_tree: Block hash not found in block indices block_hash=0x743071
e9f63b8582a5f6af31a8e995c7b86d41cd9a88716068a676a3061a4e46
                                                                                                                                                 
2023-06-28T07:57:10.514344Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x743071e9f63b8582a5f6af31a8e995c7b86d41cd9a88716068a676a3
061a4e46 }) state=ForkchoiceState { head_block_hash: 0x743071e9f63b8582a5f6af31a8e995c7b86d41cd9a88716068a676a3061a4e46, safe_block_hash: 0xd0911be392f8cf544ecff2e4d7c70d6297632b638c61c44f7f6f0346ce4361c
5, finalized_block_hash: 0x1028f6714a445c9fcb91c0293af00f196d10330efe0077bc1f73d3be5cd84cc8 }

I restarted my node and Lighthouse, and this time the node tried to insert block backwards (try_insert_validated_block) until it reached block number 17576413. At this point it only tried to insert this specific block.
My CL client spammed "Execution engine call failed" and "Failed to update execution head" messages.

I can't sync any further

Steps to reproduce

idk

Node logs

No response

Platform(s)

Linux (x86)

What version/commit are you on?

46d4795

What database version are you on?

1

If you've built Reth from source, provide the full command you used

RUSTFLAGS="-C target-cpu=native" cargo install --locked --path bin/reth --bin reth --profile maxperf --features jemalloc

Code of Conduct

  • I agree to follow the Code of Conduct
@Gaon3 Gaon3 added C-bug An unexpected or incorrect behavior S-needs-triage This issue needs to be labelled labels Jun 28, 2023
@rakita
Copy link
Collaborator

rakita commented Jun 28, 2023

Can you share bigger logs (dump the file if you can) is this happening right after the pipeline finished its initial sync.

And can you share logs after restart (both reth and lighthouse).

Why i ask for additional logs is that this log (BlockHashNotFoundInChain ) is expected, it is expected for some blocks to not be found inside reth, this just means that reth needs to fetch them or do different sync. So maybe it takes more time than expected and will unblock itself.

But what I want to see is this "I can't sync any further" Is it stuck and can I see anything useful from logs.

@Gaon3
Copy link
Contributor Author

Gaon3 commented Jun 28, 2023

I dont save logs to disk (previous time ran out of space because the file go so big), and I just fucked up while copying tmux's output (somehow erased the whole thing). But here is what happened:

what happened is Reth synced, then synced the 1300 or so blocks it missed while syncing, then synced the 130 blocks it missed while syncing the 1300 blocks, tried to insert block number 17576454 and at this point the "failed to canonicalize the head hash" error happened

Do you want me to keep trying to run it for a while to see if it unfolds itself ?

@rakita
Copy link
Collaborator

rakita commented Jun 28, 2023

I dont save logs to disk (previous time ran out of space because the file go so big), and I just fucked up while copying tmux's output (somehow erased the whole thing). But here is what happened:

what happened is Reth synced, then synced the 1300 or so blocks it missed while syncing, then synced the 130 blocks it missed while syncing the 1300 blocks, tried to insert block number 17576454 and at this point the "failed to canonicalize the head hash" error happened

Even this info is great!

Do you want me to keep trying to run it for a while to see if it unfolds itself ?

This could help, I would leave it and see is there are any changes.

Another approach as you said you already restarted it, maybe we can try that and you can scrap the logs from the start.

@Gaon3
Copy link
Contributor Author

Gaon3 commented Jun 28, 2023

wanted to post log file but I have an old bug where nothing is ever written in /.cache/reth/logs/reth.log despite running RETH_LOG=info reth node --log.persistent.
I pulled latest commit, ran the node and here what I retrived: (I deliberately not copied chunk of messages related to peers, '...')

root@Ubuntu-2204-jammy-amd64-base /home/reth # RUST_LOG=info reth node --log.persistent                                                                                                                    
2023-06-28T09:40:28.576733Z  INFO reth::cli: reth 0.1.0-alpha.1 (bb57556a) starting                                                                                                                        
2023-06-28T09:40:28.576959Z  INFO reth::cli: Configuration loaded path="/root/.local/share/reth/mainnet/reth.toml"                                                                                         
2023-06-28T09:40:28.576971Z  INFO reth::cli: Opening database path="/root/.local/share/reth/mainnet/db"                                                                                                    
2023-06-28T09:40:28.577642Z  INFO reth::cli: Database opened                                                                                                                                               
2023-06-28T09:40:28.577699Z  INFO reth::cli: Pre-merge hard forks (block based):                                                                                                                           
- Frontier                         @0                                                                                                                                                                      
- Homestead                        @1150000                                                                                                                                                                
- Dao                              @1920000                                                                                                                                                                
- Tangerine                        @2463000                                                                                                                                                                
- SpuriousDragon                   @2675000                                                                                                                                                                
- Byzantium                        @4370000                                                                                                                                                                
- Constantinople                   @7280000                                                                                                                                                                
- Petersburg                       @7280000                                                                                                                                                                
- Istanbul                         @9069000                                                                                                                                                                
- MuirGlacier                      @9200000                                                                                                                                                                
- Berlin                           @12244000                                                                                                                                                               
- London                           @12965000                                                                                                                                                               - ArrowGlacier                     @13773000                                                                                                                                                               
- GrayGlacier                      @15050000                                                                                                                                                               Merge hard forks:                                                                                                                                                                                          
- Paris                            @58750000000000000000000 (network is not known to be merged)                                                                                                                                                                                                                                                                                                                       
Post-merge hard forks (timestamp based):                                                                                                                                                                   - Shanghai                         @1681338455                                                                                                                                                             
                                                                                                                                                                                                           2023-06-28T09:40:28.577939Z  INFO reth::cli: Transaction pool initialized                                                                                                                                  
2023-06-28T09:40:28.577954Z  INFO reth::cli: Connecting to P2P network                                                                                                                                     2023-06-28T09:40:28.579048Z  INFO reth::cli: Connected to P2P network peer_id=0x35ae…167e local_addr=0.0.0.0:30303                                                                                         
2023-06-28T09:40:28.579294Z  INFO reth::cli: Consensus engine initialized                                                                                                                                  2023-06-28T09:40:28.579311Z  INFO reth::cli: Engine API handler initialized                                                                                                                                
2023-06-28T09:40:28.579451Z  INFO reth::cli: RPC auth server started url=127.0.0.1:8551                                                                                                                    2023-06-28T09:40:28.579523Z  INFO reth::cli: RPC IPC server started url=/tmp/reth.ipc                                                                                                                      2023-06-28T09:40:28.579531Z  INFO reth::cli: Starting consensus engine                                                                                                                                     
2023-06-28T09:40:28.579888Z  INFO reth::cli: Status connected_peers=0 latest_block=17576412          
...
2023-06-28T09:40:58.580238Z  INFO reth::cli: Status connected_peers=5 latest_block=17576412
2023-06-28T09:41:01.074418Z  INFO net: Session established remote_addr=162.19.171.169:30303 client_version=Geth/v1.11.5-stable/linux-amd64/go1.20.2 peer_id=0xedd435f28c11d2684f295e06bfb334c3442493ef2a068
ad3863c30e8caff44fa38f7cad25b318400499b2fcab6326acbe43d5b6aff45541cb1fb3ffb1a6a7002 total_active=6
2023-06-28T09:41:08.883591Z  INFO net: Session established remote_addr=13.37.220.1:30303 client_version= peer_id=0x2575b5865554afe96eabede07289901a7c93b8b44424a77e8b47e3140a0a4d1c6506e1fac84b3beb562dc517
7ecb114f6cde12d13fec9d34806db67121c0d6e5 total_active=7
2023-06-28T09:41:13.852984Z  INFO net: Session established remote_addr=3.209.45.79:30303 client_version=Geth/v1.12.0-stable-e501b3b0/linux-amd64/go1.20.4 peer_id=0x22a8232c3abc76a16ae9d6c3b164f98775fe226
f0917b0ca871128a74a8e9630b458460865bab457221f1d448dd9791d24c4e5d88786180ac185df813a68d4de total_active=8
2023-06-28T09:41:14.060212Z  INFO net: Session established remote_addr=18.138.108.67:30303 client_version=Geth/v1.12.0-stable-e501b3b0/linux-amd64/go1.20.4 peer_id=0xd860a01f9722d78051619d1e2351aba3f43f9
43f6f00718d1b9baa4101932a1f5011f16bb2b1bb35db20d6fe28fa0bf09636d26a87d31de9ec6203eeedb1f666 total_active=8
2023-06-28T09:41:17.961574Z  WARN make_canonical{block_hash=0xe045a72b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074c4c7d499}: blockchain_tree: Block hash not found in block indices block_hash=0xe045a7
2b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074c4c7d499
2023-06-28T09:41:17.961603Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0xe045a72b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074
c4c7d499 }) state=ForkchoiceState { head_block_hash: 0xe045a72b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074c4c7d499, safe_block_hash: 0xa5c063c67f28a27296a087dfa63b92ad21d16f6ca8ce4fa83833c8832665e53
7, finalized_block_hash: 0xd0911be392f8cf544ecff2e4d7c70d6297632b638c61c44f7f6f0346ce4361c5 }
2023-06-28T09:41:17.961689Z  WARN make_canonical{block_hash=0xe045a72b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074c4c7d499}: blockchain_tree: Block hash not found in block indices block_hash=0xe045a7
2b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074c4c7d499
2023-06-28T09:41:17.961688Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xe045a72b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074c4c7d499 safe_block_hash=0xa5c063c67f28a27296a087dfa63b9
2ad21d16f6ca8ce4fa83833c8832665e537 finalized_block_hash=0xd0911be392f8cf544ecff2e4d7c70d6297632b638c61c44f7f6f0346ce4361c5 status=Syncing
2023-06-28T09:41:17.961706Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0xe045a72b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074
c4c7d499 }) state=ForkchoiceState { head_block_hash: 0xe045a72b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074c4c7d499, safe_block_hash: 0xa5c063c67f28a27296a087dfa63b92ad21d16f6ca8ce4fa83833c8832665e53
7, finalized_block_hash: 0xd0911be392f8cf544ecff2e4d7c70d6297632b638c61c44f7f6f0346ce4361c5 }
2023-06-28T09:41:17.961739Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xe045a72b6ab11b48a57b850057fdad66b6f02ea7bfe255a16821d074c4c7d499 safe_block_hash=0xa5c063c67f28a27296a087dfa63b9
2ad21d16f6ca8ce4fa83833c8832665e537 finalized_block_hash=0xd0911be392f8cf544ecff2e4d7c70d6297632b638c61c44f7f6f0346ce4361c5 status=Syncing
2023-06-28T09:41:18.146965Z  INFO try_insert_validated_block{block=(17576476, 0xa5c063c67f28a27296a087dfa63b92ad21d16f6ca8ce4fa83833c8832665e537)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576475, 0x96879432a3927b6fe202997922170077f78fa1a7bb4a9f27ceb106f2b64c58d2) })
2023-06-28T09:41:18.147768Z  INFO reth::node::events: Executing stage pipeline_stages=1/13 stage=Headers from=17576412 checkpoint=100.0% eta=unknown
2023-06-28T09:41:18.155419Z  INFO execute{stage=Headers}: sync::stages::headers: Received headers len=32
2023-06-28T09:41:18.156509Z  INFO reth::node::events: Stage finished executing pipeline_stages=1/13 stage=Headers block=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:18.157144Z  INFO reth::node::events: Executing stage pipeline_stages=2/13 stage=TotalDifficulty from=17576412 checkpoint=100.0% eta=unknown
2023-06-28T09:41:18.157196Z  INFO reth::node::events: Stage finished executing pipeline_stages=2/13 stage=TotalDifficulty block=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:18.157296Z  INFO reth::node::events: Executing stage pipeline_stages=3/13 stage=Bodies from=17576412 checkpoint=100.0% eta=unknown
2023-06-28T09:41:18.157302Z  INFO downloaders::bodies: Downloading bodies 17576413..=17576444 distance=31
2023-06-28T09:41:19.060379Z  INFO reth::node::events: Stage finished executing pipeline_stages=3/13 stage=Bodies block=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:19.083102Z  INFO execute{stage=SenderRecovery}: sync::stages::sender_recovery: Recovering senders tx_range=2013868541..=2013873226
2023-06-28T09:41:19.083116Z  INFO reth::node::events: Executing stage pipeline_stages=4/13 stage=SenderRecovery from=17576412 checkpoint=100.0% eta=unknown
2023-06-28T09:41:19.275675Z  INFO reth::node::events: Stage finished executing pipeline_stages=4/13 stage=SenderRecovery block=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:19.276407Z  INFO reth::node::events: Executing stage pipeline_stages=5/13 stage=Execution from=17576412 checkpoint=100.0% eta=unknown
2023-06-28T09:41:19.741168Z  INFO reth::node::events: Stage finished executing pipeline_stages=5/13 stage=Execution block=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:19.828420Z  INFO execute{stage=MerkleUnwind}: sync::stages::merkle::unwind: Stage is always skipped
2023-06-28T09:41:19.828430Z  INFO reth::node::events: Executing stage pipeline_stages=6/13 stage=MerkleUnwind from=17576412 checkpoint=17576412 eta=unknown
2023-06-28T09:41:19.828461Z  INFO reth::node::events: Stage finished executing pipeline_stages=6/13 stage=MerkleUnwind block=17576444 checkpoint=17576444 eta=unknown
2023-06-28T09:41:19.828565Z  INFO reth::node::events: Executing stage pipeline_stages=7/13 stage=AccountHashing from=17576412 checkpoint=100.0% eta=unknown
2023-06-28T09:41:19.892355Z  INFO reth::node::events: Stage finished executing pipeline_stages=7/13 stage=AccountHashing block=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:19.912956Z  INFO reth::node::events: Executing stage pipeline_stages=8/13 stage=StorageHashing from=17576412 checkpoint=100.0% eta=unknown
2023-06-28T09:41:20.018244Z  INFO reth::node::events: Stage finished executing pipeline_stages=8/13 stage=StorageHashing block=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:20.076357Z  INFO reth::node::events: Executing stage pipeline_stages=9/13 stage=MerkleExecute from=17576412 checkpoint=100.0% eta=unknown
2023-06-28T09:41:22.063604Z  INFO reth::node::events: Stage finished executing pipeline_stages=9/13 stage=MerkleExecute block=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:22.245308Z  INFO reth::node::events: Executing stage pipeline_stages=10/13 stage=TransactionLookup from=17576412 checkpoint=100.0% eta=unknown
2023-06-28T09:41:22.655273Z  INFO reth::node::events: Stage finished executing pipeline_stages=10/13 stage=TransactionLookup block=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:22.689153Z  INFO reth::node::events: Executing stage pipeline_stages=11/13 stage=IndexStorageHistory from=17576412 checkpoint=17576412 eta=unknown
2023-06-28T09:41:23.057369Z  INFO reth::node::events: Stage finished executing pipeline_stages=11/13 stage=IndexStorageHistory block=17576444 checkpoint=17576444 eta=unknown
2023-06-28T09:41:23.105366Z  INFO reth::node::events: Executing stage pipeline_stages=12/13 stage=IndexAccountHistory from=17576412 checkpoint=17576412 eta=unknown
2023-06-28T09:41:23.229246Z  INFO reth::node::events: Stage finished executing pipeline_stages=12/13 stage=IndexAccountHistory block=17576444 checkpoint=17576444 eta=unknown
2023-06-28T09:41:23.270014Z  INFO reth::node::events: Executing stage pipeline_stages=13/13 stage=Finish from=17576412 checkpoint=17576412 eta=unknown
2023-06-28T09:41:23.270032Z  INFO reth::node::events: Stage finished executing pipeline_stages=13/13 stage=Finish block=17576444 checkpoint=17576444 eta=unknown
2023-06-28T09:41:28.579755Z  INFO reth::cli: Status connected_peers=7 latest_block=17576412
...

2023-06-28T09:41:35.758638Z  WARN make_canonical{block_hash=0x90cffb014e88f56ea29ab52ee93720d651937de7ebe6c772009e4b2a94134e72}: blockchain_tree: Block hash not found in block indices block_hash=0x90cffb
014e88f56ea29ab52ee93720d651937de7ebe6c772009e4b2a94134e72
2023-06-28T09:41:35.758662Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x90cffb014e88f56ea29ab52ee93720d651937de7ebe6c772009e4b2a
94134e72 }) state=ForkchoiceState { head_block_hash: 0x90cffb014e88f56ea29ab52ee93720d651937de7ebe6c772009e4b2a94134e72, safe_block_hash: 0xa5c063c67f28a27296a087dfa63b92ad21d16f6ca8ce4fa83833c8832665e53
7, finalized_block_hash: 0xd0911be392f8cf544ecff2e4d7c70d6297632b638c61c44f7f6f0346ce4361c5 }
2023-06-28T09:41:35.758717Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x90cffb014e88f56ea29ab52ee93720d651937de7ebe6c772009e4b2a94134e72 safe_block_hash=0xa5c063c67f28a27296a087dfa63b9
2ad21d16f6ca8ce4fa83833c8832665e537 finalized_block_hash=0xd0911be392f8cf544ecff2e4d7c70d6297632b638c61c44f7f6f0346ce4361c5 status=Syncing
2023-06-28T09:41:35.900547Z  INFO try_insert_validated_block{block=(17576508, 0x90cffb014e88f56ea29ab52ee93720d651937de7ebe6c772009e4b2a94134e72)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576507, 0x871fa717ae8cb6b87e5b74c5ffb7dcf684f5e4bfaebb7a65d7d59492fda7d82e) })
2023-06-28T09:41:43.942758Z  INFO net: Session established remote_addr=168.235.81.22:44124 client_version=Geth/v1.12.0-stable-e501b3b0/linux-amd64/go1.20.4 peer_id=0xcb6b5b33d62e3ab96d51d33c12e55c8e83be0
49410d91d8ba381e9c095784a064dbf743620d4c8c2d77f709720aba1d93d7767411cf15ad643fde24101e66f71 total_active=9
2023-06-28T09:41:44.317695Z  WARN make_canonical{block_hash=0x0a85c7062704c46c3b904340d9ea1f3ea0594f8daaf4698d62f78d793989f35f}: blockchain_tree: Block hash not found in block indices block_hash=0x0a85c7
062704c46c3b904340d9ea1f3ea0594f8daaf4698d62f78d793989f35f
2023-06-28T09:41:44.317723Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x0a85c7062704c46c3b904340d9ea1f3ea0594f8daaf4698d62f78d79
3989f35f }) state=ForkchoiceState { head_block_hash: 0x0a85c7062704c46c3b904340d9ea1f3ea0594f8daaf4698d62f78d793989f35f, safe_block_hash: 0x17c016871507cd5d9c1ca31efbd49937d5d821f46bc5e279d50df50c3ad216c
4, finalized_block_hash: 0x90cffb014e88f56ea29ab52ee93720d651937de7ebe6c772009e4b2a94134e72 }
2023-06-28T09:41:44.317808Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x0a85c7062704c46c3b904340d9ea1f3ea0594f8daaf4698d62f78d793989f35f safe_block_hash=0x17c016871507cd5d9c1ca31efbd49
937d5d821f46bc5e279d50df50c3ad216c4 finalized_block_hash=0x90cffb014e88f56ea29ab52ee93720d651937de7ebe6c772009e4b2a94134e72 status=Syncing
2023-06-28T09:41:44.405119Z  INFO try_insert_validated_block{block=(17576570, 0x0a85c7062704c46c3b904340d9ea1f3ea0594f8daaf4698d62f78d793989f35f)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576569, 0x2602ad0a44e4e13ded17338f827861fdd13e06bc6e5337d74ce5573d7a50500c) })
2023-06-28T09:41:44.405437Z  INFO reth::node::events: Executing stage pipeline_stages=1/13 stage=Headers from=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:44.457976Z  INFO execute{stage=Headers}: sync::stages::headers: Received headers len=64
2023-06-28T09:41:44.459450Z  INFO reth::node::events: Stage finished executing pipeline_stages=1/13 stage=Headers block=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:44.460394Z  INFO reth::node::events: Executing stage pipeline_stages=2/13 stage=TotalDifficulty from=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:44.460447Z  INFO reth::node::events: Stage finished executing pipeline_stages=2/13 stage=TotalDifficulty block=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:44.460569Z  INFO reth::node::events: Executing stage pipeline_stages=3/13 stage=Bodies from=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:44.460578Z  INFO downloaders::bodies: Downloading bodies 17576445..=17576508 distance=63
2023-06-28T09:41:46.012250Z  INFO reth::node::events: Stage finished executing pipeline_stages=3/13 stage=Bodies block=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:46.046794Z  INFO execute{stage=SenderRecovery}: sync::stages::sender_recovery: Recovering senders tx_range=2013873227..=2013884118
2023-06-28T09:41:46.046842Z  INFO reth::node::events: Executing stage pipeline_stages=4/13 stage=SenderRecovery from=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:46.542404Z  INFO reth::node::events: Stage finished executing pipeline_stages=4/13 stage=SenderRecovery block=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:46.542985Z  INFO reth::node::events: Executing stage pipeline_stages=5/13 stage=Execution from=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:48.285110Z  INFO reth::node::events: Stage finished executing pipeline_stages=5/13 stage=Execution block=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:48.415771Z  INFO execute{stage=MerkleUnwind}: sync::stages::merkle::unwind: Stage is always skipped
2023-06-28T09:41:48.415787Z  INFO reth::node::events: Executing stage pipeline_stages=6/13 stage=MerkleUnwind from=17576444 checkpoint=17576444 eta=unknown
2023-06-28T09:41:48.415806Z  INFO reth::node::events: Stage finished executing pipeline_stages=6/13 stage=MerkleUnwind block=17576508 checkpoint=17576508 eta=unknown
2023-06-28T09:41:48.415905Z  INFO reth::node::events: Executing stage pipeline_stages=7/13 stage=AccountHashing from=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:48.696002Z  INFO net: Session established remote_addr=135.125.160.96:30000 client_version=Geth/v1.10.25-stable/linux-amd64/go1.17.10 peer_id=0x1f77d935fe5e4a477f781e67bcb519adea5dc5164d2
3b0a996b0a5cd7a36c8a49c07c24bd92a0450ebc58df61ebdd88fc91afa9250e1200e1f0e1450cd57ee4e total_active=10
2023-06-28T09:41:48.710049Z  INFO reth::node::events: Stage finished executing pipeline_stages=7/13 stage=AccountHashing block=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:48.720134Z  INFO net: Session established remote_addr=65.108.75.215:15701 client_version=Nethermind/v1.19.2+42ba08b7/linux-x64/dotnet7.0.5 peer_id=0x14635f84d82d8cde55b509fdf33021a067a36
db30f3e67e7ba0ec5ebeab534b35bf71c50d78ba4261310e59374005b6f366e5436e4979b75a2b07e788536270f total_active=11
2023-06-28T09:41:48.750534Z  INFO net: Session established remote_addr=65.108.229.126:30303 client_version=Nethermind/v1.19.2+42ba08b7/linux-x64/dotnet7.0.5 peer_id=0xc621f9bc8f627556c222fcdec0deaf74364e
192fdbbc0b924677425db6437ea4448faaefcb2aa74513502a467fc2de3fb6fa443d661b44d4cdc5d04188021705 total_active=12
2023-06-28T09:41:48.773949Z  INFO reth::node::events: Executing stage pipeline_stages=8/13 stage=StorageHashing from=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:49.144486Z  INFO reth::node::events: Stage finished executing pipeline_stages=8/13 stage=StorageHashing block=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:49.254193Z  INFO reth::node::events: Executing stage pipeline_stages=9/13 stage=MerkleExecute from=17576444 checkpoint=100.0% eta=unknown
...

2023-06-28T09:41:51.293029Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x2aa4c213c60bc25a431dac799c64201359b18546ef373bf7e4b1c770940cdad0 safe_block_hash=0x24a3d8ce280db1e98309515b80209
1eebd6864e6a8e3a6e7705dd98c12b12540 finalized_block_hash=0x0a85c7062704c46c3b904340d9ea1f3ea0594f8daaf4698d62f78d793989f35f status=Syncing
2023-06-28T09:41:54.010280Z  INFO net: Session established remote_addr=15.204.197.5:30303 client_version=Geth/v1.11.5-stable/linux-amd64/go1.20.2 peer_id=0xc6e7a6a1934fa4bcad9bd448afd490661c6f2186fd07fe6
8ae76805bedbe0631594ed320f9a420cdcfe8510079135c1496f41a6b9c7c6f452dc7ebd640356f47 total_active=16
2023-06-28T09:41:55.656822Z  INFO reth::node::events: Stage finished executing pipeline_stages=9/13 stage=MerkleExecute block=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:55.943647Z  INFO reth::node::events: Executing stage pipeline_stages=10/13 stage=TransactionLookup from=17576444 checkpoint=100.0% eta=unknown
2023-06-28T09:41:57.005394Z  INFO reth::node::events: Stage finished executing pipeline_stages=10/13 stage=TransactionLookup block=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:57.090288Z  INFO reth::node::events: Executing stage pipeline_stages=11/13 stage=IndexStorageHistory from=17576444 checkpoint=17576444 eta=unknown
2023-06-28T09:41:57.734710Z  INFO reth::node::events: Stage finished executing pipeline_stages=11/13 stage=IndexStorageHistory block=17576508 checkpoint=17576508 eta=unknown
2023-06-28T09:41:57.815685Z  INFO reth::node::events: Executing stage pipeline_stages=12/13 stage=IndexAccountHistory from=17576444 checkpoint=17576444 eta=unknown
2023-06-28T09:41:58.068277Z  INFO reth::node::events: Stage finished executing pipeline_stages=12/13 stage=IndexAccountHistory block=17576508 checkpoint=17576508 eta=unknown
2023-06-28T09:41:58.120533Z  INFO reth::node::events: Executing stage pipeline_stages=13/13 stage=Finish from=17576444 checkpoint=17576444 eta=unknown
2023-06-28T09:41:58.120544Z  INFO reth::node::events: Stage finished executing pipeline_stages=13/13 stage=Finish block=17576508 checkpoint=17576508 eta=unknown
2023-06-28T09:41:58.120757Z  INFO reth::node::events: Executing stage pipeline_stages=1/13 stage=Headers from=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:58.193573Z  INFO execute{stage=Headers}: sync::stages::headers: Received headers len=62
2023-06-28T09:41:58.198730Z  INFO reth::node::events: Stage finished executing pipeline_stages=1/13 stage=Headers block=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:41:58.199483Z  INFO reth::node::events: Executing stage pipeline_stages=2/13 stage=TotalDifficulty from=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:58.199541Z  INFO reth::node::events: Stage finished executing pipeline_stages=2/13 stage=TotalDifficulty block=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:41:58.199648Z  INFO reth::node::events: Executing stage pipeline_stages=3/13 stage=Bodies from=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:41:58.199654Z  INFO downloaders::bodies: Downloading bodies 17576509..=17576570 distance=61
2023-06-28T09:41:58.252845Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xd2c16b9b0c1d02e04070b4b847781adcfec6fd8c65ce2fb1724b65f0f5b7a80a safe_block_hash=0xbc11628212cb07b0e5daa982ba208
9b238010447a71b2b5577f528410cc868c9 finalized_block_hash=0x2aa4c213c60bc25a431dac799c64201359b18546ef373bf7e4b1c770940cdad0 status=Syncing
2023-06-28T09:41:58.579660Z  INFO reth::cli: Status connected_peers=16 stage=Bodies checkpoint=100.0% eta=unknown
2023-06-28T09:41:59.869728Z  INFO reth::node::events: Stage finished executing pipeline_stages=3/13 stage=Bodies block=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:41:59.926972Z  INFO execute{stage=SenderRecovery}: sync::stages::sender_recovery: Recovering senders tx_range=2013884119..=2013894223
2023-06-28T09:41:59.926970Z  INFO reth::node::events: Executing stage pipeline_stages=4/13 stage=SenderRecovery from=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:42:00.405532Z  INFO reth::node::events: Stage finished executing pipeline_stages=4/13 stage=SenderRecovery block=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:00.408922Z  INFO reth::node::events: Executing stage pipeline_stages=5/13 stage=Execution from=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:42:02.150335Z  INFO reth::node::events: Stage finished executing pipeline_stages=5/13 stage=Execution block=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:02.306778Z  INFO execute{stage=MerkleUnwind}: sync::stages::merkle::unwind: Stage is always skipped
2023-06-28T09:42:02.306808Z  INFO reth::node::events: Executing stage pipeline_stages=6/13 stage=MerkleUnwind from=17576508 checkpoint=17576508 eta=unknown
2023-06-28T09:42:02.306831Z  INFO reth::node::events: Stage finished executing pipeline_stages=6/13 stage=MerkleUnwind block=17576570 checkpoint=17576570 eta=unknown
2023-06-28T09:42:02.306934Z  INFO reth::node::events: Executing stage pipeline_stages=7/13 stage=AccountHashing from=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:42:02.593547Z  INFO reth::node::events: Stage finished executing pipeline_stages=7/13 stage=AccountHashing block=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:02.646680Z  INFO reth::node::events: Executing stage pipeline_stages=8/13 stage=StorageHashing from=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:42:02.955421Z  INFO reth::node::events: Stage finished executing pipeline_stages=8/13 stage=StorageHashing block=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:03.025128Z  INFO reth::node::events: Executing stage pipeline_stages=9/13 stage=MerkleExecute from=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:42:05.380910Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x2f789247b79a2be0059ea12382d792fc06069819250c742eb800956443064333 safe_block_hash=0xe685f840a128f083c3410d22a9d13
e63f68e026b69868990ecff78815d368a23 finalized_block_hash=0xd2c16b9b0c1d02e04070b4b847781adcfec6fd8c65ce2fb1724b65f0f5b7a80a status=Syncing
2023-06-28T09:42:08.656022Z  INFO reth::node::events: Stage finished executing pipeline_stages=9/13 stage=MerkleExecute block=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:08.877743Z  INFO reth::node::events: Executing stage pipeline_stages=10/13 stage=TransactionLookup from=17576508 checkpoint=100.0% eta=unknown
2023-06-28T09:42:09.010651Z  INFO net: Session established remote_addr=37.19.216.43:30000 client_version=Geth/v1.10.25-stable/linux-amd64/go1.17.10 peer_id=0xf7c13b7f0618cd3624fc25b98de910d6e0aaff5c50af4
9ea6bbc2f35b54dbdfbb6fa19132f9f4222a0d00ddef9f8f332fc21e81a4058ea478ae2736c6fbe0f4a total_active=17
2023-06-28T09:42:09.933771Z  INFO reth::node::events: Stage finished executing pipeline_stages=10/13 stage=TransactionLookup block=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:10.035522Z  INFO reth::node::events: Executing stage pipeline_stages=11/13 stage=IndexStorageHistory from=17576508 checkpoint=17576508 eta=unknown
2023-06-28T09:42:10.722324Z  INFO reth::node::events: Stage finished executing pipeline_stages=11/13 stage=IndexStorageHistory block=17576570 checkpoint=17576570 eta=unknown
2023-06-28T09:42:10.812086Z  INFO reth::node::events: Executing stage pipeline_stages=12/13 stage=IndexAccountHistory from=17576508 checkpoint=17576508 eta=unknown
2023-06-28T09:42:11.081899Z  INFO reth::node::events: Stage finished executing pipeline_stages=12/13 stage=IndexAccountHistory block=17576570 checkpoint=17576570 eta=unknown
2023-06-28T09:42:11.131493Z  INFO reth::node::events: Executing stage pipeline_stages=13/13 stage=Finish from=17576508 checkpoint=17576508 eta=unknown
2023-06-28T09:42:11.131516Z  INFO reth::node::events: Stage finished executing pipeline_stages=13/13 stage=Finish block=17576570 checkpoint=17576570 eta=unknown
2023-06-28T09:42:11.131748Z  INFO reth::node::events: Executing stage pipeline_stages=1/13 stage=Headers from=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:11.229431Z  INFO execute{stage=Headers}: sync::stages::headers: Received headers len=126
2023-06-28T09:42:11.240151Z  INFO reth::node::events: Stage finished executing pipeline_stages=1/13 stage=Headers block=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:11.241447Z  INFO reth::node::events: Executing stage pipeline_stages=2/13 stage=TotalDifficulty from=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:11.241553Z  INFO reth::node::events: Stage finished executing pipeline_stages=2/13 stage=TotalDifficulty block=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:11.241679Z  INFO reth::node::events: Executing stage pipeline_stages=3/13 stage=Bodies from=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:11.241690Z  INFO downloaders::bodies: Downloading bodies 17576571..=17576696 distance=125
2023-06-28T09:42:12.256307Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x0b9383ec3a5e210aa0d8608f5a7bcde7c67d3b82300d26e5a5d35d3e3672549c safe_block_hash=0x2f789247b79a2be0059ea12382d79
2fc06069819250c742eb800956443064333 finalized_block_hash=0xe685f840a128f083c3410d22a9d13e63f68e026b69868990ecff78815d368a23 status=Syncing
2023-06-28T09:42:12.363736Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x40d62988e101511315dac8d322220597a61580ad7dc60adf13d44d892b29116b safe_block_hash=0x2f789247b79a2be0059ea12382d79
2fc06069819250c742eb800956443064333 finalized_block_hash=0xe685f840a128f083c3410d22a9d13e63f68e026b69868990ecff78815d368a23 status=Syncing
2023-06-28T09:42:12.927189Z  INFO reth::node::events: Stage finished executing pipeline_stages=3/13 stage=Bodies block=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:12.956232Z  INFO execute{stage=SenderRecovery}: sync::stages::sender_recovery: Recovering senders tx_range=2013894224..=2013915103
2023-06-28T09:42:12.956240Z  INFO reth::node::events: Executing stage pipeline_stages=4/13 stage=SenderRecovery from=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:13.816641Z  INFO reth::node::events: Stage finished executing pipeline_stages=4/13 stage=SenderRecovery block=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:13.817400Z  INFO reth::node::events: Executing stage pipeline_stages=5/13 stage=Execution from=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:18.065522Z  INFO reth::node::events: Stage finished executing pipeline_stages=5/13 stage=Execution block=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:18.322022Z  INFO execute{stage=MerkleUnwind}: sync::stages::merkle::unwind: Stage is always skipped
2023-06-28T09:42:18.322039Z  INFO reth::node::events: Executing stage pipeline_stages=6/13 stage=MerkleUnwind from=17576570 checkpoint=17576570 eta=unknown
2023-06-28T09:42:18.322073Z  INFO reth::node::events: Stage finished executing pipeline_stages=6/13 stage=MerkleUnwind block=17576696 checkpoint=17576696 eta=unknown
2023-06-28T09:42:18.322174Z  INFO reth::node::events: Executing stage pipeline_stages=7/13 stage=AccountHashing from=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:18.868429Z  INFO reth::node::events: Stage finished executing pipeline_stages=7/13 stage=AccountHashing block=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:18.974115Z  INFO reth::node::events: Executing stage pipeline_stages=8/13 stage=StorageHashing from=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:19.504936Z  INFO reth::node::events: Stage finished executing pipeline_stages=8/13 stage=StorageHashing block=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:19.611590Z  INFO reth::node::events: Executing stage pipeline_stages=9/13 stage=MerkleExecute from=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:20.895801Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x0c3d8d8df6eaa2438737298a4c33f231c54ccb3701086e8b7291adc8ee514eae safe_block_hash=0xc539421bb54b8a147d24428f1f8aa
c5966707807b8c23534874dc3e1e9b46af8 finalized_block_hash=0x40d62988e101511315dac8d322220597a61580ad7dc60adf13d44d892b29116b status=Syncing
2023-06-28T09:42:23.374382Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x94fc6656b53ced1fc695867b5056d38be5e2e1aef6ea530eb03fca7c33acd382 safe_block_hash=0xc539421bb54b8a147d24428f1f8aa
c5966707807b8c23534874dc3e1e9b46af8 finalized_block_hash=0x40d62988e101511315dac8d322220597a61580ad7dc60adf13d44d892b29116b status=Syncing
2023-06-28T09:42:24.776208Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x591401b03683fc1af3ffd27ee4e29a490f4900c1ee4e96f6b70824b8ed2fae3b safe_block_hash=0xc539421bb54b8a147d24428f1f8aa
c5966707807b8c23534874dc3e1e9b46af8 finalized_block_hash=0x40d62988e101511315dac8d322220597a61580ad7dc60adf13d44d892b29116b status=Syncing
2023-06-28T09:42:28.300926Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e0 safe_block_hash=0xc93921ccc7bf16ea74441adbc2a44
ab1fa4a5f30b957f146906ba113061898b0 finalized_block_hash=0x0c3d8d8df6eaa2438737298a4c33f231c54ccb3701086e8b7291adc8ee514eae status=Syncing
2023-06-28T09:42:28.580065Z  INFO reth::cli: Status connected_peers=16 stage=MerkleExecute checkpoint=100.0% eta=unknown
2023-06-28T09:42:29.054375Z  INFO net: Session established remote_addr=67.3.134.26:30303 client_version=Nethermind/v1.17.3+671d60ad/linux-x64/dotnet7.0.4 peer_id=0x54496383e1bd8f9284a5d51f146ec97710aa44b
90692fe92140cdad017e5e31d785ffe5ef13831dd232a0d2b899b7a37d5ed1258859ac7653db1600c3f87cdd0 total_active=17
2023-06-28T09:42:30.467715Z  INFO reth::node::events: Stage finished executing pipeline_stages=9/13 stage=MerkleExecute block=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:30.786162Z  INFO reth::node::events: Executing stage pipeline_stages=10/13 stage=TransactionLookup from=17576570 checkpoint=100.0% eta=unknown
2023-06-28T09:42:32.402046Z  INFO reth::node::events: Stage finished executing pipeline_stages=10/13 stage=TransactionLookup block=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:32.521640Z  INFO reth::node::events: Executing stage pipeline_stages=11/13 stage=IndexStorageHistory from=17576570 checkpoint=17576570 eta=unknown
2023-06-28T09:42:33.728269Z  INFO reth::node::events: Stage finished executing pipeline_stages=11/13 stage=IndexStorageHistory block=17576696 checkpoint=17576696 eta=unknown
2023-06-28T09:42:33.865205Z  INFO reth::node::events: Executing stage pipeline_stages=12/13 stage=IndexAccountHistory from=17576570 checkpoint=17576570 eta=unknown
2023-06-28T09:42:34.367185Z  INFO reth::node::events: Stage finished executing pipeline_stages=12/13 stage=IndexAccountHistory block=17576696 checkpoint=17576696 eta=unknown
2023-06-28T09:42:34.464711Z  INFO reth::node::events: Executing stage pipeline_stages=13/13 stage=Finish from=17576570 checkpoint=17576570 eta=unknown
2023-06-28T09:42:34.464735Z  INFO reth::node::events: Stage finished executing pipeline_stages=13/13 stage=Finish block=17576696 checkpoint=17576696 eta=unknown
2023-06-28T09:42:34.465007Z  INFO reth::node::events: Executing stage pipeline_stages=1/13 stage=Headers from=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:34.567878Z  INFO execute{stage=Headers}: sync::stages::headers: Received headers len=188
023-06-28T09:42:34.583331Z  INFO reth::node::events: Stage finished executing pipeline_stages=1/13 stage=Headers block=17576884 checkpoint=100.0% eta=unknown                                             
2023-06-28T09:42:34.585338Z  INFO reth::node::events: Executing stage pipeline_stages=2/13 stage=TotalDifficulty from=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:34.585476Z  INFO reth::node::events: Stage finished executing pipeline_stages=2/13 stage=TotalDifficulty block=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:42:34.585595Z  INFO reth::node::events: Executing stage pipeline_stages=3/13 stage=Bodies from=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:34.585603Z  INFO downloaders::bodies: Downloading bodies 17576697..=17576884 distance=187
2023-06-28T09:42:35.148692Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x1139d164d335da71e6ae61154d372daa4156f41ca86a94b3c2c696d500e9830d safe_block_hash=0xc93921ccc7bf16ea74441adbc2a44
ab1fa4a5f30b957f146906ba113061898b0 finalized_block_hash=0x0c3d8d8df6eaa2438737298a4c33f231c54ccb3701086e8b7291adc8ee514eae status=Syncing
2023-06-28T09:42:35.824029Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xa42716f8dd3c1c9b9f5022237213eae16c5fdc4133e4923b1383e898debad569 safe_block_hash=0xc93921ccc7bf16ea74441adbc2a44
ab1fa4a5f30b957f146906ba113061898b0 finalized_block_hash=0x0c3d8d8df6eaa2438737298a4c33f231c54ccb3701086e8b7291adc8ee514eae status=Syncing
2023-06-28T09:42:37.152508Z  INFO reth::node::events: Stage finished executing pipeline_stages=3/13 stage=Bodies block=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:42:37.186344Z  INFO reth::node::events: Executing stage pipeline_stages=4/13 stage=SenderRecovery from=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:37.186355Z  INFO execute{stage=SenderRecovery}: sync::stages::sender_recovery: Recovering senders tx_range=2013915104..=2013945229
2023-06-28T09:42:38.356060Z  INFO reth::node::events: Stage finished executing pipeline_stages=4/13 stage=SenderRecovery block=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:42:38.357734Z  INFO reth::node::events: Executing stage pipeline_stages=5/13 stage=Execution from=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:39.768446Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xf4a70082712877d6e8ec46aff70ac017b414d9b20c5a769256644016e21a5c7c safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
2023-06-28T09:42:43.447533Z  INFO reth::node::events: Stage finished executing pipeline_stages=5/13 stage=Execution block=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:42:43.756340Z  INFO execute{stage=MerkleUnwind}: sync::stages::merkle::unwind: Stage is always skipped
2023-06-28T09:42:43.756348Z  INFO reth::node::events: Executing stage pipeline_stages=6/13 stage=MerkleUnwind from=17576696 checkpoint=17576696 eta=unknown
2023-06-28T09:42:43.756382Z  INFO reth::node::events: Stage finished executing pipeline_stages=6/13 stage=MerkleUnwind block=17576884 checkpoint=17576884 eta=unknown
2023-06-28T09:42:43.756476Z  INFO reth::node::events: Executing stage pipeline_stages=7/13 stage=AccountHashing from=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:43.757288Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xa4aebc5cd960e29d00c8e32128f830f90cf84f5c4cfccf7dfe55e47164a85e5c safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
2023-06-28T09:42:43.876088Z  INFO net: Session established remote_addr=3.209.45.79:30303 client_version=Geth/v1.12.0-stable-e501b3b0/linux-amd64/go1.20.4 peer_id=0x22a8232c3abc76a16ae9d6c3b164f98775fe226
f0917b0ca871128a74a8e9630b458460865bab457221f1d448dd9791d24c4e5d88786180ac185df813a68d4de total_active=18
2023-06-28T09:42:44.079746Z  INFO net: Session established remote_addr=18.138.108.67:30303 client_version=Geth/v1.12.0-stable-e501b3b0/linux-amd64/go1.20.4 peer_id=0xd860a01f9722d78051619d1e2351aba3f43f9
43f6f00718d1b9baa4101932a1f5011f16bb2b1bb35db20d6fe28fa0bf09636d26a87d31de9ec6203eeedb1f666 total_active=18
2023-06-28T09:42:44.315069Z  INFO reth::node::events: Stage finished executing pipeline_stages=7/13 stage=AccountHashing block=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:42:44.416246Z  INFO reth::node::events: Executing stage pipeline_stages=8/13 stage=StorageHashing from=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:45.061004Z  INFO reth::node::events: Stage finished executing pipeline_stages=8/13 stage=StorageHashing block=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:42:45.198867Z  INFO reth::node::events: Executing stage pipeline_stages=9/13 stage=MerkleExecute from=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:48.646285Z  INFO net: Session established remote_addr=141.95.172.76:30304 client_version=Geth/v1.11.6-stable-ea9e62ca/linux-amd64/go1.20.3 peer_id=0xbdf3ea9a3109ae2445d7daa5aec1372f2d89f
a22a17cc780dd54192b0b6f69e6705ca19f50f65b38fd8db66a7cdeebfdef007ea466e1d86b083e33090fb3b6b0 total_active=18
2023-06-28T09:42:48.876133Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x7e024574dd39652dba53da69480bc91d61d00a52bf088b00d36e2cd7f34a673d safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
2023-06-28T09:42:53.090852Z  INFO net: Session established remote_addr=35.193.253.37:1188 client_version=erigon/v2.46.0-dev-6755f7e1/linux-amd64/go1.20.5 peer_id=0xd18fd99e38be7fa64ec8ea145504d3a33cb3010
12043ac22e5a9c8d8e8f4a5821ca787f875935505a5e1c3c38ed9bbafad2ee3b8cd2a0809d8dda7d604d0cbbb total_active=19
2023-06-28T09:42:53.895208Z  INFO net: Session established remote_addr=15.204.197.7:30303 client_version=Geth/v1.11.5-stable/linux-amd64/go1.20.2 peer_id=0xba63e5f1331809d651ab33e58fe5b014ea02aaf643484cf
70ef744c10b591ca8513adbbf9a10048baa144e191f25d4e901c6470886b1856a990356c26c175615 total_active=20
2023-06-28T09:42:57.358032Z  INFO reth::node::events: Stage finished executing pipeline_stages=9/13 stage=MerkleExecute block=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:42:57.781042Z  INFO reth::node::events: Executing stage pipeline_stages=10/13 stage=TransactionLookup from=17576696 checkpoint=100.0% eta=unknown
2023-06-28T09:42:58.579827Z  INFO reth::cli: Status connected_peers=20 stage=TransactionLookup checkpoint=100.0% eta=unknown
2023-06-28T09:43:00.059643Z  INFO reth::node::events: Stage finished executing pipeline_stages=10/13 stage=TransactionLookup block=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:00.226331Z  INFO reth::node::events: Executing stage pipeline_stages=11/13 stage=IndexStorageHistory from=17576696 checkpoint=17576696 eta=unknown
2023-06-28T09:43:00.931013Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x1fca9dd12c8d0daba7e3e364225e85cc5725d09703a46264a2827b8a77bd9587 safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
2023-06-28T09:43:01.985152Z  INFO reth::node::events: Stage finished executing pipeline_stages=11/13 stage=IndexStorageHistory block=17576884 checkpoint=17576884 eta=unknown
2023-06-28T09:43:02.121812Z  INFO reth::node::events: Executing stage pipeline_stages=12/13 stage=IndexAccountHistory from=17576696 checkpoint=17576696 eta=unknown
2023-06-28T09:43:02.738752Z  INFO reth::node::events: Stage finished executing pipeline_stages=12/13 stage=IndexAccountHistory block=17576884 checkpoint=17576884 eta=unknown
2023-06-28T09:43:02.873929Z  INFO reth::node::events: Executing stage pipeline_stages=13/13 stage=Finish from=17576696 checkpoint=17576696 eta=unknown
2023-06-28T09:43:02.873953Z  INFO reth::node::events: Stage finished executing pipeline_stages=13/13 stage=Finish block=17576884 checkpoint=17576884 eta=unknown
2023-06-28T09:43:02.874257Z  INFO reth::node::events: Executing stage pipeline_stages=1/13 stage=Headers from=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:02.943759Z  INFO execute{stage=Headers}: sync::stages::headers: Received headers len=31
2023-06-28T09:43:02.946732Z  INFO reth::node::events: Stage finished executing pipeline_stages=1/13 stage=Headers block=17576915 checkpoint=100.0% eta=unknown
2023-06-28T09:43:02.947792Z  INFO reth::node::events: Executing stage pipeline_stages=2/13 stage=TotalDifficulty from=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:02.947810Z  INFO reth::node::events: Stage finished executing pipeline_stages=2/13 stage=TotalDifficulty block=17576915 checkpoint=100.0% eta=unknown
2023-06-28T09:43:02.947950Z  INFO reth::node::events: Executing stage pipeline_stages=3/13 stage=Bodies from=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:02.947955Z  INFO downloaders::bodies: Downloading bodies 17576885..=17576915 distance=30
2023-06-28T09:43:04.275227Z  INFO reth::node::events: Stage finished executing pipeline_stages=3/13 stage=Bodies block=17576915 checkpoint=100.0% eta=unknown
2023-06-28T09:43:04.300806Z  INFO execute{stage=SenderRecovery}: sync::stages::sender_recovery: Recovering senders tx_range=2013945230..=2013949678
2023-06-28T09:43:04.300806Z  INFO reth::node::events: Executing stage pipeline_stages=4/13 stage=SenderRecovery from=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:04.499811Z  INFO reth::node::events: Stage finished executing pipeline_stages=4/13 stage=SenderRecovery block=17576915 checkpoint=100.0% eta=unknown
2023-06-28T09:43:04.500501Z  INFO reth::node::events: Executing stage pipeline_stages=5/13 stage=Execution from=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:05.283899Z  INFO reth::node::events: Stage finished executing pipeline_stages=5/13 stage=Execution block=17576915 checkpoint=100.0% eta=unknown
2023-06-28T09:43:05.378059Z  INFO execute{stage=MerkleUnwind}: sync::stages::merkle::unwind: Stage is always skipped
2023-06-28T09:43:05.378083Z  INFO reth::node::events: Executing stage pipeline_stages=6/13 stage=MerkleUnwind from=17576884 checkpoint=17576884 eta=unknown
2023-06-28T09:43:05.378098Z  INFO reth::node::events: Stage finished executing pipeline_stages=6/13 stage=MerkleUnwind block=17576915 checkpoint=17576915 eta=unknown
2023-06-28T09:43:05.378214Z  INFO reth::node::events: Executing stage pipeline_stages=7/13 stage=AccountHashing from=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:05.465556Z  INFO reth::node::events: Stage finished executing pipeline_stages=7/13 stage=AccountHashing block=17576915 checkpoint=100.0% eta=unknown
2023-06-28T09:43:05.488278Z  INFO reth::node::events: Executing stage pipeline_stages=8/13 stage=StorageHashing from=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:05.886634Z  INFO reth::node::events: Stage finished executing pipeline_stages=8/13 stage=StorageHashing block=17576915 checkpoint=100.0% eta=unknown
2023-06-28T09:43:05.972688Z  INFO reth::node::events: Executing stage pipeline_stages=9/13 stage=MerkleExecute from=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:09.937846Z  INFO reth::node::events: Stage finished executing pipeline_stages=9/13 stage=MerkleExecute block=17576915 checkpoint=100.0% eta=unknown
2023-06-28T09:43:10.110096Z  INFO reth::node::events: Executing stage pipeline_stages=10/13 stage=TransactionLookup from=17576884 checkpoint=100.0% eta=unknown
2023-06-28T09:43:10.466333Z  INFO reth::node::events: Stage finished executing pipeline_stages=10/13 stage=TransactionLookup block=17576915 checkpoint=100.0% eta=unknown
2023-06-28T09:43:10.511098Z  INFO reth::node::events: Executing stage pipeline_stages=11/13 stage=IndexStorageHistory from=17576884 checkpoint=17576884 eta=unknown
2023-06-28T09:43:10.959773Z  INFO reth::node::events: Stage finished executing pipeline_stages=11/13 stage=IndexStorageHistory block=17576915 checkpoint=17576915 eta=unknown
2023-06-28T09:43:11.030965Z  INFO reth::node::events: Executing stage pipeline_stages=12/13 stage=IndexAccountHistory from=17576884 checkpoint=17576884 eta=unknown
2023-06-28T09:43:11.132925Z  INFO reth::node::events: Stage finished executing pipeline_stages=12/13 stage=IndexAccountHistory block=17576915 checkpoint=17576915 eta=unknown
2023-06-28T09:43:11.168435Z  INFO reth::node::events: Executing stage pipeline_stages=13/13 stage=Finish from=17576884 checkpoint=17576884 eta=unknown
2023-06-28T09:43:11.168447Z  INFO reth::node::events: Stage finished executing pipeline_stages=13/13 stage=Finish block=17576915 checkpoint=17576915 eta=unknown
2023-06-28T09:43:12.172512Z  INFO try_insert_validated_block{block=(17576995, 0x8b8ce34f3f5660f860ce15bed315963b873fd9ae77a6184326d79a37c5dba34a)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576946, 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e0) })
2023-06-28T09:43:12.173026Z  WARN make_canonical{block_hash=0x1fca9dd12c8d0daba7e3e364225e85cc5725d09703a46264a2827b8a77bd9587}: blockchain_tree: Block hash not found in block indices block_hash=0x1fca9d
d12c8d0daba7e3e364225e85cc5725d09703a46264a2827b8a77bd9587
2023-06-28T09:43:12.173044Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x1fca9dd12c8d0daba7e3e364225e85cc5725d09703a46264a2827b8a
77bd9587 }) state=ForkchoiceState { head_block_hash: 0x1fca9dd12c8d0daba7e3e364225e85cc5725d09703a46264a2827b8a77bd9587, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }
2023-06-28T09:43:12.173114Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x1fca9dd12c8d0daba7e3e364225e85cc5725d09703a46264a2827b8a77bd9587 safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
2023-06-28T09:43:12.318011Z  INFO try_insert_validated_block{block=(17576946, 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e0)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576945, 0xeba4e9a02159a928ec4aca73bdef7bb9b7747293bc7ac8566aa024136bb9f3d0) })
2023-06-28T09:43:12.318083Z  WARN make_canonical{block_hash=0x8b8ce34f3f5660f860ce15bed315963b873fd9ae77a6184326d79a37c5dba34a}: blockchain_tree: Block hash not found in block indices block_hash=0x8b8ce3
4f3f5660f860ce15bed315963b873fd9ae77a6184326d79a37c5dba34a                                                                                                                                                 
2023-06-28T09:43:12.318098Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x8b8ce34f3f5660f860ce15bed315963b873fd9ae77a6184326d79a37
c5dba34a }) state=ForkchoiceState { head_block_hash: 0x8b8ce34f3f5660f860ce15bed315963b873fd9ae77a6184326d79a37c5dba34a, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }                                                                                                              
2023-06-28T09:43:12.318193Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x8b8ce34f3f5660f860ce15bed315963b873fd9ae77a6184326d79a37c5dba34a safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing                                                                 
2023-06-28T09:43:12.379586Z  INFO try_insert_validated_block{block=(17576945, 0xeba4e9a02159a928ec4aca73bdef7bb9b7747293bc7ac8566aa024136bb9f3d0)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576944, 0x3c9e6808a541bbf6a06165d73dc4a5e1034da4fd960a3c88347b2595474179e2) })                                                                                                                       
2023-06-28T09:43:12.445754Z  INFO try_insert_validated_block{block=(17576944, 0x3c9e6808a541bbf6a06165d73dc4a5e1034da4fd960a3c88347b2595474179e2)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576943, 0xd279fb6b6e17102980c2f918476ec7cdab0f269b12ea3418b0589494403b0981) })                                                                                                                       
2023-06-28T09:43:12.514798Z  INFO try_insert_validated_block{block=(17576943, 0xd279fb6b6e17102980c2f918476ec7cdab0f269b12ea3418b0589494403b0981)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576942, 0x0ea2b9c9ff4d2437825307420539819747fea98f46286323197cd6b139e7c2df) })                                                                                                                       
2023-06-28T09:43:12.575581Z  INFO try_insert_validated_block{block=(17576942, 0x0ea2b9c9ff4d2437825307420539819747fea98f46286323197cd6b139e7c2df)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576941, 0x24b0d4e178ea4df9ecc21c74096b229084eaf5bd77667bcd5f805e53b6e4cd4b) })                                                                                                                       
2023-06-28T09:43:12.726940Z  INFO try_insert_validated_block{block=(17576941, 0x24b0d4e178ea4df9ecc21c74096b229084eaf5bd77667bcd5f805e53b6e4cd4b)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576940, 0x1ca0a133247d96acebb4ea65c75e3118bcc344033699631fa55fd25fa3812ec4) })
2023-06-28T09:43:12.790507Z  INFO try_insert_validated_block{block=(17576940, 0x1ca0a133247d96acebb4ea65c75e3118bcc344033699631fa55fd25fa3812ec4)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576939, 0x65e8aa57090b69a3ef895a6e12368b1bf98f347d984cde720d83048a4f3bc3e7) })
...

2023-06-28T09:43:24.261580Z  INFO try_insert_validated_block{block=(17576996, 0xc398f78f99813e0d62e462dba2c8d79826f356a1e5ffc9f299c0d8bf9a1c10c0)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576939, 0x65e8aa57090b69a3ef895a6e12368b1bf98f347d984cde720d83048a4f3bc3e7) })
2023-06-28T09:43:24.399901Z  WARN make_canonical{block_hash=0xc398f78f99813e0d62e462dba2c8d79826f356a1e5ffc9f299c0d8bf9a1c10c0}: blockchain_tree: Block hash not found in block indices block_hash=0xc398f7
8f99813e0d62e462dba2c8d79826f356a1e5ffc9f299c0d8bf9a1c10c0
2023-06-28T09:43:24.399926Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0xc398f78f99813e0d62e462dba2c8d79826f356a1e5ffc9f299c0d8bf
9a1c10c0 }) state=ForkchoiceState { head_block_hash: 0xc398f78f99813e0d62e462dba2c8d79826f356a1e5ffc9f299c0d8bf9a1c10c0, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }
2023-06-28T09:43:24.400005Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xc398f78f99813e0d62e462dba2c8d79826f356a1e5ffc9f299c0d8bf9a1c10c0 safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
...

2023-06-28T09:43:36.041525Z  INFO try_insert_validated_block{block=(17576997, 0x2e554b84769a1724b602dff97e40ee8d98beade102531046404c3c2cba6a6834)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576939, 0x65e8aa57090b69a3ef895a6e12368b1bf98f347d984cde720d83048a4f3bc3e7) })
2023-06-28T09:43:36.175236Z  WARN make_canonical{block_hash=0x2e554b84769a1724b602dff97e40ee8d98beade102531046404c3c2cba6a6834}: blockchain_tree: Block hash not found in block indices block_hash=0x2e554b
84769a1724b602dff97e40ee8d98beade102531046404c3c2cba6a6834
2023-06-28T09:43:36.175260Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x2e554b84769a1724b602dff97e40ee8d98beade102531046404c3c2c
ba6a6834 }) state=ForkchoiceState { head_block_hash: 0x2e554b84769a1724b602dff97e40ee8d98beade102531046404c3c2cba6a6834, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }
2023-06-28T09:43:36.175337Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x2e554b84769a1724b602dff97e40ee8d98beade102531046404c3c2cba6a6834 safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
2023-06-28T09:43:48.308565Z  INFO try_insert_validated_block{block=(17576998, 0x37ce185da59bee722282b2b4ce75e83fdfdb3bbc8b06887710d3ef9a805811fc)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576939, 0x65e8aa57090b69a3ef895a6e12368b1bf98f347d984cde720d83048a4f3bc3e7) })
2023-06-28T09:43:48.435202Z  WARN make_canonical{block_hash=0x37ce185da59bee722282b2b4ce75e83fdfdb3bbc8b06887710d3ef9a805811fc}: blockchain_tree: Block hash not found in block indices block_hash=0x37ce18
5da59bee722282b2b4ce75e83fdfdb3bbc8b06887710d3ef9a805811fc
2023-06-28T09:43:48.435233Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x37ce185da59bee722282b2b4ce75e83fdfdb3bbc8b06887710d3ef9a
805811fc }) state=ForkchoiceState { head_block_hash: 0x37ce185da59bee722282b2b4ce75e83fdfdb3bbc8b06887710d3ef9a805811fc, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }
...

2023-06-28T09:44:00.134143Z  INFO try_insert_validated_block{block=(17576999, 0xca91a0d8403b054e576c18ad744d4f85de4a22a0e016a46fa36f20d5a058470e)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576939, 0x65e8aa57090b69a3ef895a6e12368b1bf98f347d984cde720d83048a4f3bc3e7) })
2023-06-28T09:44:00.268650Z  WARN make_canonical{block_hash=0xca91a0d8403b054e576c18ad744d4f85de4a22a0e016a46fa36f20d5a058470e}: blockchain_tree: Block hash not found in block indices block_hash=0xca91a0
d8403b054e576c18ad744d4f85de4a22a0e016a46fa36f20d5a058470e
2023-06-28T09:44:00.268691Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0xca91a0d8403b054e576c18ad744d4f85de4a22a0e016a46fa36f20d5
a058470e }) state=ForkchoiceState { head_block_hash: 0xca91a0d8403b054e576c18ad744d4f85de4a22a0e016a46fa36f20d5a058470e, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }
2023-06-28T09:44:00.268774Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xca91a0d8403b054e576c18ad744d4f85de4a22a0e016a46fa36f20d5a058470e safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
2023-06-28T09:44:08.852627Z  INFO net: Session established remote_addr=84.17.41.31:33160 client_version=Geth/v1.10.25-stable/linux-amd64/go1.17.10 peer_id=0x5e51901a2def48808ae2509505513198b398f73705f943
4b56d25b52dc53bb996e092f2bab79cfcf062eff175d2060a8d22b958c6e61a9fbff2cd08d2e0fbeaf total_active=38
2023-06-28T09:44:13.036410Z  INFO try_insert_validated_block{block=(17577000, 0xe83a81caf91475153940a5e08b0152f1899da38719bb57f66eeff64982680344)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576939, 0x65e8aa57090b69a3ef895a6e12368b1bf98f347d984cde720d83048a4f3bc3e7) })
2023-06-28T09:44:13.121552Z  WARN make_canonical{block_hash=0xe83a81caf91475153940a5e08b0152f1899da38719bb57f66eeff64982680344}: blockchain_tree: Block hash not found in block indices block_hash=0xe83a81
caf91475153940a5e08b0152f1899da38719bb57f66eeff64982680344
2023-06-28T09:44:13.121581Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0xe83a81caf91475153940a5e08b0152f1899da38719bb57f66eeff649
82680344 }) state=ForkchoiceState { head_block_hash: 0xe83a81caf91475153940a5e08b0152f1899da38719bb57f66eeff64982680344, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }
2023-06-28T09:44:13.121667Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xe83a81caf91475153940a5e08b0152f1899da38719bb57f66eeff64982680344 safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing

...

2023-06-28T09:44:24.867937Z  INFO try_insert_validated_block{block=(17577001, 0x80276b30859de1e8e6bb0732cbeb396c459510689f0f4d4f9af3e902ed3c17f3)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576939, 0x65e8aa57090b69a3ef895a6e12368b1bf98f347d984cde720d83048a4f3bc3e7) })
2023-06-28T09:44:24.948517Z  WARN make_canonical{block_hash=0x80276b30859de1e8e6bb0732cbeb396c459510689f0f4d4f9af3e902ed3c17f3}: blockchain_tree: Block hash not found in block indices block_hash=0x80276b
30859de1e8e6bb0732cbeb396c459510689f0f4d4f9af3e902ed3c17f3
2023-06-28T09:44:24.948547Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x80276b30859de1e8e6bb0732cbeb396c459510689f0f4d4f9af3e902
ed3c17f3 }) state=ForkchoiceState { head_block_hash: 0x80276b30859de1e8e6bb0732cbeb396c459510689f0f4d4f9af3e902ed3c17f3, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }
2023-06-28T09:44:24.948622Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x80276b30859de1e8e6bb0732cbeb396c459510689f0f4d4f9af3e902ed3c17f3 safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
2023-06-28T09:44:28.579873Z  INFO reth::cli: Status connected_peers=42 latest_block=17576412
...

2023-06-28T09:44:36.171913Z  INFO try_insert_validated_block{block=(17577002, 0x5f14ce85632b8d5392dca1600ca902364e17477076b18a640c8cdf1eefe4ee3e)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576939, 0x65e8aa57090b69a3ef895a6e12368b1bf98f347d984cde720d83048a4f3bc3e7) })
2023-06-28T09:44:36.256268Z  WARN make_canonical{block_hash=0x5f14ce85632b8d5392dca1600ca902364e17477076b18a640c8cdf1eefe4ee3e}: blockchain_tree: Block hash not found in block indices block_hash=0x5f14ce
85632b8d5392dca1600ca902364e17477076b18a640c8cdf1eefe4ee3e
2023-06-28T09:44:36.256288Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x5f14ce85632b8d5392dca1600ca902364e17477076b18a640c8cdf1e
efe4ee3e }) state=ForkchoiceState { head_block_hash: 0x5f14ce85632b8d5392dca1600ca902364e17477076b18a640c8cdf1eefe4ee3e, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }
2023-06-28T09:44:36.256352Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x5f14ce85632b8d5392dca1600ca902364e17477076b18a640c8cdf1eefe4ee3e safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing
...

2023-06-28T09:44:49.884103Z  INFO try_insert_validated_block{block=(17577003, 0xfa0391ec3d20f503e727a3d2ae3361fba97c277df661bb3b49b94eeb4896fcde)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17576939, 0x65e8aa57090b69a3ef895a6e12368b1bf98f347d984cde720d83048a4f3bc3e7) })
2023-06-28T09:44:49.965783Z  WARN make_canonical{block_hash=0xfa0391ec3d20f503e727a3d2ae3361fba97c277df661bb3b49b94eeb4896fcde}: blockchain_tree: Block hash not found in block indices block_hash=0xfa0391
ec3d20f503e727a3d2ae3361fba97c277df661bb3b49b94eeb4896fcde
2023-06-28T09:44:49.965803Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0xfa0391ec3d20f503e727a3d2ae3361fba97c277df661bb3b49b94eeb
4896fcde }) state=ForkchoiceState { head_block_hash: 0xfa0391ec3d20f503e727a3d2ae3361fba97c277df661bb3b49b94eeb4896fcde, safe_block_hash: 0xeb19e2395a07689075c2072c023a3ef95b8eec41177d17532a573a79062e69e
0, finalized_block_hash: 0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 }
2023-06-28T09:44:49.965866Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xfa0391ec3d20f503e727a3d2ae3361fba97c277df661bb3b49b94eeb4896fcde safe_block_hash=0xeb19e2395a07689075c2072c023a3
ef95b8eec41177d17532a573a79062e69e0 finalized_block_hash=0xc93921ccc7bf16ea74441adbc2a44ab1fa4a5f30b957f146906ba113061898b0 status=Syncing

...

@mattsse
Copy link
Collaborator

mattsse commented Jun 28, 2023

this warning is expected if the node didn't receive the payload and therefore can't make the new FCU.head canonical

@Gaon3
Copy link
Contributor Author

Gaon3 commented Jun 28, 2023

If it's expected, is there something to do to fix it ?

@mattsse
Copy link
Collaborator

mattsse commented Jun 28, 2023

fixing what exactly?
the missing blocks are downloaded

@rakita
Copy link
Collaborator

rakita commented Jun 28, 2023 •
edited
Loading

wanted to post log file but I have an old bug where nothing is ever written in /.cache/reth/logs/reth.log despite running RETH_LOG=info reth node --log.persistent. I pulled latest commit, ran the node and here what I retrived: (I deliberately not copied chunk of messages related to peers, '...')

Ty for logs.
Omitting peers data seems like a good feature request if you want to make one :)

That warning shouldn't be a warning, it is flow that can happen, it just means that we need to fetch the blocks that node does not have.

@mattsse do we have any logs that tell us that we reverse download the block to fill the gap?

@mattsse
Copy link
Collaborator

mattsse commented Jun 28, 2023

That warning shouldn't be a warning, it is flow that can happen, it just means that we need to fetch the blocks that node does not have.

debatable whether this should be a warning or info

do we have any logs that tell us that we reverse download the block to fill the gap?

I think not on info, but we should highlight this

@Gaon3
Copy link
Contributor Author

Gaon3 commented Jun 28, 2023 •
edited
Loading

fixing what exactly? the missing blocks are downloaded

Here is what happens after I restart the node:
Catch up to latest block, try to insert a bunch of blocks then loop trying to insert a specific block. RPC calls from CL client to node fails to such an extent that Reth sends me a warning telling me to launch a beacon client because it hasn't seen it in a while (this time I shut down the node too early to have this message).

2023-06-28T10:49:16.022157Z  INFO reth::node::events: Stage finished executing pipeline_stages=13/13 stage=Finish block=17577232 checkpoint=17577232 eta=unknown                                           
2023-06-28T10:49:16.259691Z  INFO net: Session established remote_addr=88.228.94.46:50303 client_version=Nethermind/v1.17.3+671d60ad/linux-x64/dotnet7.0.4 peer_id=0xe6a47d7370863eaac9c29fa1f48ac4efc069aa
1700c72662c264b0066ee6a026c48e40dbcab2e65aa255c510ac6797a263a7aba2b16cd37e808446db44f4e8af total_active=51                                                                                                 
2023-06-28T10:49:17.967607Z  WARN make_canonical{block_hash=0x145ed0c92a15dc55dc3adb8667368c36bbd46b91c22bd456988a99b025a2ac7b}: blockchain_tree: Block hash not found in block indices block_hash=0x145ed0
c92a15dc55dc3adb8667368c36bbd46b91c22bd456988a99b025a2ac7b                                                                                                                                                 
2023-06-28T10:49:17.967637Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0x145ed0c92a15dc55dc3adb8667368c36bbd46b91c22bd456988a99b0
25a2ac7b }) state=ForkchoiceState { head_block_hash: 0x145ed0c92a15dc55dc3adb8667368c36bbd46b91c22bd456988a99b025a2ac7b, safe_block_hash: 0x224b4a2d89356a352829ac40c584c6c9fba581f61ec4b376e90a8c0d306ed72
b, finalized_block_hash: 0xafbe8531560a21ac9459487288d9f73f2c175ed97e8973a61f126da3bbacc5be }                                                                                                              
2023-06-28T10:49:17.967727Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x145ed0c92a15dc55dc3adb8667368c36bbd46b91c22bd456988a99b025a2ac7b safe_block_hash=0x224b4a2d89356a352829ac40c584c
6c9fba581f61ec4b376e90a8c0d306ed72b finalized_block_hash=0xafbe8531560a21ac9459487288d9f73f2c175ed97e8973a61f126da3bbacc5be status=Syncing                                                                 
2023-06-28T10:49:18.185992Z  INFO try_insert_validated_block{block=(17577263, 0x014f8320dab9456f8876fb2944ff542140d0a08c571916f6389ce4e807480f5c)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577262, 0x605e124e830a09fb907cc10001859feeb80e83ede327836177c67bfe0c8ef307) })                                                                                                                       
2023-06-28T10:49:18.533150Z  INFO try_insert_validated_block{block=(17577262, 0x605e124e830a09fb907cc10001859feeb80e83ede327836177c67bfe0c8ef307)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577261, 0xbb12e78edcaba7922a31ba5ebc64efbb9a783519b6ae5a3fed1b5150a612a39a) })                                                                                                                       
2023-06-28T10:49:18.703346Z  INFO try_insert_validated_block{block=(17577261, 0xbb12e78edcaba7922a31ba5ebc64efbb9a783519b6ae5a3fed1b5150a612a39a)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577260, 0x53cb5f5e153e69dd8e776d66ffc5a89e3307bf41db0ef760ece0be773150fdd6) })                                                                                                                       
2023-06-28T10:49:19.046627Z  INFO try_insert_validated_block{block=(17577260, 0x53cb5f5e153e69dd8e776d66ffc5a89e3307bf41db0ef760ece0be773150fdd6)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577259, 0xff73206caf253336db61cea87e954ed94b877c363026ec6cb7ee473cfbfc6f2c) })                                                                                                                       
2023-06-28T10:49:19.209472Z  INFO try_insert_validated_block{block=(17577259, 0xff73206caf253336db61cea87e954ed94b877c363026ec6cb7ee473cfbfc6f2c)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577258, 0x385de25396213709e8a03e120e9b0b2d29941cd3042fd88303a79dded5740cf3) })                                                                                                                       
2023-06-28T10:49:19.523678Z  INFO try_insert_validated_block{block=(17577258, 0x385de25396213709e8a03e120e9b0b2d29941cd3042fd88303a79dded5740cf3)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577257, 0x529434e1f04ad1acf34a2a44b134ff1a680d3afcf744cd74bde58cf0a744bc39) })                                                                                                                       
2023-06-28T10:49:19.847150Z  INFO try_insert_validated_block{block=(17577257, 0x529434e1f04ad1acf34a2a44b134ff1a680d3afcf744cd74bde58cf0a744bc39)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577256, 0x018765599f1d5be6dc3af22be1af4c733206935ff817a113ded73622fda39fc3) })                                                                                                                       
2023-06-28T10:49:19.983255Z  INFO try_insert_validated_block{block=(17577256, 0x018765599f1d5be6dc3af22be1af4c733206935ff817a113ded73622fda39fc3)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577255, 0x66a61f6dcc27fec046d8837c86ff1af7ac0cafff77cac2449184a5d9dad68376) })                                                                                                                       
2023-06-28T10:49:20.266898Z  INFO try_insert_validated_block{block=(17577255, 0x66a61f6dcc27fec046d8837c86ff1af7ac0cafff77cac2449184a5d9dad68376)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577254, 0x4181460b89150b962db35f84363517affaefd26b8131f5bf4a627f4a43186096) })                
2023-06-28T10:49:20.527078Z  INFO try_insert_validated_block{block=(17577254, 0x4181460b89150b962db35f84363517affaefd26b8131f5bf4a627f4a43186096)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577253, 0xe2b808f94ec86f949c167de938679c2de3aa1b27b28cefda61c6b4d320460835) })
2023-06-28T10:49:20.807954Z  INFO try_insert_validated_block{block=(17577253, 0xe2b808f94ec86f949c167de938679c2de3aa1b27b28cefda61c6b4d320460835)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577252, 0x52faa5cdc00818bfe270ecf563ce075d8f135fa2d9a609ec5126c419f76194cd) })
2023-06-28T10:49:21.104760Z  INFO try_insert_validated_block{block=(17577252, 0x52faa5cdc00818bfe270ecf563ce075d8f135fa2d9a609ec5126c419f76194cd)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577251, 0xd3b5fe70fbe54622f84c9e12239fb6d57034c2d76167f20f7367a145004bea81) })
2023-06-28T10:49:21.356910Z  INFO try_insert_validated_block{block=(17577251, 0xd3b5fe70fbe54622f84c9e12239fb6d57034c2d76167f20f7367a145004bea81)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577250, 0xb8093691eb75c009f40b6f046e97c711ee6d028e85300b707b91475478e641b7) })
2023-06-28T10:49:21.613338Z  INFO try_insert_validated_block{block=(17577250, 0xb8093691eb75c009f40b6f046e97c711ee6d028e85300b707b91475478e641b7)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577249, 0xff31d93bccb286524173fa13e1c57c95f62c55e74fc71efebca045d7e6cd616b) })
2023-06-28T10:49:21.902908Z  INFO try_insert_validated_block{block=(17577249, 0xff31d93bccb286524173fa13e1c57c95f62c55e74fc71efebca045d7e6cd616b)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577248, 0xbee6485bdfa647b5d4b00ca35043342335c3b761c8c6b995203769630804e11a) })
2023-06-28T10:49:22.175525Z  INFO try_insert_validated_block{block=(17577248, 0xbee6485bdfa647b5d4b00ca35043342335c3b761c8c6b995203769630804e11a)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577247, 0x96a6bcf46021b072a6d890e2052800d6031fddc97aa9c1d2b3ef3910a536cd3b) })
2023-06-28T10:49:22.434541Z  INFO try_insert_validated_block{block=(17577247, 0x96a6bcf46021b072a6d890e2052800d6031fddc97aa9c1d2b3ef3910a536cd3b)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577246, 0x67dfc3e417fba9cb135461c86fdbcb01d90ac36fb2cf658ebbbfb7dca1af6f4b) })
2023-06-28T10:49:22.596473Z  INFO try_insert_validated_block{block=(17577246, 0x67dfc3e417fba9cb135461c86fdbcb01d90ac36fb2cf658ebbbfb7dca1af6f4b)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577245, 0x396eecf677594d7e13b84fd29e3a2b0743a99bd9ce1be129ca3f23af7031a30b) })
2023-06-28T10:49:22.891595Z  INFO try_insert_validated_block{block=(17577245, 0x396eecf677594d7e13b84fd29e3a2b0743a99bd9ce1be129ca3f23af7031a30b)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577244, 0x62c0b21bb8e252c113973645ed3fb9f79c5bc38a815cc81bc58e88ea1ad82fe0) })
2023-06-28T10:49:23.159295Z  INFO try_insert_validated_block{block=(17577244, 0x62c0b21bb8e252c113973645ed3fb9f79c5bc38a815cc81bc58e88ea1ad82fe0)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577243, 0x960a9fcfcd41e7f064be2282d25620e95719245b7ce2bcae583b3cb805299a4c) })
2023-06-28T10:49:23.410860Z  INFO try_insert_validated_block{block=(17577243, 0x960a9fcfcd41e7f064be2282d25620e95719245b7ce2bcae583b3cb805299a4c)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577242, 0x7a55b4658fb415bf415ad041375210afa4398de8d7e38141ab679a711f0fcd42) })
2023-06-28T10:49:23.655372Z  INFO try_insert_validated_block{block=(17577242, 0x7a55b4658fb415bf415ad041375210afa4398de8d7e38141ab679a711f0fcd42)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577241, 0xcae779d1584a9fc0b3559a7fe2a17e7c5b1c4be4967136ccd7be20a3567e15f5) })
2023-06-28T10:49:23.912577Z  INFO try_insert_validated_block{block=(17577241, 0xcae779d1584a9fc0b3559a7fe2a17e7c5b1c4be4967136ccd7be20a3567e15f5)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577240, 0x7d62d7af826dc0abb1f079cd17b047b3d1efa382eb865e3c996d9858a8941829) })
2023-06-28T10:49:24.163383Z  INFO try_insert_validated_block{block=(17577240, 0x7d62d7af826dc0abb1f079cd17b047b3d1efa382eb865e3c996d9858a8941829)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577239, 0x47a59aaf46a2e27df0bef3287eacf24c73adcf1dec07a0cfc063f19e48f8cfb6) })
2023-06-28T10:49:24.408620Z  INFO try_insert_validated_block{block=(17577239, 0x47a59aaf46a2e27df0bef3287eacf24c73adcf1dec07a0cfc063f19e48f8cfb6)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577238, 0x65f9f59a2d53ded33bb857879c2a6dd837d25fd1bd373b9bc6b21466fcfbc785) })
2023-06-28T10:49:24.542072Z  INFO try_insert_validated_block{block=(17577325, 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577238, 0x65f9f59a2d53ded33bb857879c2a6dd837d25fd1bd373b9bc6b21466fcfbc785) })
2023-06-28T10:49:24.560109Z  INFO try_insert_validated_block{block=(17577238, 0x65f9f59a2d53ded33bb857879c2a6dd837d25fd1bd373b9bc6b21466fcfbc785)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577237, 0x4116071bf9aba11d156b88a289a57bb9221b4ef3202c2081a7a0ce83263bb27c) })
2023-06-28T10:49:24.642084Z  WARN make_canonical{block_hash=0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488}: blockchain_tree: Block hash not found in block indices block_hash=0xa9a4d3
4b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488
2023-06-28T10:49:24.642100Z  WARN consensus::engine: Failed to canonicalize the head hash error=Execution(BlockHashNotFoundInChain { block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703c
adb58488 }) state=ForkchoiceState { head_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, safe_block_hash: 0x224b4a2d89356a352829ac40c584c6c9fba581f61ec4b376e90a8c0d306ed72
b, finalized_block_hash: 0xafbe8531560a21ac9459487288d9f73f2c175ed97e8973a61f126da3bbacc5be }
2023-06-28T10:49:24.642158Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488 safe_block_hash=0x224b4a2d89356a352829ac40c584c
6c9fba581f61ec4b376e90a8c0d306ed72b finalized_block_hash=0xafbe8531560a21ac9459487288d9f73f2c175ed97e8973a61f126da3bbacc5be status=Syncing
2023-06-28T10:49:24.812888Z  INFO try_insert_validated_block{block=(17577237, 0x4116071bf9aba11d156b88a289a57bb9221b4ef3202c2081a7a0ce83263bb27c)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577236, 0xf06a9705fadc4e1f348bc4866db8d017a4991a389036c13db6a1b9a61c3cead5) })
2023-06-28T10:49:25.061258Z  INFO try_insert_validated_block{block=(17577236, 0xf06a9705fadc4e1f348bc4866db8d017a4991a389036c13db6a1b9a61c3cead5)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577235, 0x9c4989d0b60a53f8e6dc0cfc827c68a25e1c9d44ecc728809f049d17913e2a57) })
2023-06-28T10:49:25.108484Z  INFO reth::cli: Status connected_peers=51 latest_block=17577168
2023-06-28T10:49:25.322715Z  INFO try_insert_validated_block{block=(17577235, 0x9c4989d0b60a53f8e6dc0cfc827c68a25e1c9d44ecc728809f049d17913e2a57)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577234, 0x6d7441b369694704fe70c7d53fa340ccef3af0abc1a0fa18a972907c92030433) })
2023-06-28T10:49:25.517566Z  INFO try_insert_validated_block{block=(17577234, 0x6d7441b369694704fe70c7d53fa340ccef3af0abc1a0fa18a972907c92030433)}: blockchain_tree: return=Ok(Disconnected { missing_paren
t: (17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3) })
2023-06-28T10:49:26.122118Z  INFO try_insert_validated_block{block=(17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3)}:try_append_canonical_chain:try_insert_validated_block{bl
ock=(17577234, 0x6d7441b369694704fe70c7d53fa340ccef3af0abc1a0fa18a972907c92030433)}: blockchain_tree: return=Ok(Valid)
2023-06-28T10:49:26.469080Z  INFO try_insert_validated_block{block=(17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3)}:try_append_canonical_chain:try_insert_validated_block{bl
ock=(17577235, 0x9c4989d0b60a53f8e6dc0cfc827c68a25e1c9d44ecc728809f049d17913e2a57)}: blockchain_tree: return=Ok(Valid)
2023-06-28T10:49:27.286757Z  INFO try_insert_validated_block{block=(17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3)}:try_append_canonical_chain:try_insert_validated_block{bl
ock=(17577236, 0xf06a9705fadc4e1f348bc4866db8d017a4991a389036c13db6a1b9a61c3cead5)}: blockchain_tree: return=Ok(Valid)
2023-06-28T10:49:27.941459Z  INFO try_insert_validated_block{block=(17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3)}:try_append_canonical_chain:try_insert_validated_block{bl
ock=(17577237, 0x4116071bf9aba11d156b88a289a57bb9221b4ef3202c2081a7a0ce83263bb27c)}: blockchain_tree: return=Ok(Valid)
2023-06-28T10:49:28.739578Z  INFO try_insert_validated_block{block=(17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3)}:try_append_canonical_chain:try_insert_validated_block{bl
ock=(17577238, 0x65f9f59a2d53ded33bb857879c2a6dd837d25fd1bd373b9bc6b21466fcfbc785)}: blockchain_tree: return=Ok(Valid)
2023-06-28T10:49:29.669646Z  INFO try_insert_validated_block{block=(17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3)}:try_append_canonical_chain:try_insert_validated_block{bl
ock=(17577239, 0x47a59aaf46a2e27df0bef3287eacf24c73adcf1dec07a0cfc063f19e48f8cfb6)}: blockchain_tree: return=Ok(Valid)
2023-06-28T10:49:30.700751Z  INFO try_insert_validated_block{block=(17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3)}:try_append_canonical_chain:try_insert_validated_block{bl
ock=(17577240, 0x7d62d7af826dc0abb1f079cd17b047b3d1efa382eb865e3c996d9858a8941829)}: blockchain_tree: return=Ok(Valid)
2023-06-28T10:49:32.006299Z  INFO try_insert_validated_block{block=(17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3)}:try_append_canonical_chain:try_insert_validated_block{bl
ock=(17577241, 0xcae779d1584a9fc0b3559a7fe2a17e7c5b1c4be4967136ccd7be20a3567e15f5)}: blockchain_tree: return=Ok(Valid)
2023-06-28T10:49:33.319780Z  INFO try_insert_validated_block{block=(17577233, 0x3f1abb500e4c8dff196324e6b20bf10ba8907eb9f5e05907d991397a737c0fd3)}:try_append_canonical_chain:try_insert_validated_block{bl
ock=(17577242, 0x7a55b4658fb415bf415ad041375210afa4398de8d7e38141ab679a711f0fcd42)}: blockchain_tree: return=Ok(Valid)

etc... until it reaches block 17577233 and loop trying to insert it.

CL client logs:

Jun 28 10:50:06.508 CRIT Failed to update execution head         error: ExecutionForkChoiceUpdateFailed(EngineError(Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_
be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) })), service: beacon
Jun 28 10:50:08.000 WARN Did not advance head state              reason: Err(HeadMissingFromSnapshotCache(0x7903ccf1285e9694825e17871472d2be9ea2c0cd79d2277d4c8daef467749948)), service: state_advance     
Jun 28 10:50:17.001 INFO Downloading historical blocks           est_time: 18 hrs 9 mins, speed: 16.00 slots/sec, distance: 1045601 slots (20 weeks 5 days), service: slot_notifier                        
Jun 28 10:50:17.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier                       
Jun 28 10:50:17.001 INFO Synced                                  slot: 6760449, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 79, service: slot_notifier                                                                                                                                                                       Jun 28 10:50:19.176 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec                                                                                     
Jun 28 10:50:19.176 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec                                                                 
Jun 28 10:50:20.001 WARN Did not advance head state              reason: Err(HeadMissingFromSnapshotCache(0x7903ccf1285e9694825e17871472d2be9ea2c0cd79d2277d4c8daef467749948)), service: state_advance     
Jun 28 10:50:29.000 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier                                                                                                                             
Jun 28 10:50:29.000 INFO Synced                                  slot: 6760450, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified), peers: 80, service: slot_notifier                                                                                                                                                                       
Jun 28 10:50:29.374 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec                                                                                     
Jun 28 10:50:29.374 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec                                                                      
Jun 28 10:50:41.000 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier                                                                                                                             
Jun 28 10:50:41.000 INFO Synced                                  slot: 6760451, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified), peers: 82, service: slot_notifier                                                                                                                                                                       
Jun 28 10:50:42.299 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec                                                                                     
Jun 28 10:50:42.299 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec                                                                      
Jun 28 10:50:53.000 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier                                                                                                                             
Jun 28 10:50:53.000 INFO Synced                                  slot: 6760452, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified), peers: 81, service: slot_notifier                                                                                                                                                                       
Jun 28 10:50:54.446 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec                                                                                     
Jun 28 10:50:54.446 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec                                                                      
Jun 28 10:51:05.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier                                     
Jun 28 10:51:05.835 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:05.835 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:17.001 INFO Downloading historical blocks           est_time: 18 hrs 8 mins, speed: 16.00 slots/sec, distance: 1044641 slots (20 weeks 5 days), service: slot_notifier
Jun 28 10:51:17.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier
Jun 28 10:51:17.001 INFO Synced                                  slot: 6760454, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 80, service: slot_notifier
Jun 28 10:51:18.051 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:18.051 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:29.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier
Jun 28 10:51:29.001 INFO Synced                                  slot: 6760455, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 79, service: slot_notifier
Jun 28 10:51:31.135 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:31.135 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:40.802 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:40.802 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:41.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier
Jun 28 10:51:41.001 INFO Synced                                  slot: 6760456, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 78, service: slot_notifier
Jun 28 10:51:50.479 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:50.479 WARN ErJun 28 10:50:06.508 CRIT Failed to update execution head         error: ExecutionForkChoiceUpdateFailed(EngineError(Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_
be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) })), service: beacon
Jun 28 10:50:08.000 WARN Did not advance head state              reason: Err(HeadMissingFromSnapshotCache(0x7903ccf1285e9694825e17871472d2be9ea2c0cd79d2277d4c8daef467749948)), service: state_advance     
Jun 28 10:50:17.001 INFO Downloading historical blocks           est_time: 18 hrs 9 mins, speed: 16.00 slots/sec, distance: 1045601 slots (20 weeks 5 days), service: slot_notifier                        
Jun 28 10:50:17.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier                       
Jun 28 10:50:17.001 INFO Synced                                  slot: 6760449, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 79, service: slot_notifier                                                                                                                                                                       Jun 28 10:50:19.176 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec                                                                                     
Jun 28 10:50:19.176 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec                                                                 
Jun 28 10:50:20.001 WARN Did not advance head state              reason: Err(HeadMissingFromSnapshotCache(0x7903ccf1285e9694825e17871472d2be9ea2c0cd79d2277d4c8daef467749948)), service: state_advance     
Jun 28 10:50:29.000 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier                                                                                                                             
Jun 28 10:50:29.000 INFO Synced                                  slot: 6760450, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified), peers: 80, service: slot_notifier                                                                                                                                                                       
Jun 28 10:50:29.374 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec                                                                                     
Jun 28 10:50:29.374 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec                                                                      
Jun 28 10:50:41.000 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier                                                                                                                             
Jun 28 10:50:41.000 INFO Synced                                  slot: 6760451, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified), peers: 82, service: slot_notifier                                                                                                                                                                       
Jun 28 10:50:42.299 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec                                                                                     
Jun 28 10:50:42.299 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec                                                                      
Jun 28 10:50:53.000 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier                                                                                                                             
Jun 28 10:50:53.000 INFO Synced                                  slot: 6760452, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified), peers: 81, service: slot_notifier                                                                                                                                                                       
Jun 28 10:50:54.446 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec                                                                                     
Jun 28 10:50:54.446 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec                                                                      
Jun 28 10:51:05.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier                                     
Jun 28 10:51:05.835 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:05.835 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:17.001 INFO Downloading historical blocks           est_time: 18 hrs 8 mins, speed: 16.00 slots/sec, distance: 1044641 slots (20 weeks 5 days), service: slot_notifier
Jun 28 10:51:17.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier
Jun 28 10:51:17.001 INFO Synced                                  slot: 6760454, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 80, service: slot_notifier
Jun 28 10:51:18.051 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:18.051 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:29.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier
Jun 28 10:51:29.001 INFO Synced                                  slot: 6760455, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 79, service: slot_notifier
Jun 28 10:51:31.135 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:31.135 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:40.802 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:40.802 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:41.001 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier
Jun 28 10:51:41.001 INFO Synced                                  slot: 6760456, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 78, service: slot_notifier
Jun 28 10:51:50.479 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Jun 28 10:51:50.479 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:53.000 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier
Jun 28 10:51:53.000 INFO Synced                                  slot: 6760457, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 78, service: slot_notifier
Jun 28 10:51:55.237 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }), service: exec
Jun 28 10:51:55.238 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }) }, service: exec
Jun 28 10:51:55.238 ERRO Error during execution engine upcheck   error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Conn
ection refused" })) }), service: exec
ror whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Jun 28 10:51:53.000 WARN Head is optimistic                      execution_block_hash: 0xa9a4d34b264a774bc1e2b3cfaf78ce7044550fe7146b8f24ae74703cadb58488, info: chain not fully verified, block and attest
ation production disabled until execution engine syncs, service: slot_notifier
Jun 28 10:51:53.000 INFO Synced                                  slot: 6760457, block:    …  empty, epoch: 211264, finalized_epoch: 211262, finalized_root: 0xd6b0…6396, exec_hash: 0xa9a4…8488 (unverified
), peers: 78, service: slot_notifier
Jun 28 10:51:55.237 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }), service: exec
Jun 28 10:51:55.238 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: No
ne, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }) }, service: exec
Jun 28 10:51:55.238 ERRO Error during execution engine upcheck   error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Som
e(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Conn
ection refused" })) }), service: exec

@mattsse
Copy link
Collaborator

mattsse commented Jun 28, 2023

there's currently an issue where reth is temporarily unresponsive while it inserts downloaded blocks, eventually, it should catch up and function normally.

working on improving this here #3416

@onbjerg onbjerg added A-engine Related to the engine implementation and removed S-needs-triage This issue needs to be labelled labels Jun 29, 2023
@rkrasiuk rkrasiuk mentioned this issue Jul 5, 2023
Merged
@rkrasiuk rkrasiuk linked a pull request Jul 5, 2023 that will close this issue
perf(trie): post state cursors #3588
Merged
@github-project-automation github-project-automation bot moved this from Todo to Done in Reth Tracker Jul 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-engine Related to the engine implementation C-bug An unexpected or incorrect behavior
Projects
Reth Tracker
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

perf(trie): post state cursors paradigmxyz/reth
4 participants
@onbjerg @rakita @mattsse @Gaon3