Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Holocene hardfork test (kurtosis) #13192

Closed
vdamle opened this issue Dec 3, 2024 · 14 comments
Closed

Holocene hardfork test (kurtosis) #13192

vdamle opened this issue Dec 3, 2024 · 14 comments
Assignees
Labels
A-op-batcher Area: op-batcher

Comments

@vdamle
Copy link
Contributor

vdamle commented Dec 3, 2024

When running a multi-client chain, there was a reported chain-spit that occurred when Holocene was activated. For more info, see PR. This issue is written to outline a plan for testing Holocene hard fork in Kurtosis using both op-geth and op-reth that includes the linked fix and ensure there is no chain-split.

Prerequisites

Since we are testing a fix that was implemented in reth, we will use the latest images for all components except op-reth, for which we will use a locally built image for testing.

optimism_package:
  chains:
    - participants:
      - el_type: op-geth
        cl_type: op-node
      - el_type: op-reth
        el_image: "ghcr.io/paradigmxyz/reth:holocene-fix"
        cl_type: op-node
      network_params:
        fjord_time_offset: 0
        granite_time_offset: 0
        holocene_time_offset: 4

Tests

We will execute the first test below with reth:v1.1.2 and check that we are able to reproduce the issue. Subsequently, we will a reth image which has the fix and make sure there is no chain split.

  • Without Transactions
    • Start the chain in kurtosis with 2 EL/CL nodes using kurtosis run . --args-file <file path>, as outlined in the config yaml above -- without sending any transactions.
    • Test for consistent block hashes on both EL nodes, at the holocene activation block.
    • Test for consistent block hashes on a few subsequent block on both op-geth and op-reth endpoints.
    • Invoke SystemConfig.setEIP1559Params using w3
  • With Transactions
    • Wipe existing enclaves/data using kurtosis clean -a
    • Start the chain in kurtosis with 2 EL/CL nodes using kurtosis run . --args-file <file path>, as outlined in the config yaml above -- without sending any transactions.
    • Start the transaction fuzzer as an additional service in Kurtosis to send transactions.
    • Test for consistent block hashes on both EL nodes, at the holocene activation block.
    • Test for consistent block hashes on a few subsequent block on both op-geth and op-reth endpoints.
    • Invoke SystemConfig.setEIP1559Params using w3

In both cases, we need to ensure that there is no divergence between the hashes reported by op-geth and op-reth.

Script to check and dump block hashes

#!/bin/bash

# Check if the correct number of arguments is provided
if [ "$#" -ne 2 ]; then
  echo "Usage: $0 <port1-geth> <port2-reth>"
  exit 1
fi

# Assign arguments to variables
PORT1=$1
PORT2=$2

# Hard-coded block numbers - a few before holocene activation, holocene activation block (4) and a couple blocks after
BLOCK_NUMBERS=(0 1 3 4 5 100)

# Iterate through the hard-coded block numbers
for BLOCK in "${BLOCK_NUMBERS[@]}"; do
  echo "Checking block hash for $BLOCK..."

  # Run the cast command for both ports
  HASH1=$(cast block $BLOCK --rpc-url 127.0.0.1:$PORT1 --json | jq -r .hash)
  HASH2=$(cast block $BLOCK --rpc-url 127.0.0.1:$PORT2 --json | jq -r .hash)

  # Display results
  echo "Geth RPC $PORT1 - Block $BLOCK Hash: $HASH1"
  echo "Reth RPC $PORT2 - Block $BLOCK Hash: $HASH2"
  echo ""
done
@vdamle vdamle self-assigned this Dec 3, 2024
@geoknee
Copy link
Contributor

geoknee commented Dec 3, 2024

@vdamle it would also be great to run it with an unpatched op-reth and confirm that we see the chain split.

@vdamle
Copy link
Contributor Author

vdamle commented Dec 3, 2024

@vdamle it would also be great to run it with an unpatched op-reth and confirm that we see the chain split.

My bad, I missed noting that. Yes, I've built an image based off of `v1.1.2 and will check that I can rep the original issue.

@tynes
Copy link
Contributor

tynes commented Dec 3, 2024

Once we have something working e2e, it would be good to:

  • include nethermind in the tests
  • consider how to modularize the setup so that we can add more tests for future hardforks and run in CI

@geoknee geoknee changed the title Holocene hardfork test Holocene hardfork test (kurtosis) Dec 4, 2024
@geoknee geoknee self-assigned this Dec 4, 2024
@vdamle
Copy link
Contributor Author

vdamle commented Dec 4, 2024

Without transactions, with fix 6cea9955

  • Reth logs show correct version and fork timestamp for holocene:
2024-12-04T13:20:24.678853Z  INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/2151908
2024-12-04T13:20:24.681567Z  INFO Starting reth version="1.1.2-dev (6cea9955)"
2024-12-04T13:20:24.681716Z  INFO Opening database path="/data/op-reth/execution-data/db"
2024-12-04T13:20:24.699872Z  INFO Saving prune config to toml file
2024-12-04T13:20:24.700200Z  INFO Configuration loaded path="/data/op-reth/execution-data/reth.toml"
2024-12-04T13:20:24.700892Z  INFO Verifying storage consistency.
2024-12-04T13:20:24.726454Z  INFO Database opened
2024-12-04T13:20:24.726509Z  INFO Starting metrics endpoint at 0.0.0.0:9001
Post-merge hard forks (timestamp based):
- Regolith                         @0
- Shanghai                         @0
- Canyon                           @0
- Cancun                           @0
- Ecotone                          @0
- Fjord                            @0
- Granite                          @0
- Holocene                         @1733318418
2024-12-04T13:31:42.810281Z  INFO Status connected_peers=1 latest_block=0
2024-12-04T13:31:43.542625Z  INFO Block added to canonical chain number=1 hash=0xbf5f9fed60870ebb22513168c357b706832f34e8bf753e0a3e16b351c75f4e60 peers=1 txs=1 gas=160.44 Kgas gas_throughput=3.57 Mgas/second full=0.3% base_fee=1.00gwei blobs=0 excess_blobs=0 elapsed=44.894175ms
2024-12-04T13:31:43.556073Z  INFO Canonical chain committed number=1 hash=0xbf5f9fed60870ebb22513168c357b706832f34e8bf753e0a3e16b351c75f4e60 elapsed=105.664µs
2024-12-04T13:31:43.559202Z  INFO Block added to canonical chain number=2 hash=0xc2fa2c807b02aff42303e31f4be20152a039afc92ac76a9dabcea482074434a0 peers=1 txs=1 gas=43.84 Kgas gas_throughput=21.61 Mgas/second full=0.1% base_fee=0.99gwei blobs=0 excess_blobs=0 elapsed=2.029127ms
2024-12-04T13:31:43.559668Z  INFO Canonical chain committed number=2 hash=0xc2fa2c807b02aff42303e31f4be20152a039afc92ac76a9dabcea482074434a0 elapsed=75.04µs
2024-12-04T13:31:43.561686Z  INFO Block added to canonical chain number=3 hash=0x9bde1550c17a776c20a824375a9fe0160a24ebcf9c33444d6a3d47dcb055ce11 peers=1 txs=1 gas=52.24 Kgas gas_throughput=37.00 Mgas/second full=0.1% base_fee=0.99gwei blobs=0 excess_blobs=0 elapsed=1.412056ms
2024-12-04T13:31:43.562063Z  INFO Canonical chain committed number=3 hash=0x9bde1550c17a776c20a824375a9fe0160a24ebcf9c33444d6a3d47dcb055ce11 elapsed=13.333µs
2024-12-04T13:31:43.563530Z  INFO Block added to canonical chain number=4 hash=0x18072d9f821355da8f12c64c6bd1aa35bce6f8c9c81c3de965fa92523321efd5 peers=1 txs=1 gas=43.85 Kgas gas_throughput=40.84 Mgas/second full=0.1% base_fee=0.98gwei blobs=0 excess_blobs=0 elapsed=1.073729ms
2024-12-04T13:31:43.564060Z  INFO Canonical chain committed number=4 hash=0x18072d9f821355da8f12c64c6bd1aa35bce6f8c9c81c3de965fa92523321efd5 elapsed=16.792µs
2024-12-04T13:31:43.565695Z  INFO Block added to canonical chain number=5 hash=0xff79963042d75f0a1c477af719a4ae1a14fa8a3f01c3f25a2e4b70c6ae332e8d peers=1 txs=1 gas=43.85 Kgas gas_throughput=41.70 Mgas/second full=0.1% base_fee=0.98gwei blobs=0 excess_blobs=0 elapsed=1.051772ms
2024-12-04T13:31:43.566169Z  INFO Canonical chain committed number=5 hash=0xff79963042d75f0a1c477af719a4ae1a14fa8a3f01c3f25a2e4b70c6ae332e8d elapsed=14.291µs
2024-12-04T13:31:43.567703Z  INFO Block added to canonical chain number=6 hash=0xa8b33501380d531e8b8b1cca0b082a9110a845258323f0d2aeca8441185ffdd0 peers=1 txs=1 gas=43.85 Kgas gas_throughput=44.54 Mgas/second full=0.1% base_fee=0.98gwei blobs=0 excess_blobs=0 elapsed=984.648µs
  • block hashes are consistent pre-holocene, at holocene (block 4) and post holocene
➜  ~ ~/scripts/check-block-cast.sh 33023 33030
Checking block hash for 0...
Geth RPC 33023 - Block 0 Hash: 0x0f88d8da443ce62946b773d7ea66f8facca308420ccd8d3bfeca0bd040ea384a
Reth RPC 33030 - Block 0 Hash: 0x0f88d8da443ce62946b773d7ea66f8facca308420ccd8d3bfeca0bd040ea384a

Checking block hash for 1...
Geth RPC 33023 - Block 1 Hash: 0xbf5f9fed60870ebb22513168c357b706832f34e8bf753e0a3e16b351c75f4e60
Reth RPC 33030 - Block 1 Hash: 0xbf5f9fed60870ebb22513168c357b706832f34e8bf753e0a3e16b351c75f4e60

Checking block hash for 3...
Geth RPC 33023 - Block 3 Hash: 0x9bde1550c17a776c20a824375a9fe0160a24ebcf9c33444d6a3d47dcb055ce11
Reth RPC 33030 - Block 3 Hash: 0x9bde1550c17a776c20a824375a9fe0160a24ebcf9c33444d6a3d47dcb055ce11

Checking block hash for 4...
Geth RPC 33023 - Block 4 Hash: 0x18072d9f821355da8f12c64c6bd1aa35bce6f8c9c81c3de965fa92523321efd5
Reth RPC 33030 - Block 4 Hash: 0x18072d9f821355da8f12c64c6bd1aa35bce6f8c9c81c3de965fa92523321efd5

Checking block hash for 5...
Geth RPC 33023 - Block 5 Hash: 0xff79963042d75f0a1c477af719a4ae1a14fa8a3f01c3f25a2e4b70c6ae332e8d
Reth RPC 33030 - Block 5 Hash: 0xff79963042d75f0a1c477af719a4ae1a14fa8a3f01c3f25a2e4b70c6ae332e8d

Checking block hash for 100...
Geth RPC 33023 - Block 100 Hash: 0x04a0bd9df4d015c18e9b8d6be7582910846540bd430643b4432077b2e922b8e9
Reth RPC 33030 - Block 100 Hash: 0x04a0bd9df4d015c18e9b8d6be7582910846540bd430643b4432077b2e922b8e9

@vdamle
Copy link
Contributor Author

vdamle commented Dec 4, 2024

Without transactions, on release v1.1.2

I did not observe a chain split with this test

  • Reth logs show correct version and fork timestamp for holocene:
2024-12-04T16:02:00.600014Z  INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/2151908
2024-12-04T16:02:00.602263Z  INFO Starting reth version="1.1.2-dev (496bf0bf)"
2024-12-04T16:02:00.602371Z  INFO Opening database path="/data/op-reth/execution-data/db"
2024-12-04T16:02:00.629342Z  INFO Saving prune config to toml file
2024-12-04T16:02:00.629536Z  INFO Configuration loaded path="/data/op-reth/execution-data/reth.toml"
2024-12-04T16:02:00.630123Z  INFO Verifying storage consistency.
2024-12-04T16:02:00.655553Z  INFO Database opened
....
Post-merge hard forks (timestamp based):
- Regolith                         @0
- Shanghai                         @0
- Canyon                           @0
- Cancun                           @0
- Ecotone                          @0
- Fjord                            @0
- Granite                          @0
- Holocene                         @1733328114
2024-12-04T16:02:00.725364Z  INFO Transaction pool initialized
2024-12-04T16:02:00.739686Z  INFO P2P networking initialized enode=enode://addf62c286f8f3ed7a47cdaab6b2d56f4642ebd132e4faaf89f81906fb9de0c98f564abdd5bba4532ad1a734ebbbdda376398c0804b5cea8f4ae8d141940b98a@172.16.0.21:30303
2024-12-04T16:02:00.740225Z  INFO StaticFileProducer initialized
  • Block hashes are again consistent:
➜  ~ ~/scripts/check-block-cast.sh 33075 33082
Checking block hash for 0...
Geth RPC 33075 - Block 0 Hash: 0x910ad0af7965cd61b9cdaafab6014add621ee0961bb39c04cc818c7522b34975
Reth RPC 33082 - Block 0 Hash: 0x910ad0af7965cd61b9cdaafab6014add621ee0961bb39c04cc818c7522b34975

Checking block hash for 1...
Geth RPC 33075 - Block 1 Hash: 0x142956217ed6014cd45b828ffc4b0d541eb69ab525f79107a8b6c01856f7ce12
Reth RPC 33082 - Block 1 Hash: 0x142956217ed6014cd45b828ffc4b0d541eb69ab525f79107a8b6c01856f7ce12

Checking block hash for 3...
Geth RPC 33075 - Block 3 Hash: 0x8db0085449343985adad7b65f01ffb3c537106ea67210de5b5a2d84651377227
Reth RPC 33082 - Block 3 Hash: 0x8db0085449343985adad7b65f01ffb3c537106ea67210de5b5a2d84651377227

Checking block hash for 4...
Geth RPC 33075 - Block 4 Hash: 0x67c4118acedf7bb5178fa44f032baa060285c4c3175221dd500f152cd031482d
Reth RPC 33082 - Block 4 Hash: 0x67c4118acedf7bb5178fa44f032baa060285c4c3175221dd500f152cd031482d

Checking block hash for 5...
Geth RPC 33075 - Block 5 Hash: 0x04a6f3652be38d6d578b78f9c40dbee637af48225c38f403f1146950500e01a9
Reth RPC 33082 - Block 5 Hash: 0x04a6f3652be38d6d578b78f9c40dbee637af48225c38f403f1146950500e01a9

Checking block hash for 100...
Geth RPC 33075 - Block 100 Hash: 0x755ac773c06d2c5513e0010b5591871a6560cb911776239339ccb69b2adf36e0
Reth RPC 33082 - Block 100 Hash: 0x755ac773c06d2c5513e0010b5591871a6560cb911776239339ccb69b2adf36e0

@vdamle
Copy link
Contributor Author

vdamle commented Dec 4, 2024

@geoknee and I looked into this. As noted above, I attempted to trigger a chain split using an unpatched version of op-reth. However, it appears that the initial values of the EIP1559 params parsed from the extradata (i.e. those coming from the holocene route) are the same as the pre holocene chain spec values. I don't know if it is possible to change the values before the holocene activation in a way which won't break the chain (cause it to halt). Ordinarily we wouldn't set dynamically eip1559 params before holocene activates.

https://github.com/ethereum-optimism/op-geth/blob/optimism/consensus/misc/eip1559/eip1559.go#L144
https://github.com/ethereum-optimism/op-geth/blob/optimism/params/config.go#L897-L899

I believe in the case of the Kurtosis setup, both c.Optimism.EIP1559DenominatorCanyon and c.Optimism.EIP1559Denominator are the same and we're not seeing any difference between the clients due to that?

Based on the above determination, we believe the right next step is to concentrate on another bug paradigmxyz/reth#13121 that Minyhuk found, which should allow us to trigger a chain split at any point in time. To do this, we just need to call setEIP1559Params() on the SystemConfigProxy L1 contract. This is permissioned and needs to be invoked by the SystemConfigOwner . We should use the values quoted in the above issue.

@vdamle
Copy link
Contributor Author

vdamle commented Dec 5, 2024

Reproduced the bug reported in paradigmxyz/reth#13121 using the image with the commit 6cea9955 with following procedure in Kurtosis:

  • In order to invoke setEIP1559Params() on the SystemConfigProxy contract, we need the address and private key of the SystemConfigOwner role, to send the transaction.
  • The address can be found by inspecting the state.json file generated by the kurtosis enclave. The command to dump all the files generated by kurtosis and the path to the state.json file are as follows:
kurtosis enclave ls
kurtosis enclave inspect <name>  -- this lists all running containers and the URLs/ports for RPC/P2P etc.
kurtosis enclave dump <name>
cd <dump_dir>

➜  lively-rift--bf8dbc805ba84277b691c316fc8f5e55 git:(main) ✗ cat files/op-deployer-configs/state.json| grep systemConfigProxyAddress
      "systemConfigProxyAddress": "0x42de298b17bbaa3b0f55abc1d3db7b0d0b0fd656",
➜  lively-rift--bf8dbc805ba84277b691c316fc8f5e55 git:(main) ✗ cat files/op-deployer-configs/state.json| grep systemConfigOwner
          "systemConfigOwner": "0x8e44febeae9c2f2bc89480ae8084328badc496c2",

As seen above, systemConfigOwner is "0x8e44febeae9c2f2bc89480ae8084328badc496c2" and the systemConfigProxy is deployed at address "0x42de298b17bbaa3b0f55abc1d3db7b0d0b0fd656".

  • The addresses for all roles are based off a a HD Wallet - the seed and the path can be found in ./files/op-deployer-fund-script/fund.sh. The key for each role is at the following path "m/44'/60'/2'/$chain_id/$role_index. We can generate the private key for the systemConfigOwner and verify that the address is the same as the one specified in state.json
➜  ~ cast wallet private-key 'test test test test test test test test test test test junk' 'm/44'\''/60'\''/2'\''/2151908/10'
0x28978385446dde6ce493da611d6c9ae6193d960c1cb73ecfb25800e9c9017f4c
➜  ~ cast wallet address 0x28978385446dde6ce493da611d6c9ae6193d960c1cb73ecfb25800e9c9017f4c
0x8e44fEbeaE9C2F2Bc89480aE8084328Badc496c2
  • To be able to invoke the contract function from the systemConfigOwner, we need to fund it using the pre-funded wallet (grab the L1 RPC port from the L1 EL container listed in the output of kurtosis enclave inspect)
➜  ~ cast nonce 0x589A698b7b7dA0Bec545177D3963A2741105C7C9  --rpc-url 127.0.0.1:33197
31
➜  ~ cast send 0x8e44fEbeaE9C2F2Bc89480aE8084328Badc496c2 --value 10ether  --private-key eaba42282ad33c8ef2524f07277c03a776d98ae19f581990ce75becb7cfa1c23  --nonce 31 --rpc-url 127.0.0.1:33197 --async
0x50f698e0274c2b6232260cae4f81e126548e989f95595bd4a36572aa4ad45774
  • Finally, invoke the contract function setEIP1559Params() on the SystemConfigProxy contract, using the private key we derived above and send it to the systemConfigProxyAddress
➜  ~ cast send --private-key 0x28978385446dde6ce493da611d6c9ae6193d960c1cb73ecfb25800e9c9017f4c  0x42de298b17bbaa3b0f55abc1d3db7b0d0b0fd656  "setEIP1559Params(uint32,uint32)" "0xfa" "0x3c" --rpc-url 127.0.0.1:33197

blockHash               0xba58836f1540521f0b403f4af6b764959a71417f5176d03991ddd391f1423234
blockNumber             118
contractAddress
cumulativeGasUsed       54016
effectiveGasPrice       5000000241
from                    0x8e44fEbeaE9C2F2Bc89480aE8084328Badc496c2
gasUsed                 54016
logs                    [{"address":"0x42de298b17bbaa3b0f55abc1d3db7b0d0b0fd656","topics":["0x1d2b0bda21d56b8bd12d4f94ebacffdfb35f5e226f84b461103bb8beab6353be","0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000000000000000000000000000000000000000000000000004"],"data":"0x00000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000fa0000003c","blockHash":"0xba58836f1540521f0b403f4af6b764959a71417f5176d03991ddd391f1423234","blockNumber":"0x76","transactionHash":"0x213035f8e5fc3a440543ada39c7447a98981d62fd544fca1c1dbec5298062819","transactionIndex":"0x0","logIndex":"0x0","removed":false}]
logsBloom               0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001008000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000020000000000000000000800000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000008000000000000000000020000010000000000000000000000020000000000000000000000000008000000000
root
status                  1 (success)
transactionHash         0x213035f8e5fc3a440543ada39c7447a98981d62fd544fca1c1dbec5298062819
transactionIndex        0
type                    2
blobGasPrice
blobGasUsed
authorizationList
to                      0x42dE298b17bbAA3B0F55ABC1d3Db7B0d0B0FD656
  • We can check that the values were set correctly
➜  ~ cast call 0x42de298b17bbaa3b0f55abc1d3db7b0d0b0fd656 "eip1559Denominator()" --rpc-url 127.0.0.1:33197
0x00000000000000000000000000000000000000000000000000000000000000fa
➜  ~ cast call 0x42de298b17bbaa3b0f55abc1d3db7b0d0b0fd656 "eip1559Elasticity()" --rpc-url 127.0.0.1:33197
0x000000000000000000000000000000000000000000000000000000000000003c

We can now see in the logs of the op-node paired with reth and the block is invalid:

t=2024-12-05T19:38:37+0000 lvl=info msg="Advancing bq origin" origin=0x0a35703f0572c4f0d9b2e7791cb294e2e7fa055fa25c064bc0c1c2cd26f34520:121 originBehind=false
t=2024-12-05T19:38:38+0000 lvl=info msg="Received signed execution payload from p2p" id=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 peer=16Uiu2HAm7mnrzyuGEsHr24w5XtH6L5eDSy3JkNnfMSjeEhym4zyv txs=1
t=2024-12-05T19:38:38+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356
t=2024-12-05T19:38:39+0000 lvl=info msg="failed to insert payload" ref=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 txs=1 err="temp: cannot process unsafe payload: new - 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356; parent: 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f:355; err: execution payload 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 was INVALID! Latest valid hash is 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ignoring bad block: 0x4001ec3680"
t=2024-12-05T19:38:39+0000 lvl=warn msg="Dropping invalid unsafe payload" hash=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83 number=356 timestamp=1733427519
t=2024-12-05T19:38:39+0000 lvl=warn msg="Payload was invalid" block=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 err="execution payload 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 was INVALID! Latest valid hash is 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ignoring bad block: 0x4001ec3680" timestamp=1733427519
t=2024-12-05T19:38:39+0000 lvl=warn msg="Engine temporary error" err="temp: cannot process unsafe payload: new - 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356; parent: 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f:355; err: execution payload 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 was INVALID! Latest valid hash is 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ignoring bad block: 0x4001ec3680"
t=2024-12-05T19:38:40+0000 lvl=info msg="Received signed execution payload from p2p" id=0xa9672293397c93d9fd3f05c108b6392e6beba36e11ae42e6fa73c91fdf3c6ca1:357 peer=16Uiu2HAm7mnrzyuGEsHr24w5XtH6L5eDSy3JkNnfMSjeEhym4zyv txs=1
t=2024-12-05T19:38:40+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xa9672293397c93d9fd3f05c108b6392e6beba36e11ae42e6fa73c91fdf3c6ca1:357
t=2024-12-05T19:38:41+0000 lvl=info msg="processing L2 range request" target=355 end=0xa9672293397c93d9fd3f05c108b6392e6beba36e11ae42e6fa73c91fdf3c6ca1:357 rangeReqId=2
t=2024-12-05T19:38:41+0000 lvl=info msg="Received signed execution payload from p2p" id=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 peer=16Uiu2HAm7mnrzyuGEsHr24w5XtH6L5eDSy3JkNnfMSjeEhym4zyv txs=1
t=2024-12-05T19:38:41+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356
t=2024-12-05T19:38:41+0000 lvl=info msg="failed to insert payload" ref=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 txs=1 err="temp: cannot process unsafe payload: new - 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356; parent: 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f:355; err: execution payload 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 was INVALID! Latest valid hash is 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ignoring bad block: 0x40007bfc80"
t=2024-12-05T19:38:41+0000 lvl=warn msg="Dropping invalid unsafe payload" hash=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83 number=356 timestamp=1733427519
t=2024-12-05T19:38:41+0000 lvl=warn msg="Payload was invalid" block=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 err="execution payload 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 was INVALID! Latest valid hash is 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ignoring bad block: 0x40007bfc80" timestamp=1733427519
t=2024-12-05T19:38:41+0000 lvl=warn msg="Engine temporary error" err="temp: cannot process unsafe payload: new - 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356; parent: 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f:355; err: execution payload 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 was INVALID! Latest valid hash is 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ignoring bad block: 0x40007bfc80"
t=2024-12-05T19:38:42+0000 lvl=info msg="Received signed execution payload from p2p" id=0x37605cd343efeef5c629d30f1feeeee6df92aebcdd91b55c09d0e721eb829805:358 peer=16Uiu2HAm7mnrzyuGEsHr24w5XtH6L5eDSy3JkNnfMSjeEhym4zyv txs=1
t=2024-12-05T19:38:42+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x37605cd343efeef5c629d30f1feeeee6df92aebcdd91b55c09d0e721eb829805:358
t=2024-12-05T19:38:44+0000 lvl=info msg="Received signed execution payload from p2p" id=0x9e11fec3aac55a8c4134681d3f9e5176ce137b09615e25612fcf7e9a8694eca4:359 peer=16Uiu2HAm7mnrzyuGEsHr24w5XtH6L5eDSy3JkNnfMSjeEhym4zyv txs=1
t=2024-12-05T19:38:44+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x9e11fec3aac55a8c4134681d3f9e5176ce137b09615e25612fcf7e9a8694eca4:359
t=2024-12-05T19:38:45+0000 lvl=info msg="processing L2 range request" target=355 end=0xa9672293397c93d9fd3f05c108b6392e6beba36e11ae42e6fa73c91fdf3c6ca1:357 rangeReqId=3
t=2024-12-05T19:38:45+0000 lvl=info msg="Received signed execution payload from p2p" id=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 peer=16Uiu2HAm7mnrzyuGEsHr24w5XtH6L5eDSy3JkNnfMSjeEhym4zyv txs=1
t=2024-12-05T19:38:45+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356
t=2024-12-05T19:38:45+0000 lvl=info msg="failed to insert payload" ref=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 txs=1 err="temp: cannot process unsafe payload: new - 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356; parent: 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f:355; err: execution payload 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 was INVALID! Latest valid hash is 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ignoring bad block: 0x4001b94530"
t=2024-12-05T19:38:45+0000 lvl=warn msg="Dropping invalid unsafe payload" hash=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83 number=356 timestamp=1733427519
t=2024-12-05T19:38:45+0000 lvl=warn msg="Payload was invalid" block=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 err="execution payload 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83:356 was INVALID! Latest valid hash is 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ignoring bad block: 0x4001b94530" timestamp=1733427519

and in reth as well:

2024-12-05T19:38:37.002065Z  INFO Canonical chain committed number=355 hash=0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f elapsed=59.126µs
2024-12-05T19:38:38.995501Z  WARN Failed to validate header 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83 against parent: block base fee mismatch: got 241700508, expected 241654910 block=SealedBlockWithSenders { block: SealedBlock { header: SealedHeader { hash: 0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83, header: Header { parent_hash: 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xb48b808b33e926ee3b17e8b4027fffcdcc2e7058b3b33164d406fa13e905859f, transactions_root: 0x37bb7714ba13b6d498d90f17b70d2c1c8053419d842d3b56bab36850c9b1a887, receipts_root: 0x9441f3e15878077b6969efc4830bbe393e600bba5cc8ddd9a6a006124f365986, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0, number: 356, gas_limit: 60000000, gas_used: 49394, timestamp: 1733427519, extra_data: 0x00000000fa0000003c, mix_hash: 0x6cde3099341ba98e0680256792b4534af72bf14e075bb1597b8b89fddd2edc98, nonce: 0x0000000000000000, base_fee_per_gas: Some(241700508), withdrawals_root: Some(0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x6f46d7cec19c69325a4185396847de932d3cd1e97443994f736f1bd8ea69e4f4), requests_hash: None, target_blobs_per_block: None } }, body: BlockBody { transactions: [TransactionSigned { hash: OnceLock(<uninit>), signature: PrimitiveSignature { y_parity: false, r: 0, s: 0 }, transaction: Deposit(TxDeposit { source_hash: 0xc63ba642e3f2a474aac34d81e6a7e2d9398caa53a8094eac7f56fa9a5b77420d, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0, gas_limit: 1000000, is_system_transaction: false, input: 0x440a5e2000000558000c5fc500000000000000000000000067520117000000000000007700000000000000000000000000000000000000000000000000000000000000d40000000000000000000000000000000000000000000000000000000000000001344af750b8527102592d76a6b6215e1cb6ef8e738f635a9f009a30bc224e713b000000000000000000000000d3f2c5afb2d76f5579f326b0cd7da5f5a4126c35 }) }], ommers: [], withdrawals: Some(Withdrawals([])) } }, senders: [0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001] }
2024-12-05T19:38:39.013302Z  WARN Invalid block error on new payload invalid_hash=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83 invalid_number=356 validation_err=block base fee mismatch: got 241700508, expected 241654910
2024-12-05T19:38:39.013352Z  WARN Bad block with hash hash=0x6c4a3fe708668af66b29a361ef3773d8f61a9d13a5044e37e474e31e69d0ce83 header=Header { parent_hash: 0x70b361d7f5c1adf2181bbac6aab041fdd4d9b93442c8f2eba99060451c183a9f, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xb48b808b33e926ee3b17e8b4027fffcdcc2e7058b3b33164d406fa13e905859f, transactions_root: 0x37bb7714ba13b6d498d90f17b70d2c1c8053419d842d3b56bab36850c9b1a887, receipts_root: 0x9441f3e15878077b6969efc4830bbe393e600bba5cc8ddd9a6a006124f365986, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0, number: 356, gas_limit: 60000000, gas_used: 49394, timestamp: 1733427519, extra_data: 0x00000000fa0000003c, mix_hash: 0x6cde3099341ba98e0680256792b4534af72bf14e075bb1597b8b89fddd2edc98, nonce: 0x0000000000000000, base_fee_per_gas: Some(241700508), withdrawals_root: Some(0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x6f46d7cec19c69325a4185396847de932d3cd1e97443994f736f1bd8ea69e4f4), requests_hash: None, target_blobs_per_block: None }

@vdamle
Copy link
Contributor Author

vdamle commented Dec 5, 2024

Verified that the bug cannot be reproduced in reth with the latest tip of main that includes the fix:

2024-12-05T21:24:37.420462Z  INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/2151908
2024-12-05T21:24:37.422898Z  INFO Starting reth version="1.1.2-dev (8226fa0c)"
2024-12-05T21:24:37.423022Z  INFO Opening database path="/data/op-reth/execution-data/db"
2024-12-05T21:24:37.446958Z  INFO Saving prune config to toml file
2024-12-05T21:24:37.447269Z  INFO Configuration loaded path="/data/op-reth/execution-data/reth.toml"
2024-12-05T21:24:37.468301Z  INFO Verifying storage consistency.
2024-12-05T21:24:37.503755Z  INFO Database opened

We can see that updated extraData is present in all blocks: 0x00000000fa0000003c after invoking setEIP1559Params()

Checking block hash for latest...
Geth RPC 33232 - Block latest Hash: "0x784315014d85fe2f2b79ecbaa5b9b17c1fbb0886374e1c4ea70b181655c3073f 0x1f8 0x00000000fa0000003c"
Reth RPC 33239 - Block latest Hash: "0x784315014d85fe2f2b79ecbaa5b9b17c1fbb0886374e1c4ea70b181655c3073f 0x1f8 0x00000000fa0000003c"
➜  ~ ~/scripts/check-block-cast.sh 33232 33239
Checking block hash for latest...
Geth RPC 33232 - Block latest Hash: "0x007996546fb6336f8aaa053ae7d0d0c350cd3169f2985ca7d850de7d262e88ee 0x253 0x00000000fa0000003c"
Reth RPC 33239 - Block latest Hash: "0x007996546fb6336f8aaa053ae7d0d0c350cd3169f2985ca7d850de7d262e88ee 0x253 0x00000000fa0000003c"

@swapnilraj
Copy link

@vdamle what's the best way to include Nethermind in your testing?
could you point to us to places where we should make PRs and follow the discussion on e2e testing?

@vdamle
Copy link
Contributor Author

vdamle commented Dec 9, 2024

@vdamle what's the best way to include Nethermind in your testing? could you point to us to places where we should make PRs and follow the discussion on e2e testing?

@swapnilraj If you build a Nethermind docker image, you could spin up a 2 node chain running op-geth and nethermind using kurtosis. The instructions in that repo are fairly easy to follow along, but if you need any help/have questions with it, please do tag me on any of the protocol #tests discord channel and I'd be happy to take a look! Let me know how it goes for you.

@vdamle
Copy link
Contributor Author

vdamle commented Dec 10, 2024

Earlier today, @geoknee outlined a couple more scenarios for testing with Kurtosis using op-batcher/v1.10.0-rc.3 image for op-batcher. See discord thread

Environment setup:

  • Built an image of the batcher in the monorepo at tag op-batcher/v1.10.0-rc.3 using make golang-docker in the monorepo.
  • Tweaked the kurtosis package yaml to use the custom batcher image:
optimism_package:
  chains:
    - participants:
      - el_type: op-geth
        cl_type: op-node
      - el_type: op-reth
        el_image: "op-reth:holocene-fix-2"
        cl_type: op-node
      network_params:
        fjord_time_offset: 0
        granite_time_offset: 0
        holocene_time_offset: 10
      batcher_params:
        image: us-docker.pkg.dev/oplabs-tools-artifacts/images/op-batcher:1.10.0
./livefuzzer spam  --sk "0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80" --rpc http://127.0.0.1:32987 --slot-time 2

Batcher Restart Test after a Pause

  • Initially, verified that the batcher was operating normally using the command kurtosis service logs <enclave-name> op-batcher-op-kurtosis -f
  • Stopped the op-batcher service: kurtosis service stop <enclave-name> op-batcher-op-kurtosis
  • Started the op-batcher service after a few minutes using: kurtosis service start <enclave-name> op-batcher-op-kurtosis

op-batcher appeared to recover correctly after a restart, the logs show that it detected a backlog of blocks and proceeded to work on the backlog successfully and moved forward.

  • Logs before restart:
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Channel is fully submitted" id=d3e2ec1ba14446bd18c81382f3c78916 min_inclusion_block=46 max_inclusion_block=46
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=warn msg="Aborting state publishing, max duration exceeded"
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x3fcc0474677c089832ae6e807ce7f8c4d0f40f37950db645b69795b37e1b4419 nonce=19 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Loading range of multiple blocks into state" start=122 end=127
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Added L2 block to local state" block=0x053caece73683cf83fbf7ade0b5354d3f24be6b85d5d0cf56e84b5a14ded3ab2:122 tx_count=1 time=1733803985
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0x3fcc0474677c089832ae6e807ce7f8c4d0f40f37950db645b69795b37e1b4419 nonce=19 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000 tx=0x3fcc0474677c089832ae6e807ce7f8c4d0f40f37950db645b69795b37e1b4419
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Added L2 block to local state" block=0xa960e600ac05b668f2a60c9fc4d5ac7e101065ebc9a46f047835cd8f9791f1cc:123 tx_count=120 time=1733803987
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Added L2 block to local state" block=0x08d7c297d0d2e80d5e62ba9d8f695b9f2abef2a3563191b7b30733fa4876bf8f:124 tx_count=1 time=1733803989
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Added L2 block to local state" block=0x9a368b011c93ee6e0c1504ab0b0999fe7070dd3ee11fcb771bedd5e353b28231:125 tx_count=1 time=1733803991
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Added L2 block to local state" block=0x0f9a289577c8db151987f580ebcdae07fcd945735cc3ee499b05a11fa9b0edcd:126 tx_count=101 time=1733803993
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Added L2 block to local state" block=0xc89d851fc05639443b1176e6a284cbfe96c3994c4e9d929570a42a624aed3bbf:127 tx_count=54 time=1733803995
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Created channel" id=dc7ccdf36a86596d43c3ab2d081ff286 l1Head=0x6abc3bbe2f5e470a6b757e411a5719f430750d02062deff2f68db46eed01aae9:47 blocks_pending=6 l1OriginLastSubmittedChannel=0x01eb53061c10beba342d10fbbe54be59dc622df3f1064bf1e46298fe3a3b9e81:38 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=130043 use_blobs=true
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Channel closed" id=dc7ccdf36a86596d43c3ab2d081ff286 blocks_pending=0 num_frames=1 input_bytes=51497 output_bytes=44547 oldest_l1_origin=0x633c890882c10459a32422076a9c03de20ccaf8ff50ac845f8b2639a008379e2:39 l1_origin=0x5a6647ea9825686bba4ab8f43ae15d766d0189bfc963ce6a99ebebcb9f175c37:40 oldest_l2=0x053caece73683cf83fbf7ade0b5354d3f24be6b85d5d0cf56e84b5a14ded3ab2:122 latest_l2=0xc89d851fc05639443b1176e6a284cbfe96c3994c4e9d929570a42a624aed3bbf:127 full_reason="channel full: max channel duration reached" compr_ratio=0.8650406819814747
[op-batcher-op-kurtosis] t=2024-12-10T04:13:15+0000 lvl=info msg="Building Blob transaction candidate" size=44547 last_size=44547 num_blobs=1
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Transaction confirmed" service=batcher tx=0x3fcc0474677c089832ae6e807ce7f8c4d0f40f37950db645b69795b37e1b4419 block=0x0a3dd9307c66a90a3f5e3c6f7b4166b9ed4fb78898321b2d4124f01889af947b:48 effectiveGasPrice=5002708861
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Handling receipt" id=0c40acc8f2e7bf74d2945f8d2f0266fe:0
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Transaction confirmed" tx_id=0c40acc8f2e7bf74d2945f8d2f0266fe:0 tx=0x3fcc0474677c089832ae6e807ce7f8c4d0f40f37950db645b69795b37e1b4419 block=0x0a3dd9307c66a90a3f5e3c6f7b4166b9ed4fb78898321b2d4124f01889af947b:48
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Channel is fully submitted" id=0c40acc8f2e7bf74d2945f8d2f0266fe min_inclusion_block=48 max_inclusion_block=48
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=warn msg="Aborting state publishing, max duration exceeded"
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x2b9b6d4b866b34b3ba12df1d5b872ceeb817dc826cfb05c5d106cab770e55d0c nonce=20 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Loading range of multiple blocks into state" start=128 end=133
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Added L2 block to local state" block=0x81974e2fc7f9f5c7e5b130ba301e2179561a65fda5996585a3e7a50e1fb848fd:128 tx_count=73 time=1733803997
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Added L2 block to local state" block=0xc0e1614181ba48883fe512eec05ebabf6a1ccde09838f6c2021f921df8c4ee1b:129 tx_count=1 time=1733803999
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Added L2 block to local state" block=0x0596d7bfd0b93331f3e258404c7d7783ff27ef8d20553ee88f9d8c5093974db9:130 tx_count=1 time=1733804001
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0x2b9b6d4b866b34b3ba12df1d5b872ceeb817dc826cfb05c5d106cab770e55d0c nonce=20 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000 tx=0x2b9b6d4b866b34b3ba12df1d5b872ceeb817dc826cfb05c5d106cab770e55d0c
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Added L2 block to local state" block=0x28ebee23321a3c6be1790e968f52fcca10c0313b9bbbd2ed05774c60bca08d48:131 tx_count=101 time=1733804003
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Added L2 block to local state" block=0x6c239c58aa9bec321d841998b64bb31621dbe60e8c400f4e59fd72acc1553dc6:132 tx_count=1 time=1733804005
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Added L2 block to local state" block=0x1e1b3cfdecbbc0edd5cc01f167f04cb3b0a6406ae997bc03afbe0e96c0c3b96a:133 tx_count=149 time=1733804007
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Created channel" id=69aa531e62ac39741f6dc1ff804533c5 l1Head=0xbfba7276d721512e2f2c72e98f9d578d4e8f85f41f077dd0bed09d087417adf2:49 blocks_pending=6 l1OriginLastSubmittedChannel=0x5a6647ea9825686bba4ab8f43ae15d766d0189bfc963ce6a99ebebcb9f175c37:40 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=130043 use_blobs=true
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Channel closed" id=69aa531e62ac39741f6dc1ff804533c5 blocks_pending=0 num_frames=1 input_bytes=61935 output_bytes=53225 oldest_l1_origin=0xd97d78da8e6a8fe47ec85451161d927a8390e16d00e08c7260755f5fd3323bbe:41 l1_origin=0x579274746ac1867b3183f7577bc5273a8609dc01a4dbf6dcc3a7a6f18e7f6195:42 oldest_l2=0x81974e2fc7f9f5c7e5b130ba301e2179561a65fda5996585a3e7a50e1fb848fd:128 latest_l2=0x1e1b3cfdecbbc0edd5cc01f167f04cb3b0a6406ae997bc03afbe0e96c0c3b96a:133 full_reason="channel full: max channel duration reached" compr_ratio=0.8593686929845806
[op-batcher-op-kurtosis] t=2024-12-10T04:13:27+0000 lvl=info msg="Building Blob transaction candidate" size=53225 last_size=53225 num_blobs=1
  • Logs after restart
[op-batcher-op-kurtosis] t=2024-12-10T04:14:53+0000 lvl=info msg="Starting JSON-RPC server"
[op-batcher-op-kurtosis] t=2024-12-10T04:14:53+0000 lvl=info msg="Starting batcher" notSubmittingOnStart=false
[op-batcher-op-kurtosis] t=2024-12-10T04:14:53+0000 lvl=info msg="Starting Batch Submitter"
[op-batcher-op-kurtosis] t=2024-12-10T04:14:53+0000 lvl=info msg="Clearing state"
[op-batcher-op-kurtosis] t=2024-12-10T04:14:53+0000 lvl=info msg="Clearing state with safe L1 origin" origin=0x579274746ac1867b3183f7577bc5273a8609dc01a4dbf6dcc3a7a6f18e7f6195:42
[op-batcher-op-kurtosis] t=2024-12-10T04:14:53+0000 lvl=info msg="State cleared"
[op-batcher-op-kurtosis] t=2024-12-10T04:14:53+0000 lvl=info msg="Batch Submitter started"
[op-batcher-op-kurtosis] t=2024-12-10T04:14:53+0000 lvl=info msg="Starting DA throttling loop"
[op-batcher-op-kurtosis] t=2024-12-10T04:14:53+0000 lvl=info msg="Starting receipts processing loop"
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="no blocks in state" syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: <nil>, blocksToLoad: &{134 176}}"
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Loading range of multiple blocks into state" start=134 end=176
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x541694795137840056c5c69bbed07dd792f7c1b4197977494757092ba51d54b0:134 tx_count=1 time=1733804009
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x228935ede19c327090562f7f247e43dc18974f941d37076f37423682e04c6fb8:135 tx_count=1 time=1733804011
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x45e24542576fd0f7fee3496e214318b1f36a60a50b462206d89078a8bfcffe48:136 tx_count=101 time=1733804013
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x8d950652f29311bd3b8f573b9b97b0defa66c7c3e6f2f09f988813c193a80f03:137 tx_count=1 time=1733804015
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x57cfd702915abbcb099278862da1f46d5ac0ddcbc3e446f0a6783a9c1e32b5c3:138 tx_count=126 time=1733804017
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x9c878b03fd8f3bef495cffbe859e517bef89aa340efbf34481a2619d5ae28f8a:139 tx_count=1 time=1733804019
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x24bc491a7c9baa43ad0f06e944ccf9307737efb871d65f41f5f2919ee87165c8:140 tx_count=1 time=1733804021
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xe956a1d95ae4e03a1eb317836cc22fc01839ce7e9e3787779ee9fe2ae9b7bef0:141 tx_count=74 time=1733804023
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x4d9330c19755caf6d5dfaf87b8f2062e6bac2ac0b54bc76631c762dfd4766c09:142 tx_count=28 time=1733804025
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x6cb6384e29fd4e38a1104f781f7c56bb3efe2ba46768e437e82e8ae114ff4d5a:143 tx_count=1 time=1733804027
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xd134b77c891001595f96d8b08fa8dc7d3329a4e8591ba76d896197fe8f84ba45:144 tx_count=133 time=1733804029
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xb4f85165b296d4f666952b06079d68cfc69cd1d380c37800e97f79af9011016d:145 tx_count=1 time=1733804031
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x019abef09e83e187c98ae61f70da7bd16989162ccf4119bfef4ba0f2062af24e:146 tx_count=1 time=1733804033
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x484370f0afb23e664206d5114f5cc7fac996a4672b3c162113f307d6e7e9f1d7:147 tx_count=85 time=1733804035
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xa89c189c58093cbe932582484a3e70bf1185b6cffd92a44cde542a9b93788386:148 tx_count=17 time=1733804037
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x23373757c25c85cc2d525bfca41b05503b5d028c0fbfa377ac8ab997478da997:149 tx_count=1 time=1733804039
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x7915ce3a820ffa3c36d05c4a9403d943bc5374d5a7837fc7561f24712497e133:150 tx_count=137 time=1733804041
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x5b22c054a286e552399bcf8d4e2cc142dfa61332fbde7c1fb8f0e79c53afec1d:151 tx_count=45 time=1733804043
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x9a8cf0edc4ddbd1d20eb81158640a2073d05b2591960ff1a3f316f9bb15f3e78:152 tx_count=57 time=1733804045
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x19ca42b19bd11fa430913a1fa0e4290e0d8b47da27880e99f95ea4f6678d41c8:153 tx_count=1 time=1733804047
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x457a3f2d80f3c92bb651584c57130affb4056eda7ce6fc894ab21ec666e8181f:154 tx_count=117 time=1733804049
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xe485e674ed3e182a99c7e7cb9e6127866c92f0234d0c6f01e9b39e142de0c18f:155 tx_count=1 time=1733804051
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x0b13604ebcfa7042bfd9ca7c26fb5b61cd4fb7313958a580850141064134ef0c:156 tx_count=1 time=1733804053
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x2e3ef78b81a74748d51cf738a3a4b12da5af7f8be8a6b3cffcf04e21e23009b5:157 tx_count=67 time=1733804055
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x2bad9950d846d82cb4fdb136af1fccaba62912684609345f00c1f6cca4268319:158 tx_count=35 time=1733804057
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xfec05c5cf0b2240a68176f5069aa42aa667723b4d9fe576a1caf98759c0a411d:159 tx_count=1 time=1733804059
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xbf9ddb319b489a3076cd9194ca1468df28b0a8fc15c6bb4ee5b0644a9316e078:160 tx_count=133 time=1733804061
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x41ae1376a5ab8019a19d72ed8545b4e5c70ab0df091c4f576266ef0d310e7eb6:161 tx_count=1 time=1733804063
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x947560a47620b8a89a230aa48a60d210dd99b52dff66176c246ab549d5b51589:162 tx_count=1 time=1733804065
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xb754f993f72bf6b81336948e505ad09fa66e20464280bbce9a8d56ab8b89dacc:163 tx_count=84 time=1733804067
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x4dad6e79c76ca5f264b8610d95f6bbb8200a42f684931dbd777d4c14436024c1:164 tx_count=18 time=1733804069
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x72daac6a284dd78dbf80e535b675c9cf0f79ffe11fac00b53b4fbd9afba4a816:165 tx_count=1 time=1733804071
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xf9b8715601f6ca82a919141df8fde1be38c242b3310f7fb584e062537fdf94e7:166 tx_count=169 time=1733804073
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xd2c7281c5ea7ce2722326066b9fad6f693998334011f0dff1113448a27f369dc:167 tx_count=1 time=1733804075
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x2ee0cfad5318742c421ff44ae2968ced29140f904d2318e224b07e30c3c58fcb:168 tx_count=1 time=1733804077
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x006f6ca075545911711900965db532f721c2ac3be40a3d76aba87db177e345cb:169 tx_count=85 time=1733804079
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x2cdab790133ae51afd45cb1a2c06a28b1d5b24ef936f8d58960f95b82ad80dd2:170 tx_count=17 time=1733804081
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x11a027df3838ac210f0d81dc211e44d135221f9e9b8b2e981230875d07fe8f60:171 tx_count=1 time=1733804083
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x95ebc5d27af12de5afba3ca10dad55ca125827afd4b1f65d90b50663d6720c14:172 tx_count=160 time=1733804085
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x87ee79e6d3bc5099d7d5f7777468988be591f24f058f272c8cd19b37114f8e6f:173 tx_count=1 time=1733804087
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x658b4ef2784460679e544979a065121639c730ae84be32999b8bfdc01c73c688:174 tx_count=1 time=1733804089
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0xf872b15f8162d3e531c20e4cc2da1f9229262916725d6b59ddf4a7075c3f137a:175 tx_count=101 time=1733804091
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Added L2 block to local state" block=0x457dcefb7d24ca18f1df282e1fe7254814a07750b91978fc5c242460105c3dbd:176 tx_count=1 time=1733804093
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Created channel" id=421aa55ea54d4d41a23815bb1eae0be0 l1Head=0x2e5558686c7a5fd49d8e66498b3a1211387a59e1b2570c82b1dcb291f4a1f15b:63 blocks_pending=43 l1OriginLastSubmittedChannel=0x579274746ac1867b3183f7577bc5273a8609dc01a4dbf6dcc3a7a6f18e7f6195:42 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=130043 use_blobs=true
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Channel closed" id=421aa55ea54d4d41a23815bb1eae0be0 blocks_pending=23 num_frames=1 input_bytes=163589 output_bytes=117026 oldest_l1_origin=0x2bd648a16df07ee185ebe243c5cab69b64f975ef217a112cded8f06d71390ebc:43 l1_origin=0xbfba7276d721512e2f2c72e98f9d578d4e8f85f41f077dd0bed09d087417adf2:49 oldest_l2=0x541694795137840056c5c69bbed07dd792f7c1b4197977494757092ba51d54b0:134 latest_l2=0x19ca42b19bd11fa430913a1fa0e4290e0d8b47da27880e99f95ea4f6678d41c8:153 full_reason="channel full: compressor is full" compr_ratio=0.715365947588163
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Building Blob transaction candidate" size=117026 last_size=117026 num_blobs=1
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x822bad9422895eb877cd2ed0884ab64f6350a3ca85daa00626961b99569d0196 nonce=22 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Created channel" id=ae64fa41a1a30a0926697e345ef8064d l1Head=0x2e5558686c7a5fd49d8e66498b3a1211387a59e1b2570c82b1dcb291f4a1f15b:63 blocks_pending=23 l1OriginLastSubmittedChannel=0xbfba7276d721512e2f2c72e98f9d578d4e8f85f41f077dd0bed09d087417adf2:49 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=130043 use_blobs=true
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Channel closed" id=ae64fa41a1a30a0926697e345ef8064d blocks_pending=5 num_frames=1 input_bytes=167116 output_bytes=111926 oldest_l1_origin=0xbfba7276d721512e2f2c72e98f9d578d4e8f85f41f077dd0bed09d087417adf2:49 l1_origin=0xd4d85d240f230860490fdc632c837e24549b12e6df3ea6b7477633ce836ca614:55 oldest_l2=0x457a3f2d80f3c92bb651584c57130affb4056eda7ce6fc894ab21ec666e8181f:154 latest_l2=0x11a027df3838ac210f0d81dc211e44d135221f9e9b8b2e981230875d07fe8f60:171 full_reason="channel full: compressor is full" compr_ratio=0.6697503530481821
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Building Blob transaction candidate" size=111926 last_size=111926 num_blobs=1
[op-batcher-op-kurtosis] t=2024-12-10T04:14:54+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0x822bad9422895eb877cd2ed0884ab64f6350a3ca85daa00626961b99569d0196 nonce=22 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000 tx=0x822bad9422895eb877cd2ed0884ab64f6350a3ca85daa00626961b99569d0196
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Transaction confirmed" service=batcher tx=0x822bad9422895eb877cd2ed0884ab64f6350a3ca85daa00626961b99569d0196 block=0xbe824bdfcaa830a52989acb11fea1be034edbca0d6bb9d3e8f85ab015ae00285:65 effectiveGasPrice=5000280021
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Handling receipt" id=421aa55ea54d4d41a23815bb1eae0be0:0
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Transaction confirmed" tx_id=421aa55ea54d4d41a23815bb1eae0be0:0 tx=0x822bad9422895eb877cd2ed0884ab64f6350a3ca85daa00626961b99569d0196 block=0xbe824bdfcaa830a52989acb11fea1be034edbca0d6bb9d3e8f85ab015ae00285:65
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Channel is fully submitted" id=421aa55ea54d4d41a23815bb1eae0be0 min_inclusion_block=65 max_inclusion_block=65
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=warn msg="Aborting state publishing, max duration exceeded"
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Publishing transaction" service=batcher tx=0xdeccdb5d31fe3ef413d4d35ea98f88805282e650068baf41e7d33240bca11dbd nonce=23 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Loading range of multiple blocks into state" start=177 end=182
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Added L2 block to local state" block=0xa95d19e1f4592fd1ee563697f55a62aef5a690645930a2613ee426a2f81cad56:177 tx_count=138 time=1733804095
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Added L2 block to local state" block=0x1562530eb5aa9a19e35f94a741858b3a69e35c1029756cb17085e4429552e98a:178 tx_count=1 time=1733804097
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0xdeccdb5d31fe3ef413d4d35ea98f88805282e650068baf41e7d33240bca11dbd nonce=23 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000 tx=0xdeccdb5d31fe3ef413d4d35ea98f88805282e650068baf41e7d33240bca11dbd
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Added L2 block to local state" block=0xf6b15822aef1d2d3168cc86897663635c9982f70c5306eb4c8a4a04e21f4156b:179 tx_count=1 time=1733804099
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Added L2 block to local state" block=0x89d93cebd72ef77fe65710b683454949dc613eaa19a2274b4321bf44351529a1:180 tx_count=66 time=1733804101
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Added L2 block to local state" block=0x74fca39a416cf5da144ab87fb78c8e544e56603d1cb93b040d6df2a0d19b469c:181 tx_count=36 time=1733804103
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Added L2 block to local state" block=0x3a83f62843e681aa53d468cfb0e03dc6d93d5c3426802c6f94df55a41694281c:182 tx_count=1 time=1733804105
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Created channel" id=4e9e200e7830989e9099d9dde2f6fa8b l1Head=0xbe824bdfcaa830a52989acb11fea1be034edbca0d6bb9d3e8f85ab015ae00285:65 blocks_pending=11 l1OriginLastSubmittedChannel=0xd4d85d240f230860490fdc632c837e24549b12e6df3ea6b7477633ce836ca614:55 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=130043 use_blobs=true
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Channel closed" id=4e9e200e7830989e9099d9dde2f6fa8b blocks_pending=0 num_frames=1 input_bytes=91272 output_bytes=78106 oldest_l1_origin=0xd4d85d240f230860490fdc632c837e24549b12e6df3ea6b7477633ce836ca614:55 l1_origin=0x7a9c6859341c6c8bdc342de397e0c0090bc45e581ec13533bf0a9d2edff9001e:58 oldest_l2=0x95ebc5d27af12de5afba3ca10dad55ca125827afd4b1f65d90b50663d6720c14:172 latest_l2=0x3a83f62843e681aa53d468cfb0e03dc6d93d5c3426802c6f94df55a41694281c:182 full_reason="channel full: max channel duration reached" compr_ratio=0.8557498466123236
[op-batcher-op-kurtosis] t=2024-12-10T04:15:06+0000 lvl=info msg="Building Blob transaction candidate" size=78106 last_size=78106 num_blobs=1
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Transaction confirmed" service=batcher tx=0xdeccdb5d31fe3ef413d4d35ea98f88805282e650068baf41e7d33240bca11dbd block=0x94f6b7019261f8027f9b66b0cb6bebcc51c6718f1453265ff3863404dbc96af5:67 effectiveGasPrice=5000214435
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Handling receipt" id=ae64fa41a1a30a0926697e345ef8064d:0
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Transaction confirmed" tx_id=ae64fa41a1a30a0926697e345ef8064d:0 tx=0xdeccdb5d31fe3ef413d4d35ea98f88805282e650068baf41e7d33240bca11dbd block=0x94f6b7019261f8027f9b66b0cb6bebcc51c6718f1453265ff3863404dbc96af5:67
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Channel is fully submitted" id=ae64fa41a1a30a0926697e345ef8064d min_inclusion_block=67 max_inclusion_block=67
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=warn msg="Aborting state publishing, max duration exceeded"
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Publishing transaction" service=batcher tx=0xf995f3aae185486239a868494eb0c1ec82f27ae3313dd9f6fe1f283498e93da1 nonce=24 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Loading range of multiple blocks into state" start=183 end=188
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Added L2 block to local state" block=0x8e1939559007212005049e7d18b8c3076da672d08fa33a1a8d0293a12bc26d25:183 tx_count=124 time=1733804107
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0xf995f3aae185486239a868494eb0c1ec82f27ae3313dd9f6fe1f283498e93da1 nonce=24 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000 tx=0xf995f3aae185486239a868494eb0c1ec82f27ae3313dd9f6fe1f283498e93da1
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Added L2 block to local state" block=0x83d706a89345df71c97d64cc657f57675001a5ac109bde9ac555f957957d6136:184 tx_count=1 time=1733804109
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Added L2 block to local state" block=0x43bcc57acb51de179f552f68428ec182d89f8126f8cce0740d39403c83e0d46c:185 tx_count=1 time=1733804111
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Added L2 block to local state" block=0xf2066786d4181e24c4a430c07e4239f30d2ffd43ee37f2ead1ba13d1ddb31eb9:186 tx_count=84 time=1733804113
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Added L2 block to local state" block=0x6b4891ac6924d64dff26d73e2b03d6909df90c70844db52d1315a092b332eca5:187 tx_count=18 time=1733804115
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Added L2 block to local state" block=0x957a9ffaba75ad3c0fc7269f9e015148f22fdc3f49955c6126567e69b2e3f47b:188 tx_count=1 time=1733804117
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Created channel" id=9ddc739916686f98849924ccb9fb1a79 l1Head=0x94f6b7019261f8027f9b66b0cb6bebcc51c6718f1453265ff3863404dbc96af5:67 blocks_pending=6 l1OriginLastSubmittedChannel=0x7a9c6859341c6c8bdc342de397e0c0090bc45e581ec13533bf0a9d2edff9001e:58 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=130043 use_blobs=true
[op-batcher-op-kurtosis] t=2024-12-10T04:15:18+0000 lvl=info msg="Channel closed" id=9ddc739916686f98849924ccb9fb1a79 blocks_pending=0 num_frames=1 input_bytes=39933 output_bytes=34326 oldest_l1_origin=0x28cb23a5a2d63b679d3a401539e4ed54049573789839fc185bfe4c0026a832b4:59 l1_origin=0x3dbcf3065b6db88aca8a837a4d899d101d5ed40140e6f9014286cfadfb57f16c:60 oldest_l2=0x8e1939559007212005049e7d18b8c3076da672d08fa33a1a8d0293a12bc26d25:183 latest_l2=0x957a9ffaba75ad3c0fc7269f9e015148f22fdc3f49955c6126567e69b2e3f47b:188 full_reason="channel full: max channel duration reached" compr_ratio=0.8595898129366689

@vdamle
Copy link
Contributor Author

vdamle commented Dec 10, 2024

Batcher and Sequencer restart in quick succession

restart timestamp: 2024-12-10T15:39:37+0000

Sequencer log snippets

  • Before restart, noting 2 L1 safe heads
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:32:06+0000 lvl=info msg="Scheduled sequencer action" delta=5.709564ms
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:32:07+0000 lvl=info msg="Started sequencing new block" parent=0xd9ea715bb251957318fc97e883316c820a12de0072c02881bf09a42a4e621855:197 l1Origin=0xa1a4e158751f8287681fc9c46fbecd047d33e13eddb1fea0340bd665ebcf8bba:63
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:32:07+0000 lvl=info msg="Scheduled sequencer action" delta=1.921258519s
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:32:08+0000 lvl=info msg="New L1 finalized block" l1_finalized=0xb08e6e54ea9e26da666c3e3431e9fc1364a54eac9c960fc7076cf35d20b5c4cd:48
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:32:08+0000 lvl=info msg="New L1 safe block" l1_safe=0xa4d4888af92b797ca0dc6fbd71966bf3a4ec84c7d2c45c485d6c4dc819316f6f:56
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:32:08+0000 lvl=warn msg="L1 head signal indicates a possible L1 re-org" old_l1_head=0xe290b765f81d7f3653ec3661ecefd19d46881a534222008d6c4903c55aaa63ff:68 new_l1_head_parent=0x2ba6328507389dfc95a655a7a5b23150880dc08234cda0cc13174dd31b2987e5 new_l1_head=0xd1b1afd7bfc621a61d210860f4c359be9b88b96ac25795aa57015ee0e1ad3125:70

[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:38:30+0000 lvl=info msg="Scheduled sequencer action" delta=32.653032ms
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:38:31+0000 lvl=info msg="Started sequencing new block" parent=0x46fe991547861504a3dd7b4485327de429e3025c1802fa225bb0cf9611ab60a8:389 l1Origin=0x49a100d26be70d889202bc411d6fcadcfdc5f795b83da35a1f1e68f9ea8aa312:127
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:38:31+0000 lvl=info msg="Scheduled sequencer action" delta=1.942107488s
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:38:32+0000 lvl=info msg="New L1 safe block" l1_safe=0xfdbfea5f647c5993a87749ff569b337465f46ca19519b517758c82e9cf8badda:120
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:38:32+0000 lvl=info msg="New L1 finalized block" l1_finalized=0x7baf0c34ab82b424222494244dd25558a0ed9333026bd34702cce7d07e7b4ce0:112
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:38:32+0000 lvl=warn msg="L1 head signal indicates a possible L1 re-org" old_l1_head=0xac85f67cb0104068a0adba5d38b8a2dd8203b58ed0a16e0509d68abc11251364:132 new_l1_head_parent=0x1e6b44f4e7659588515e5bd2430d23b8b5ef13d592b3085c1443bba55bc1f4a5 new_l1_head=0x0d6d8c8c0572ecc47644215b2843f59b7895d9b9fbde462fd3425ed47ba55252:134
  • After restart
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:25+0000 lvl=info msg="Scheduled sequencer action" delta=-83.158µs
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:25+0000 lvl=warn msg="Deriver system is resetting" err="reset: cannot continue derivation until Engine has been reset"
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:25+0000 lvl=error msg="Sequencer encountered reset signal, aborting work" err="reset: cannot continue derivation until Engine has been reset"
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:25+0000 lvl=info msg="Loaded current L2 heads" unsafe=0x17bdbe63d705fc35bfdc1fa0b265a0ccf02d1c5e1a5922bda80fefbb899d60a5:454 safe=0xebaa445bf421001d1af6a3633461622d1f3b2e065717c2e7226a9eaab99cff55:422 finalized=0x86e558d5bc796a6594b11633cb1dc473020b77626229e8b690acf6c06462153c:313 unsafe_origin=0xe9cb66d85355023e73bb5e4d0b5744e1cdec0ffa45b62be216f043196a5b2a22:149 safe_origin=0x600e1f6610b6aa8bfd697959ab267e8966741f8c69b346506c6a06bb7ff23f66:139
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:25+0000 lvl=info msg="Walking back L1Block by number" curr=0xe9cb66d85355023e73bb5e4d0b5744e1cdec0ffa45b62be216f043196a5b2a22:149 next=0xe9cb66d85355023e73bb5e4d0b5744e1cdec0ffa45b62be216f043196a5b2a22:149 l2block=0x17bdbe63d705fc35bfdc1fa0b265a0ccf02d1c5e1a5922bda80fefbb899d60a5:454
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:25+0000 lvl=info msg="Walking back L1Block by hash" curr=0xe9cb66d85355023e73bb5e4d0b5744e1cdec0ffa45b62be216f043196a5b2a22:149 next=0xaa577de407eeff8c6ffaf0ec74bd51f9b6c6e97811a090699b6fa6abcc8d7f43:148 l2block=0xe672320693224215acdb583f479dbfda26d1944da7bb1bd4a0769f60027c9e1c:451
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:25+0000 lvl=info msg="Walking back L1Block by hash" curr=0xaa577de407eeff8c6ffaf0ec74bd51f9b6c6e97811a090699b6fa6abcc8d7f43:148 next=0xc5824ec99c50aaba39de6bbe5d3be837bed44c5341ef668d2065610520c090bc:147 l2block=0x97b94da16c7c799f07ff0175cb5a2cbbec928dc56834e30de070475334ac8948:450
...
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Walking back L1Block by hash" curr=0x081da71dda149287ba3372388b03d117d1efded4b6917dc4316abf11fef5607e:104 next=0x8a50f131a33077abaf04ff75915f9563f88230c26eb9410d21f9dbe021621f6c:103 l2block=0xe6595d92caf635c4032fd91d6a5871bcd7c00889e1f1de0e9d8c7944f043d277:318
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Walking back L1Block by hash" curr=0x8a50f131a33077abaf04ff75915f9563f88230c26eb9410d21f9dbe021621f6c:103 next=0xde45f221d4c8ff7d331e22edad20efce98671ab3440e5bfed20c289037dd76ec:102 l2block=0x86e558d5bc796a6594b11633cb1dc473020b77626229e8b690acf6c06462153c:313
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Hit finalized L2 head, returning immediately" unsafe=0x17bdbe63d705fc35bfdc1fa0b265a0ccf02d1c5e1a5922bda80fefbb899d60a5:454 safe=0xebaa445bf421001d1af6a3633461622d1f3b2e065717c2e7226a9eaab99cff55:422 finalized=0x86e558d5bc796a6594b11633cb1dc473020b77626229e8b690acf6c06462153c:313 unsafe_origin=0xe9cb66d85355023e73bb5e4d0b5744e1cdec0ffa45b62be216f043196a5b2a22:149 safe_origin=0x600e1f6610b6aa8bfd697959ab267e8966741f8c69b346506c6a06bb7ff23f66:139
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Current hardfork version detected" forkName=holocene
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Engine reset confirmed, sequencer may continue" next=true
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Scheduled sequencer action" delta=-9.457µs
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Started sequencing new block" parent=0x17bdbe63d705fc35bfdc1fa0b265a0ccf02d1c5e1a5922bda80fefbb899d60a5:454 l1Origin=0xb1e4917e9a9ea1070f7249d205ef1bcc71fedb7f5329ae83fc73ea8e56a5bf70:150
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Scheduled sequencer action" delta=-3µs
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Sequencer sealed block" payloadID=0x03d7d6ead86770a7 block=0xb0f96f12728acccc3d08aae0f80c6874b86555921e11923069a9be13382a2ce2:455 parent=0x17bdbe63d705fc35bfdc1fa0b265a0ccf02d1c5e1a5922bda80fefbb899d60a5:454 txs=28 time=1733845243
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Publishing signed execution payload on p2p" id=0xb0f96f12728acccc3d08aae0f80c6874b86555921e11923069a9be13382a2ce2:455
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Sequencer inserted block" block=0xb0f96f12728acccc3d08aae0f80c6874b86555921e11923069a9be13382a2ce2:455 parent=0x17bdbe63d705fc35bfdc1fa0b265a0ccf02d1c5e1a5922bda80fefbb899d60a5:454
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0xb0f96f12728acccc3d08aae0f80c6874b86555921e11923069a9be13382a2ce2 number=455 state_root=0x1cde93edb41180e701a9191ac769eb885651d5fb0d37f8c68a53c836f83048bf timestamp=1733845243 parent=0x17bdbe63d705fc35bfdc1fa0b265a0ccf02d1c5e1a5922bda80fefbb899d60a5 prev_randao=0x9ed1df7e0d8ba25149303f56d78f0cba9038206aaf026293d6c6fabea3b1ee8c fee_recipient=0x4200000000000000000000000000000000000011 txs=28 build_time=14.482ms insert_time=24.367ms total_time=38.849ms mgas=0.616406 mgasps=15.866373692449402
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Scheduled sequencer action" delta=-20.581µs
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Started sequencing new block" parent=0xb0f96f12728acccc3d08aae0f80c6874b86555921e11923069a9be13382a2ce2:455 l1Origin=0x4f76fc49e0ec77ac3b4f7dda00e302d10d0e7b480471ffd56081a688b69bde93:151
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Scheduled sequencer action" delta=-1.375µs
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Sequencer sealed block" payloadID=0x03b64dcb8d9f7fb0 block=0x75513b2b542dc0117b4b0f741f150d3da27f61b060a77f95c1de7b7c5ea0784d:456 parent=0xb0f96f12728acccc3d08aae0f80c6874b86555921e11923069a9be13382a2ce2:455 txs=1 time=1733845245
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Publishing signed execution payload on p2p" id=0x75513b2b542dc0117b4b0f741f150d3da27f61b060a77f95c1de7b7c5ea0784d:456
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Sequencer inserted block" block=0x75513b2b542dc0117b4b0f741f150d3da27f61b060a77f95c1de7b7c5ea0784d:456 parent=0xb0f96f12728acccc3d08aae0f80c6874b86555921e11923069a9be13382a2ce2:455
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0x75513b2b542dc0117b4b0f741f150d3da27f61b060a77f95c1de7b7c5ea0784d number=456 state_root=0x6c380f1024385731f1602cfa515b0d28220ebb543dde8b6ee9b2f62098cbe825 timestamp=1733845245 parent=0xb0f96f12728acccc3d08aae0f80c6874b86555921e11923069a9be13382a2ce2 prev_randao=0x6569742e45d2a0f785def0221741bf8f5b3c440f375fec5b856e062c8b5eb223 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 build_time=3.300ms insert_time=9.887ms total_time=13.188ms mgas=0.046594 mgasps=3.532899625434087
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Scheduled sequencer action" delta=-14.29µs
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Started sequencing new block" parent=0x75513b2b542dc0117b4b0f741f150d3da27f61b060a77f95c1de7b7c5ea0784d:456 l1Origin=0xd4d6671efa7b30281b8554eb33aeeb592cac7e76d5f16762091a9efc759a996c:152
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Scheduled sequencer action" delta=-1.5µs
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Sequencer sealed block" payloadID=0x03acf080cd5b3a3a block=0x0dc0df7cbb24987d9e9dcaa56651c9f611236de7c062a60fba5ce1bcab0ee247:457 parent=0x75513b2b542dc0117b4b0f741f150d3da27f61b060a77f95c1de7b7c5ea0784d:456 txs=1 time=1733845247
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Publishing signed execution payload on p2p" id=0x0dc0df7cbb24987d9e9dcaa56651c9f611236de7c062a60fba5ce1bcab0ee247:457
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Sequencer inserted block" block=0x0dc0df7cbb24987d9e9dcaa56651c9f611236de7c062a60fba5ce1bcab0ee247:457 parent=0x75513b2b542dc0117b4b0f741f150d3da27f61b060a77f95c1de7b7c5ea0784d:456
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0x0dc0df7cbb24987d9e9dcaa56651c9f611236de7c062a60fba5ce1bcab0ee247 number=457 state_root=0x03443b12ba1d159f5e36e5c91309eac1bdbe88502a0aaa448615751827631771 timestamp=1733845247 parent=0x75513b2b542dc0117b4b0f741f150d3da27f61b060a77f95c1de7b7c5ea0784d prev_randao=0x3efdc18aa2c5478162f3db79e7de2a023456c23165b182ae1909bad505d7a038 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 build_time=3.445ms insert_time=9.696ms total_time=13.142ms mgas=0.046606 mgasps=3.5463227085182356
[op-cl-1-op-node-op-geth-op-kurtosis] t=2024-12-10T15:41:26+0000 lvl=info msg="Scheduled sequencer action" delta=-14.082µs

Batcher code snippets

  • After restart
[op-batcher-op-kurtosis] t=2024-12-10T15:41:27+0000 lvl=info msg="Starting Batch Submitter"
[op-batcher-op-kurtosis] t=2024-12-10T15:41:27+0000 lvl=info msg="Clearing state"
[op-batcher-op-kurtosis] t=2024-12-10T15:41:27+0000 lvl=info msg="Clearing state with safe L1 origin" origin=0xde45f221d4c8ff7d331e22edad20efce98671ab3440e5bfed20c289037dd76ec:102
[op-batcher-op-kurtosis] t=2024-12-10T15:41:27+0000 lvl=info msg="State cleared"
[op-batcher-op-kurtosis] t=2024-12-10T15:41:27+0000 lvl=info msg="Batch Submitter started"
[op-batcher-op-kurtosis] t=2024-12-10T15:41:27+0000 lvl=info msg="Starting DA throttling loop"
[op-batcher-op-kurtosis] t=2024-12-10T15:41:27+0000 lvl=info msg="Starting receipts processing loop"
[op-batcher-op-kurtosis] t=2024-12-10T15:41:28+0000 lvl=info msg="Received empty sync status, backing off" backoff=1s
[op-batcher-op-kurtosis] t=2024-12-10T15:41:29+0000 lvl=info msg="Received empty sync status, backing off" backoff=2s
[op-batcher-op-kurtosis] t=2024-12-10T15:41:31+0000 lvl=info msg="Received empty sync status, backing off" backoff=4s
[op-batcher-op-kurtosis] t=2024-12-10T15:41:35+0000 lvl=info msg="Received empty sync status, backing off" backoff=8s
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="no blocks in state" syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: <nil>, blocksToLoad: &{441 485}}"
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Loading range of multiple blocks into state" start=441 end=485
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" block=0x46680c7df828b5cbcabb0d60c9e0cc4284cec8b81c11481b1c71e6bd2e527286:441 tx_count=1 time=1733845215
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" block=0x23550cae19a6ba939f188b37271f63f6b8056f8d4f9cd4295491344c100e56c2:442 tx_count=144 time=1733845217
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" block=0xfab0abf2898f922550307b5fe4de400a90d9c6458dd32b24f692153aebb7f886:443 tx_count=1 time=1733845219
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" block=0x57aecf6cde3ac8ead9539d1f79fdc17be59a33a0bef2179eeb9841133ce18338:444 tx_count=1 time=1733845221
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" ...[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" block=0x097ccaa2440c11f7f2332feeda17a5bde10642340bcf9ec02c54f7f74b8532cc:481 tx_count=1 time=1733845295
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" block=0x599224bd38d2ddc1d8948c50df07321326979b579d81188f21e5c48ae77a72bc:482 tx_count=101 time=1733845297
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" block=0x70ccad142bff6a4e772b5ecff729feb2105de6500b7d6b692f82e3739e0f9be7:483 tx_count=1 time=1733845299
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" block=0x0d2516391c6ee0a273787825d57d7ce75046a7302bcc24f07f29a37b6c6eb7d6:484 tx_count=147 time=1733845301
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Added L2 block to local state" block=0x5a7cbac3d678ea3aa29f5489824a293425697a74d4656888fbd73d564f2b090f:485 tx_count=35 time=1733845303
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Created channel" id=3a3da8b0f2e1cdc0edef09998d0ed6b4 l1Head=0xee0f2633fd080fdb70b506a509f8ff38f433e2d0f6416ccbbaab78c26756b44b:166 blocks_pending=45 l1OriginLastSubmittedChannel=0xde45f221d4c8ff7d331e22edad20efce98671ab3440e5bfed20c289037dd76ec:102 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=130043 use_blobs=true
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Channel closed" id=3a3da8b0f2e1cdc0edef09998d0ed6b4 blocks_pending=7 num_frames=1 input_bytes=166985 output_bytes=113484 oldest_l1_origin=0xa8083d631269b98a50b5cf355bf148028c4c30772c7e192ed7cfea9695f87338:145 l1_origin=0x40da472d26dd0c63ed220b4285e5c75d330e010c5e993ade1fb6d1de033222e8:163 oldest_l2=0x46680c7df828b5cbcabb0d60c9e0cc4284cec8b81c11481b1c71e6bd2e527286:441 latest_l2=0xcddc7b6cfb4497459d3e81a393ea7ae8fddf89139c2272f0cab6722610341148:478 full_reason="channel full: compressor is full" compr_ratio=0.6796059526304757
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Building Blob transaction candidate" size=113484 last_size=113484 num_blobs=1
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x1ef11772e73a54845daf77c7175bd3c59543f7ea10ab3d98c33f3ca506da5642 nonce=73 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Created channel" id=b5f0e8303bd79c92352199e2f9ee6017 l1Head=0xee0f2633fd080fdb70b506a509f8ff38f433e2d0f6416ccbbaab78c26756b44b:166 blocks_pending=7 l1OriginLastSubmittedChannel=0x40da472d26dd0c63ed220b4285e5c75d330e010c5e993ade1fb6d1de033222e8:163 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=130043 use_blobs=true
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Channel closed" id=b5f0e8303bd79c92352199e2f9ee6017 blocks_pending=0 num_frames=1 input_bytes=87250 output_bytes=74993 oldest_l1_origin=0x750c62a61ff29b5b00133c8a79d139a3aaba49cacfb6a8f429a4c15e7746b8e1:164 l1_origin=0x670e71abf44c07c27de2cb185950124e77e2d920eedf57f4d4fcd85fe1fb8f59:165 oldest_l2=0x2056049b758a42983b86654affd63a741927dd25ec940f4d62065b6be641d5dc:479 latest_l2=0x5a7cbac3d678ea3aa29f5489824a293425697a74d4656888fbd73d564f2b090f:485 full_reason="channel full: max channel duration reached" compr_ratio=0.8595186246418338
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Building Blob transaction candidate" size=74993 last_size=74993 num_blobs=1
[op-batcher-op-kurtosis] t=2024-12-10T15:41:43+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0x1ef11772e73a54845daf77c7175bd3c59543f7ea10ab3d98c33f3ca506da5642 nonce=73 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000 tx=0x1ef11772e73a54845daf77c7175bd3c59543f7ea10ab3d98c33f3ca506da5642
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Transaction confirmed" service=batcher tx=0x1ef11772e73a54845daf77c7175bd3c59543f7ea10ab3d98c33f3ca506da5642 block=0x700d5642288a47b5295794c22441b13abc34adab54f5bc8d5447c6f8ed13a772:168 effectiveGasPrice=5000000007
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Handling receipt" id=3a3da8b0f2e1cdc0edef09998d0ed6b4:0
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Transaction confirmed" tx_id=3a3da8b0f2e1cdc0edef09998d0ed6b4:0 tx=0x1ef11772e73a54845daf77c7175bd3c59543f7ea10ab3d98c33f3ca506da5642 block=0x700d5642288a47b5295794c22441b13abc34adab54f5bc8d5447c6f8ed13a772:168
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Channel is fully submitted" id=3a3da8b0f2e1cdc0edef09998d0ed6b4 min_inclusion_block=168 max_inclusion_block=168
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=warn msg="Aborting state publishing, max duration exceeded"
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x538dec595a9362e7a915a43624836f464dac043c63e14d05d09816b2d50e880d nonce=74 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Loading range of multiple blocks into state" start=486 end=491
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Added L2 block to local state" block=0xec2e5dbbc4a8ca28bf9cbf0e4c2dcb94b813a015b6dc4ebbf7d2ba18f6778aaa:486 tx_count=1 time=1733845305
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Added L2 block to local state" block=0x352660e8fd4c5cf6631cb83ea2935c878503a4fd8787f1fd580295ac7e021717:487 tx_count=101 time=1733845307
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Added L2 block to local state" block=0x4d67fd24f169bb1173d73d72238d4a9281e978bce34957ff6883a2a1abda154b:488 tx_count=1 time=1733845309
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0x538dec595a9362e7a915a43624836f464dac043c63e14d05d09816b2d50e880d nonce=74 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000 tx=0x538dec595a9362e7a915a43624836f464dac043c63e14d05d09816b2d50e880d
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Added L2 block to local state" block=0x95f4ed24516747e548ce9d6e3f2bea4c7eb43d6180a7063f7d5c20de5c46accb:489 tx_count=134 time=1733845311
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Added L2 block to local state" block=0xd7776b4d28902cfd5182165dbe5eaeb9ed23c6d880567d44d0c2db15e2b85f76:490 tx_count=1 time=1733845313
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Added L2 block to local state" block=0x3c4e1fc5d0c9f03b6d12600f64e7a8342c6665923278aa1a6eef7ca4a49449d8:491 tx_count=1 time=1733845315
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Created channel" id=f8053f8dcbcb2833167beec96595ec21 l1Head=0x700d5642288a47b5295794c22441b13abc34adab54f5bc8d5447c6f8ed13a772:168 blocks_pending=6 l1OriginLastSubmittedChannel=0x670e71abf44c07c27de2cb185950124e77e2d920eedf57f4d4fcd85fe1fb8f59:165 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=130043 use_blobs=true
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Channel closed" id=f8053f8dcbcb2833167beec96595ec21 blocks_pending=0 num_frames=1 input_bytes=42426 output_bytes=36300 oldest_l1_origin=0x670e71abf44c07c27de2cb185950124e77e2d920eedf57f4d4fcd85fe1fb8f59:165 l1_origin=0x670e71abf44c07c27de2cb185950124e77e2d920eedf57f4d4fcd85fe1fb8f59:165 oldest_l2=0xec2e5dbbc4a8ca28bf9cbf0e4c2dcb94b813a015b6dc4ebbf7d2ba18f6778aaa:486 latest_l2=0x3c4e1fc5d0c9f03b6d12600f64e7a8342c6665923278aa1a6eef7ca4a49449d8:491 full_reason="channel full: max channel duration reached" compr_ratio=0.8556074105501343
[op-batcher-op-kurtosis] t=2024-12-10T15:41:55+0000 lvl=info msg="Building Blob transaction candidate" size=36300 last_size=36300 num_blobs=1
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Transaction confirmed" service=batcher tx=0x538dec595a9362e7a915a43624836f464dac043c63e14d05d09816b2d50e880d block=0x3d77a1ffce4ccb1f751d32daf92611918f30dd627d1c0d6f184db0d20a20e864:170 effectiveGasPrice=5000000007
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Handling receipt" id=b5f0e8303bd79c92352199e2f9ee6017:0
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Transaction confirmed" tx_id=b5f0e8303bd79c92352199e2f9ee6017:0 tx=0x538dec595a9362e7a915a43624836f464dac043c63e14d05d09816b2d50e880d block=0x3d77a1ffce4ccb1f751d32daf92611918f30dd627d1c0d6f184db0d20a20e864:170
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Channel is fully submitted" id=b5f0e8303bd79c92352199e2f9ee6017 min_inclusion_block=170 max_inclusion_block=170
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=warn msg="Aborting state publishing, max duration exceeded"
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x46aa881e01465ca70dd137440fb24f286e4ccee23e67e6cd98dcada0e7db8bc3 nonce=75 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Loading range of multiple blocks into state" start=492 end=497
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Added L2 block to local state" block=0xfaf7b00d20958105d265e8ece220e2c03bef2e876009254c9d28bb8bcfdb15d7:492 tx_count=70 time=1733845317
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Added L2 block to local state" block=0x581eca0d5bc1c639cfc2015ae539bf8954e400fb796f04950e015396de36ff04:493 tx_count=32 time=1733845319
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Added L2 block to local state" block=0xe0c4f46e6bd499c7dc053a38d2cc3e3915054d8c38142cd6b6b744dd5942e5d9:494 tx_count=1 time=1733845321
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0x46aa881e01465ca70dd137440fb24f286e4ccee23e67e6cd98dcada0e7db8bc3 nonce=75 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=21000 blobs=1 blobFeeCap=1000000000 tx=0x46aa881e01465ca70dd137440fb24f286e4ccee23e67e6cd98dcada0e7db8bc3
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Added L2 block to local state" block=0x4fdabb0567e62a37e86b6f151e53c3665251a0b97ddde0ca6c2055680f229ba0:495 tx_count=137 time=1733845323
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Added L2 block to local state" block=0x131b8ca84e015d700690026359aaed33fe79a53fcdd3c03d65abae9a798a06ea:496 tx_count=1 time=1733845325
[op-batcher-op-kurtosis] t=2024-12-10T15:42:07+0000 lvl=info msg="Added L2 block to local state" block=0x3c3d2a5a3a6a455294cf3e7aed476d049c57f0e0be653d19c3845b9e59b49619:497 tx_count=1 time=1733845327

batcher-logs-before-after-restart.log
sequencer-logs-before-after-restart.log

@vdamle
Copy link
Contributor Author

vdamle commented Dec 11, 2024

Holocene Straddle Awareness in Batcher

As noted in: #12122

Logs from recreate in Kurtosis attached.

  • Holocene activation at block 50
  • Batcher parameters: --max-channel-duration=60, --max-l1-tx-size-bytes=10000, --target-num-frames=5, --data-availability-type=calldata, --batch-type=0 (calldata)
  • Batcher image: 1.10.0-rc3
  • Kurtosis package network params
optimism_package:
  chains:
    - participants:
      - el_type: op-geth
        cl_type: op-node
      - el_type: op-reth
        el_image: "op-reth:v1.1.3"
        cl_type: op-node
      network_params:
        fjord_time_offset: 0
        granite_time_offset: 0
        holocene_time_offset: 100
      batcher_params:
        image: us-docker.pkg.dev/oplabs-tools-artifacts/images/op-batcher:1.10.0
        extra_params: [--max-channel-duration=60, --max-l1-tx-size-bytes=10000, --target-num-frames=5, --data-availability-type=calldata, --batch-type=0]
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Handling receipt" id=e092b07d4dfc60b20de2f0242f4db417:2
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Transaction confirmed" tx_id=e092b07d4dfc60b20de2f0242f4db417:2 tx=0x6ce9e260d89ce1282611903391fedd827be0f0129fc7f044ca6d93fe7392ada6 block=0x79ae6587da1a49c449cabf9b9cca215f34421ce9166e71fb7e4f4e1a7ec9990f:30
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=warn msg="Aborting state publishing, max duration exceeded"
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x4e3e4e419cc598cd7041f5eac3e067f9c7de586fd4b062faa59515bca6460322 nonce=8 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=180340
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0x4e3e4e419cc598cd7041f5eac3e067f9c7de586fd4b062faa59515bca6460322 nonce=8 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=180340 tx=0x4e3e4e419cc598cd7041f5eac3e067f9c7de586fd4b062faa59515bca6460322
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=warn msg="sequencer did not make expected progress" syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: 0x270033a1aa86fe162003e3e6c0b1b1fde7120264b39143ee3c7fd9b2de525747:7, blocksToLoad: &{1 74}}" existingBlock=0x19309a04d5ff7916910a7d076efcb2d5627dfa2a131bdce826a8ef559ced7344:27 safeL2=0x51fe293ae5277fcbd1c051f4f20024d3b5775a88b29ebbbbe98bd46038bd065e:0
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Loading range of multiple blocks into state" start=1 end=74
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Added L2 block to local state" block=0x4691ed3762731478aad643e07569bf88aaefec5b8eab0b49ecc9c751b7e021f1:1 tx_count=1 time=1733931942
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Added L2 block to local state" block=0x74e56b76c8b659560621dc83a9a861b0874fc553fca0bc19dbc27baf9584d790:2 tx_count=1 time=1733931944
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Added L2 block to local state" block=0x991a85b84507efd845706f7ba80fb4b4eb475de6d89f9c04485efdcc0af8c13e:3 tx_count=1 time=1733931946
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Added L2 block to local state" block=0xf58b4d098a32ae6ec230c7d8c27b076846e9c54ce4ecfea702f65e8268c1722b:4 tx_count=1 time=1733931948
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Added L2 block to local state" block=0x39e48b87dcd0b316394c4341ad6f7f7a9a326c9d33835b5f7cbe4bf0cc3b776f:5 tx_count=1 time=1733931950
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Added L2 block to local state" block=0x6b8718a31a3525c0b5e50f6371d6b53379399af93f792c3caeabd374556525c1:6 tx_count=1 time=1733931952
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Added L2 block to local state" block=0x2cde4c708e4128ee236ab57a9c2bfadea29ecf739dac480131b40e5054b055a6:7 tx_count=1 time=1733931954
[op-batcher-op-kurtosis] t=2024-12-11T15:48:08+0000 lvl=info msg="Added L2 block to local state" block=0xc60742dccf29c2460155c3fc2bf2d91419fa6e3212ca8c4974fb11bcce2e22a0:8 tx_count=1 time=1733931956
...
[op-batcher-op-kurtosis] t=2024-12-11T15:48:09+0000 lvl=info msg="Added L2 block to local state" block=0x95235be971e6c0cc3e050fc0f7888f55082300f7d7645e4a45f16ed236c4f5e1:72 tx_count=1 time=1733932084
[op-batcher-op-kurtosis] t=2024-12-11T15:48:09+0000 lvl=info msg="Added L2 block to local state" block=0xe0c8786c4f2ac19ad463ca9a4f9614f7f0b03ce7949d42e4ebc7cf2788f10458:73 tx_count=125 time=1733932086
[op-batcher-op-kurtosis] t=2024-12-11T15:48:09+0000 lvl=info msg="Added L2 block to local state" block=0xf48023c2b238769e5bd54734e86e298d983f738abcc38342a7bbfe19d68fe862:74 tx_count=1 time=1733932088
[op-batcher-op-kurtosis] t=2024-12-11T15:48:09+0000 lvl=info msg="Created channel" id=1366952367c88c6be2477af019f2ecca l1Head=0x91b60111fda47a1b1c71f68db67492270d2fea2485c5fda1ba8fe95c2090194b:31 blocks_pending=74 l1OriginLastSubmittedChannel=0x270033a1aa86fe162003e3e6c0b1b1fde7120264b39143ee3c7fd9b2de525747:7 batch_type=0 compression_algo=zlib target_num_frames=5 max_frame_size=9999 use_blobs=false
[op-batcher-op-kurtosis] t=2024-12-11T15:48:09+0000 lvl=info msg="Channel closed" id=1366952367c88c6be2477af019f2ecca blocks_pending=47 num_frames=5 input_bytes=59637 output_bytes=40714 oldest_l1_origin=0x270033a1aa86fe162003e3e6c0b1b1fde7120264b39143ee3c7fd9b2de525747:7 l1_origin=0xd3474989a3d3b6745133a51063067c706dc338ab29468b90806ff53603be8fc2:11 oldest_l2=0x4691ed3762731478aad643e07569bf88aaefec5b8eab0b49ecc9c751b7e021f1:1 latest_l2=0x19309a04d5ff7916910a7d076efcb2d5627dfa2a131bdce826a8ef559ced7344:27 full_reason="channel full: compressor is full" compr_ratio=0.6826969834163354
[op-batcher-op-kurtosis] t=2024-12-11T15:48:09+0000 lvl=info msg="Building Calldata transaction candidate" size=10000
[op-batcher-op-kurtosis] t=2024-12-11T15:48:20+0000 lvl=info msg="Transaction confirmed" service=batcher tx=0x4e3e4e419cc598cd7041f5eac3e067f9c7de586fd4b062faa59515bca6460322 block=0xd295251398270515753cc200c8835a3934cbb327df6a5d05f4c680c64c5d90c1:33 effectiveGasPrice=5020247484
[op-batcher-op-kurtosis] t=2024-12-11T15:48:20+0000 lvl=info msg="Handling receipt" id=e092b07d4dfc60b20de2f0242f4db417:3
[op-batcher-op-kurtosis] t=2024-12-11T15:48:20+0000 lvl=info msg="Transaction confirmed" tx_id=e092b07d4dfc60b20de2f0242f4db417:3 tx=0x4e3e4e419cc598cd7041f5eac3e067f9c7de586fd4b062faa59515bca6460322 block=0xd295251398270515753cc200c8835a3934cbb327df6a5d05f4c680c64c5d90c1:33
[op-batcher-op-kurtosis] t=2024-12-11T15:48:20+0000 lvl=warn msg="transaction from unknown channel marked as confirmed" id=e092b07d4dfc60b20de2f0242f4db417:3
[op-batcher-op-kurtosis] t=2024-12-11T15:48:20+0000 lvl=warn msg="Aborting state publishing, max duration exceeded"
[op-batcher-op-kurtosis] t=2024-12-11T15:48:20+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x205392a32d59a23f8560d473fd3500d3448efe746c7bcea93e80adb7e67df00c nonce=9 gasTipCap=5000000000 gasFeeCap=7000000000 gasLimit=180616
[op-batcher-op-kurtosis] t=2024-12-11T15:48:20+0000 lvl=info msg="Loading range of multiple blocks into state" start=75 end=80

batcher-holocene-activation-awareness.log
safe-head-progression-holocene-activation-awareness.log
sequencer-holocene-activation-awareness.log

barnabasbusa pushed a commit to ethpandaops/optimism-package that referenced this issue Dec 13, 2024
…dme updates (#108)

* Log private keys and addresses for Chain Operator roles when running a
local L1. Removed trace logs in bash script fund.sh since the relevant
info is now logged. I note that the `l1_network` binding wasn't set at
all in `main.star` and is now set to `local` for a local L1 network. The
private key/addresses are needed when a transaction needs to be sent to
the contract -- which is a likely step in executing a test locally in
Kurtosis. See
[here](ethereum-optimism/optimism#13192 (comment))
* Log L1 faucet private key and address
* Log L2 faucet private key and address (useful for running tx-fuzz)
* Updates to readme with info on helpful kurtosis commands for
inspecting local state.
* Add sample config for geth/reth with custom batcher config

Driven from the learnings of testing holocene hard fork chain split
between `op-geth` and `op-reth` as noted in:
ethereum-optimism/optimism#13192 (comment)

The revised output from `fund.sh` locally is as follows:

```
Collect keys, and fund addresses
Command returned with exit code '0' and the following output:
--------------------
0x3be78a6b8544134adb22fa909bfb93ee69a9e2b755076d3d859455832adb9218
0x0cf6c854e32a30be7c79b41d7841ab456b06530d31115ba651bb4bbe26545b5d
0x721cd59d1901ff718f299f14c0ecd2bbfa39b9248cbdfaccaca415e008ea0cb9
0xcdac4618e2128f1a3bc503ddc35e722896174681620ac7e41efa28c5a5f53414
0x77c5e2210e89b2d2cfd591f29b47723789cd76c58b6dde603a58c820171d0097
0xaa397e8c9d218413d130c38efe2666d15a73abdd604ae0411e4206c660eb9d62
0xa9ad8b78b3b0ddbd41b75271d6d9bdbfbe7b60c58e937272a7255ad542542d28
0xbb8c5f7490df7e2286aae608b7c2a1876b73164bde3d5bc9abf19590bcc464a5
0x5db0f1227071a0a719a978f6fea60d63c6974bf114eaaa08a1cfb777b2b58e54
0x924a64b99b825dcb1e88eb451ba90627a18c4f974d91fd4518704bd5ac49c197
Wallet private key and addresses
{
  "proposerPrivateKey": "0x9d77c0558ed6f31e7be16c2f45fd1f3ec3f66d20526b7fe3961872d38ac98fe7",
  "proposerAddress": "0xb0994E702b603df7191cd68E6544f99126135e34",
  "batcherPrivateKey": "0xb3d2d558e3491a3709b7c451100a0366b5872520c7aa020c17a0e7fa35b6a8df",
  "batcherAddress": "0xD3F2c5AFb2D76f5579F326b0cD7DA5F5a4126c35",
  "sequencerPrivateKey": "0x459f58fd8ef3d9123333514d0f2445153fc29ddc43be239356dad70f506475d9",
  "sequencerAddress": "0xbb900Cf56918A2639dAA90c3f7DC5DCD2f5B9935",
  "challengerPrivateKey": "0x717c53f6d6c266889465d78a885cd0a2e22d41f73e21fa1f07ba5849c82d79c3",
  "challengerAddress": "0xf08f610d1956CAAAb34f18e9e0A122E389496529",
  "l2ProxyAdminPrivateKey": "0x93be1ab4efe7633d951f19783912817a264f70889cc230e6015627695041744b",
  "l2ProxyAdminAddress": "0x21DD8D15472844b905E603Ddd48A485a630B233b",
  "l1ProxyAdminPrivateKey": "0xa8574059a5e36acd4d6ea03e90fe8b5233cc87222b0b396f9b1d6f3e5d6368fa",
  "l1ProxyAdminAddress": "0x39Dc844B8b461c6de6c572cad9585d9f5A8DEa6A",
  "baseFeeVaultRecipientPrivateKey": "0x227e712ee48868fff4651301e8f366bea05e822db3429470bbc7ed2ee045cb67",
  "baseFeeVaultRecipientAddress": "0x9ef74F0e58fdEf70c11E587A45c0a85A3258DCeA",
  "l1FeeVaultRecipientPrivateKey": "0xe6414f68c9f0d5e3470955a39c1af602a3132f709130fea6304f126a78b7f6aa",
  "l1FeeVaultRecipientAddress": "0xA7B30F9e99A248C6D623eaaD7bce81217Bb84038",
  "sequencerFeeVaultRecipientPrivateKey": "0xc11b0cec865b74505976f6c7fc5736a701f1cdf9e46aa72c02d51c0fe6ec3fed",
  "sequencerFeeVaultRecipientAddress": "0x15947bF05662caD8DF55f004c055939b574f18c0",
  "systemConfigOwnerPrivateKey": "0x28978385446dde6ce493da611d6c9ae6193d960c1cb73ecfb25800e9c9017f4c",
  "systemConfigOwnerAddress": "0x8e44fEbeaE9C2F2Bc89480aE8084328Badc496c2",
  "l1FaucetPrivateKey": "0xeaba42282ad33c8ef2524f07277c03a776d98ae19f581990ce75becb7cfa1c23",
  "l1FaucetAddress": "0x589A698b7b7dA0Bec545177D3963A2741105C7C9",
  "l2FaucetPrivateKey": "0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80",
  "l2FaucetAddress": "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"
}
```

---------

Co-authored-by: Vinod Damle <5338861+vdamle@users.noreply.github.com>
@geoknee geoknee added the A-op-batcher Area: op-batcher label Dec 16, 2024
@emlautarom1
Copy link

emlautarom1 commented Dec 20, 2024

On behalf of @swapnilraj, we updated Nethermind recently to include a fix for Holocene that we didn't catch until we started using Kurtosis with the Optimism Package. With this fix, it is possible to create a Docker image (we provide one in our main repo, https://github.com/NethermindEth/nethermind) and include it in the network-params.yaml file. As of today, we don't have a public image with the fix implemented but we expect to have one before end of year.

In our testing, all three clients (geth, reth and nethermind) behave as expected when enabling Holocene.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-op-batcher Area: op-batcher
Projects
None yet
Development

No branches or pull requests

5 participants