OP Holocene (Kurtosis): Incorrect chain config, rejects holocene blocks.

[vdamle]

Description

I'm attempting to run Nethermind as one of the nodes in a 3 node OP chain in kurtosis with holocene enabled. Although the chainspec shows the correct timestamp for "opHoloceneTransitionTimestamp", I see that admin_nodeInfo from nethermind shows that the hardfork is not configured in nethermind and hence it rejects holocene blocks. op-geth and op-reth show the correct configuration in the log and the output of admin_nodeInfo and they have no issues syncing blocks.

Steps to Reproduce

1. Using the below network-config.yaml, start a 3 node OP chain locally in Kurtosis. As specified in the config, Holocene time offset is set to 10 or starting at block 5.

optimism_package:
  chains:
    - participants:
      - el_type: op-geth
        cl_type: op-node
      - el_type: op-reth
        cl_type: op-node
      - el_type: op-nethermind
        cl_type: op-node
      network_params:
        fjord_time_offset: 0
        granite_time_offset: 0
        holocene_time_offset: 10

2. kurtosis run github.com/ethpandaops/optimism-package --args-file network-config.yaml
3. Check the logs for nethermind EL/op-node CL and they will both show that engine_forkchoiceUpdatedV3 was rejected at block 5 due to the following error: "message": "EIP1559Params should be null before Holocene"

Actual behavior

• Nethermind logs (full logs attached)

[op-el-3-op-nethermind-op-node-op-kurtosis] ======================== Nethermind initialization completed ========================
[op-el-3-op-nethermind-op-node-op-kurtosis] This node    : enode://f2d1ef76fdf45e300f3fbf8eee5686cf5dd5ada4d5edcf94614556183c6f4c9f6fd3525750ff1193449b18adc8c43a4f6956d2ed6e10106f71020f2ea25de5a0@172.16.0.24:30303
[op-el-3-op-nethermind-op-node-op-kurtosis] RPC modules  : Admin, Eth, Net, Subscribe, Web3
[op-el-3-op-nethermind-op-node-op-kurtosis] Node address : 0xbb3d6461a3512bc42e34c8dba18dbe9014ac2f3b (do not use as an account)
[op-el-3-op-nethermind-op-node-op-kurtosis] Mem est DB   :    32 MB
[op-el-3-op-nethermind-op-node-op-kurtosis] JSON RPC     : http://0.0.0.0:8545 ; http://0.0.0.0:8546 ; http://0.0.0.0:8551
[op-el-3-op-nethermind-op-node-op-kurtosis] Genesis hash : 0x02d4deb513b0dff8b0d966d04bbefaaacaf6496027a195a2beebcea501f9c65d
[op-el-3-op-nethermind-op-node-op-kurtosis] External IP  : 172.16.0.24
[op-el-3-op-nethermind-op-node-op-kurtosis] Ethereum     : tcp://172.16.0.24:30303
[op-el-3-op-nethermind-op-node-op-kurtosis] Discovery    : udp://172.16.0.24:30303
[op-el-3-op-nethermind-op-node-op-kurtosis] Client id    : Nethermind/v1.30.1+2b75a75a/linux-arm64/dotnet9.0.0
[op-el-3-op-nethermind-op-node-op-kurtosis] Chainspec    : /network-configs/chainspec-2151908.json
[op-el-3-op-nethermind-op-node-op-kurtosis] Chain head   : 0
[op-el-3-op-nethermind-op-node-op-kurtosis] Chain ID     : 2151908
[op-el-3-op-nethermind-op-node-op-kurtosis] =====================================================================================
....
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 | Received ForkChoice: 4 (0x3f4755...8a0b19), Safe: 4 (0x3f4755...8a0b19), Finalized: 0 (0x02d4de...f9c65d)
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 | ForkChoiceUpdate: block 4 (0x3f4755...8a0b19) was processed.
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 | Responded to Id:30, engine_forkchoiceUpdatedV3([{"headBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","safeBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","finalizedBlockHash":"0x02d4deb513b0dff8b0d966d04bbefaaacaf6496027a195a2beebcea501f9c65d"},null])
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 |   Id:30, engine_forkchoiceUpdatedV3([{"headBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","safeBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","finalizedBlockHash":"0x02d4deb513b0dff8b0d966d04bbefaaacaf6496027a195a2beebcea501f9c65d"},null]) handled in 0ms
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 | JSON RPC request Id:31, engine_forkchoiceUpdatedV3([{"headBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","safeBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","finalizedBlockHash":"0x02d4deb513b0dff8b0d966d04bbefaaacaf6496027a195a2beebcea501f9c65d"},{"timestamp":"0x675afe0a","prevRandao":"0xea61e50593684c7fa14cfad825bf3be6c297ebf366d15ec6101e306c0e611761","suggestedFeeRecipient":"0x4200000000000000000000000000000000000011","withdrawals":[],"parentBeaconBlockRoot":"0x4ed012924d331ceeea551e38e7aad89298f78bb532b94a0ce5346f355323559a","transactions":["0x7ef8f8a000c8d38a4cfa3aadadf92dbdf886538a072e37a9f831467d77beddf7f865cf1794deaddeaddeaddeaddeaddeaddeaddeaddead00019442000000000000000000000000000000000000158080830f424080b8a4440a5e2000000558000c5fc5000000000000000200000000675afe060000000000000006000000000000000000000000000000000000000000000000000000002bc5fce00000000000000000000000000000000000000000000000000000000000000001971029ec061f780d7a2673a253d0289a029be0bcdefe7c08ede01ac6c37e8f28000000000000000000000000d3f2c5afb2d76f5579f326b0cd7da5f5a4126c35"],"noTxPool":true,"gasLimit":"0x3938700","eip1559Params":"0x0000000000000000"}])
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 | Executing JSON RPC call engine_forkchoiceUpdatedV3 with params [{"headBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","safeBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","finalizedBlockHash":"0x02d4deb513b0dff8b0d966d04bbefaaacaf6496027a195a2beebcea501f9c65d"}, {"timestamp":"0x675afe0a","prevRandao":"0xea61e50593684c7fa14cfad825bf3be6c297ebf366d15ec6101e306c0e611761","suggestedFeeRecipient":"0x4200000000000000000000000000000000000011","withdrawals":[],"parentBeaconBlockRoot":"0x4ed012924d331ceeea551e38e7aad89298f78bb532b94a0ce5346f355323559a","transactions":["0x7ef8f8a000c8d38a4cfa3aadadf92dbdf886538a072e37a9f831467d77beddf7f865cf1794deaddeaddeaddeaddeaddeaddeaddeaddead00019442000000000000000000000000000000000000158080830f424080b8a4440a5e2000000558000c5fc5000000000000000200000000675afe060000000000000006000000000000000000000000000000000000000000000000000000002bc5fce00000000000000000000000000000000000000000000000000000000000000001971029ec061f780d7a2673a253d0289a029be0bcdefe7c08ede01ac6c37e8f28000000000000000000000000d3f2c5afb2d76f5579f326b0cd7da5f5a4126c35"],"noTxPool":true,"gasLimit":"0x3938700","eip1559Params":"0x0000000000000000"}]
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 | Received ForkChoice: 4 (0x3f4755...8a0b19), Safe: 4 (0x3f4755...8a0b19), Finalized: 0 (0x02d4de...f9c65d)
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 | ForkChoiceUpdate: block 4 (0x3f4755...8a0b19) was processed.
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 | Sending error response, method: engine_forkchoiceUpdatedV3, id: 31, errorType: -38003, message: EIP1559Params should be null before Holocene, errorData:
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 | Error when handling Id:31, engine_forkchoiceUpdatedV3([{"headBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","safeBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","finalizedBlockHash":"0x02d4deb513b0dff8b0d966d04bbefaaacaf6496027a195a2beebcea501f9c65d"},{"timestamp":"0x675afe0a","prevRandao":"0xea61e50593684c7fa14cfad825bf3be6c297ebf366d15ec6101e306c0e611761","suggestedFeeRecipient":"0x4200000000000000000000000000000000000011","withdrawals":[],"parentBeaconBlockRoot":"0x4ed012924d331ceeea551e38e7aad89298f78bb532b94a0ce5346f355323559a","transactions":["0x7ef8f8a000c8d38a4cfa3aadadf92dbdf886538a072e37a9f831467d77beddf7f865cf1794deaddeaddeaddeaddeaddeaddeaddeaddead00019442000000000000000000000000000000000000158080830f424080b8a4440a5e2000000558000c5fc5000000000000000200000000675afe060000000000000006000000000000000000000000000000000000000000000000000000002bc5fce00000000000000000000000000000000000000000000000000000000000000001971029ec061f780d7a2673a253d0289a029be0bcdefe7c08ede01ac6c37e8f28000000000000000000000000d3f2c5afb2d76f5579f326b0cd7da5f5a4126c35"],"noTxPool":true,"gasLimit":"0x3938700","eip1559Params":"0x0000000000000000"}]) | {
[op-el-3-op-nethermind-op-node-op-kurtosis]   "jsonrpc": "2.0",
[op-el-3-op-nethermind-op-node-op-kurtosis]   "error": {
[op-el-3-op-nethermind-op-node-op-kurtosis]     "code": -38003,
[op-el-3-op-nethermind-op-node-op-kurtosis]     "message": "EIP1559Params should be null before Holocene"
[op-el-3-op-nethermind-op-node-op-kurtosis]   },
[op-el-3-op-nethermind-op-node-op-kurtosis]   "id": 31
[op-el-3-op-nethermind-op-node-op-kurtosis] }
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:19:54 |   Id:31, engine_forkchoiceUpdatedV3([{"headBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","safeBlockHash":"0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19","finalizedBlockHash":"0x02d4deb513b0dff8b0d966d04bbefaaacaf6496027a195a2beebcea501f9c65d"},{"timestamp":"0x675afe0a","prevRandao":"0xea61e50593684c7fa14cfad825bf3be6c297ebf366d15ec6101e306c0e611761","suggestedFeeRecipient":"0x4200000000000000000000000000000000000011","withdrawals":[],"parentBeaconBlockRoot":"0x4ed012924d331ceeea551e38e7aad89298f78bb532b94a0ce5346f355323559a","transactions":["0x7ef8f8a000c8d38a4cfa3aadadf92dbdf886538a072e37a9f831467d77beddf7f865cf1794deaddeaddeaddeaddeaddeaddeaddeaddead00019442000000000000000000000000000000000000158080830f424080b8a4440a5e2000000558000c5fc5000000000000000200000000675afe060000000000000006000000000000000000000000000000000000000000000000000000002bc5fce00000000000000000000000000000000000000000000000000000000000000001971029ec061f780d7a2673a253d0289a029be0bcdefe7c08ede01ac6c37e8f28000000000000000000000000d3f2c5afb2d76f5579f326b0cd7da5f5a4126c35"],"noTxPool":true,"gasLimit":"0x3938700","eip1559Params":"0x0000000000000000"}]) handled in 0ms
[op-el-3-op-nethermind-op-node-op-kurtosis] 12 Dec 15:20:04 | Dropped 0 useless peers

• Nethermind output of admin_nodeInfo

 cast rpc admin_nodeInfo --rpc-url localhost:32828 | jq
{
  "enode": "enode://f2d1ef76fdf45e300f3fbf8eee5686cf5dd5ada4d5edcf94614556183c6f4c9f6fd3525750ff1193449b18adc8c43a4f6956d2ed6e10106f71020f2ea25de5a0@172.16.0.24:30303",
  "id": "7790e6717a2b2da93f1d9f23bb3d6461a3512bc42e34c8dba18dbe9014ac2f3b",
  "ip": "172.16.0.24",
  "listenAddr": "172.16.0.24:30303",
  "name": "Nethermind/v1.30.1+2b75a75a/linux-arm64/dotnet9.0.0",
  "ports": {
    "discovery": 30303,
    "listener": 30303
  },
  "protocols": {
    "eth": {
      "difficulty": "0x0",
      "genesis": "0x02d4deb513b0dff8b0d966d04bbefaaacaf6496027a195a2beebcea501f9c65d",
      "head": "0x3f475547fe334d277c16dec927f1b9054419fe81dc034cfcc573d5df538a0b19",
      "network": "0x20d5e4",
      "chainId": "0x20d5e4",
      "config": {
        "maxCodeSize": "0x6000",
        "maxCodeSizeTransition": "0x0",
        "gasLimitBoundDivisor": "0x400",
        "maximumExtraDataSize": "0x20",
        "minGasLimit": "0x1388",
        "eip150Transition": "0x0",
        "eip152Transition": "0x0",
        "eip160Transition": "0x0",
        "eip161abcTransition": "0x0",
        "eip161dTransition": "0x0",
        "eip155Transition": "0x0",
        "eip140Transition": "0x0",
        "eip211Transition": "0x0",
        "eip214Transition": "0x0",
        "eip658Transition": "0x0",
        "eip145Transition": "0x0",
        "eip1014Transition": "0x0",
        "eip1052Transition": "0x0",
        "eip1108Transition": "0x0",
        "eip1283Transition": "0x0",
        "eip1283DisableTransition": "0x0",
        "eip1344Transition": "0x0",
        "eip1884Transition": "0x0",
        "eip2028Transition": "0x0",
        "eip2200Transition": "0x0",
        "eip1559Transition": "0x0",
        "eip2565Transition": "0x0",
        "eip2929Transition": "0x0",
        "eip2930Transition": "0x0",
        "eip3198Transition": "0x0",
        "eip3529Transition": "0x0",
        "eip3541Transition": "0x0",
        "eip3607Transition": "0x0",
        "eip1559BaseFeeInitialValue": "0x3b9aca00",
        "eip1559BaseFeeMaxChangeDenominator": "0x32",
        "eip1559ElasticityMultiplier": "0x6",
        "eip1559FeeCollectorTransition": "0x0",
        "feeCollector": "0x4200000000000000000000000000000000000019",
        "terminalTotalDifficulty": "0x0",
        "eip3651TransitionTimestamp": "0x0",
        "eip3855TransitionTimestamp": "0x0",
        "eip3860TransitionTimestamp": "0x0",
        "eip4895TransitionTimestamp": "0x0",
        "eip4844TransitionTimestamp": "0x0",
        "eip1153TransitionTimestamp": "0x0",
        "eip5656TransitionTimestamp": "0x0",
        "eip6780TransitionTimestamp": "0x0",
        "eip4788TransitionTimestamp": "0x0",
        "eip4788ContractAddress": "0x000f3df6d732807ef1319fb7b8bb8522d0beac02",
        "depositContractAddress": "0x00000000219ab540356cbb839cbe05303d7705fa",
        "eip7002ContractAddress": "0x09fc772d0857550724b07b850a4323f39112aaaa",
        "eip7251ContractAddress": "0x01abea29659e5e97c95107f20bb753cd3e09bbbb",
        "eip2935ContractAddress": "0x0aae40965e6800cd9b1f4b05ff21581047e3f91e",
        "rip7212TransitionTimestamp": "0x0",
        "opGraniteTransitionTimestamp": "0x0"
      }
    }
  }
}

• Chainspec file shows holocene correctly specified (complete file attached):

    "opHoloceneTransitionTimestamp": "0x675afe0a",

Expected behavior
Nethermind should show chain config that includes Holocene hard fork, similar to op-reth and op-geth and should not have any issues syncing holocene blocks. See admin_nodeInfo output from op-reth below:

cast rpc admin_nodeInfo --rpc-url localhost:32821 | jq
{
  "id": "02725592c56f5dd5af7a24ab8ee98ad781682170ef7111172fae02d252ec191118",
  "name": "reth/v1.1.4-15fac087/aarch64-unknown-linux-gnu",
  "enode": "enode://725592c56f5dd5af7a24ab8ee98ad781682170ef7111172fae02d252ec191118e4b6f8573c5e394f5f4be8b49570567c23075e4e1c9d7f86043d1fb9dc0c70da@107.15.210.35:30303?discport=9200",
  "enr": "enr:-Iu4QHdfXAMN1XUDqUp4iO6BRZIFi7L5hIKqpwbIaUGaNR1FZbKsI3kteB7fOzUrEb0e6cwyTq14cYFgM9aKM5zTZg4BgmlkgnY0gmlwhGsP0iOJc2VjcDI1NmsxoQJyVZLFb13Vr3okq47piteBaCFw73ERFy-uAtJS7BkRGIN0Y3CCdl-DdWRwgiPw",
  "ip": "107.15.210.35",
  "ports": {
    "discovery": 9200,
    "listener": 30303
  },
  "listenAddr": "107.15.210.35:30303",
  "protocols": {
    "eth": {
      "network": 2151908,
      "difficulty": "0x0",
      "genesis": "0x02d4deb513b0dff8b0d966d04bbefaaacaf6496027a195a2beebcea501f9c65d",
      "config": {
        "chainId": 2151908,
        "homesteadBlock": 0,
        "daoForkSupport": false,
        "eip150Block": 0,
        "eip155Block": 0,
        "eip158Block": 0,
        "byzantiumBlock": 0,
        "constantinopleBlock": 0,
        "petersburgBlock": 0,
        "istanbulBlock": 0,
        "muirGlacierBlock": 0,
        "berlinBlock": 0,
        "londonBlock": 0,
        "arrowGlacierBlock": 0,
        "grayGlacierBlock": 0,
        "mergeNetsplitBlock": 0,
        "shanghaiTime": 0,
        "cancunTime": 0,
        "terminalTotalDifficulty": "0x0",
        "terminalTotalDifficultyPassed": true,
        "bedrockBlock": 0,
        "canyonTime": 0,
        "ecotoneTime": 0,
        "fjordTime": 0,
        "graniteTime": 0,
        "holoceneTime": 1734016522,
        "optimism": {
          "eip1559Denominator": 50,
          "eip1559DenominatorCanyon": 250,
          "eip1559Elasticity": 6
        },
        "regolithTime": 0
      },
      "head": "0xf5a17ee3d27f8cf06336a13a6772fd6a1319a461333d38da6d94ce868019389d"
    }
  }
}

Environment
Kurtosis with docker, running on MacOS.

Logs
nethermind-holocene.log
chainspec-2151908.json

[emlautarom1]

Hi, thanks for reporting this issue. We've been working on getting the Optimism Kurtosis package working to test Holocene properly and your instructions have been extremely valuable (ethereum-optimism/optimism#13192).

In #7926 we fix several Holocene issues and running Kurtosis with OP-Geth, OP-Reth and an image of that branch shows no issues, with or without transactions, and with or without calls to setEIP1559Params().

Once we merge the PR we'll reach out to include Nethermind as part of your testing pipeline (we'll most likely set up or own using Kurtosis, but maybe you also want to run your own tests).