Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flake in e2e prover coordination #9635

Closed
just-mitch opened this issue Oct 31, 2024 · 1 comment · Fixed by #9964
Closed

Flake in e2e prover coordination #9635

just-mitch opened this issue Oct 31, 2024 · 1 comment · Fixed by #9964
Assignees
Labels
P-high 🔥 Priority: high. Do this task next. T-bug Type: Bug. Something is broken. team-alpha
Milestone

Comments

@just-mitch
Copy link
Collaborator

aztec:archiver [VERBOSE] Retrieved no new L1 -> L2 messages between L1 blocks 33 and 34. +1s
  aztec:archiver [VERBOSE] Updating the proven block number to 3 and epoch to 0 +2ms
  aztec:archiver [VERBOSE] No blocks to retrieve from 33 to 34 +0ms
  aztec:archiver [VERBOSE] L2 prune will occur on next submission. Rolling back to last proven block. +2ms
  aztec:archiver [VERBOSE] Unwinding 2 blocks from block 5 +0ms
  aztec:l2_block_stream [VERBOSE] Reorg detected. Pruning blocks from 4 to 5. +24s
  aztec:p2p [INFO] Detected chain prune. Removing invalid txs count=1 newLatestBlock=3 previousLatestBlock=5 +12s
  aztec:p2p [INFO] Moving 1 mined txs back to pending +2ms
  aztec:l2_block_stream [VERBOSE] Reorg detected. Pruning blocks from 4 to 5. +24s
  aztec:world_state [INFO] Chain pruned to block 3 +1s
  aztec:sequencer [VERBOSE] Sequencer sync check failed worldStateNumber=5 worldStateHash=undefined l2BlockSourceNumber=3 l2BlockSourceHash=0x05905c33c919b8dcce9cfc43efa35e99e4ac1dd8648c754914cdfa8b1cd4a1fa p2pNumber=3 l1ToL2MessageSourceNumber=3 +1s
  aztec:sequencer [VERBOSE] Sequencer sync check succeeded worldStateNumber=7 worldStateHash=0x2df727cb9225a411f2c25c7721e4a006b53a89601436d36171b0acb668a5e854 l2BlockSourceNumber=7 l2BlockSourceHash=0x2df727cb9225a411f2c25c7721e4a006b53a89601436d36171b0acb668a5e854 p2pNumber=7 l1ToL2MessageSourceNumber=7 +1s
  aztec:sequencer [VERBOSE] Sequencer sync check succeeded worldStateNumber=3 worldStateHash=0x05905c33c919b8dcce9cfc43efa35e99e4ac1dd8648c754914cdfa8b1cd4a1fa l2BlockSourceNumber=3 l2BlockSourceHash=0x05905c33c919b8dcce9cfc43efa35e99e4ac1dd8648c754914cdfa8b1cd4a1fa p2pNumber=3 l1ToL2MessageSourceNumber=3 +1s
  aztec:sequencer [INFO] Building blockNumber=4 txCount=1 slotNumber=65 +8ms
  aztec:sequencer [VERBOSE] Retrieved 0 L1 to L2 messages for block 4 +1ms
  aztec:sequencer-client:block_builder_light [VERBOSE] Starting new block numTxs=2 globalVariables=[object Object] l1ToL2Messages= +0ms
  aztec:sequencer-client:block_builder_light [VERBOSE] Adding new tx to block txHash=1cabab011f74c24e365d76c32b65541e3ea90cb2241d328ca18a8ddc9c039d50 +5ms
  aztec:sequencer-client:block_builder_light [VERBOSE] Setting block as completed and adding 1 padding txs +39ms
  aztec:sequencer-client:block_builder_light [VERBOSE] Adding new tx to block txHash=0000000000000000000000000000000000000000000000000000000000000000 +0ms
  aztec:sequencer-client:block_builder_light [VERBOSE] Finalising block +36ms
  aztec:sequencer [VERBOSE] Assembled block 4 (txEffectsHash: 00d2c66885855bfbe033c7649efb79e1a9b9f6a9790d5da2008ae61935b760db) eventName=l2-block-built creator=0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266 duration=91.50715900000068 publicProcessDuration=40.93021099999896 rollupCircuitsDuration=88.43819499999518 txCount=1 blockNumber=4 blockTimestamp=1730392893 noteEncryptedLogLength=525 noteEncryptedLogCount=1 encryptedLogLength=8 encryptedLogCount=0 unencryptedLogCount=0 unencryptedLogSize=8 +91ms
  aztec:sequencer [VERBOSE] Collecting attestations +1ms
  aztec:sequencer [VERBOSE] Attestations collected +3ms
  aztec:sequencer [VERBOSE] Collecting proof quotes +0ms
  aztec:sequencer [VERBOSE] Retrieved 0 quotes, slot: 65, epoch to prove: 2 +1ms
  aztec:sequencer [VERBOSE] Failed to find any valid proof quotes +0ms
  aztec:sequencer [VERBOSE] No proof quote available +0ms
  aztec:sequencer:publisher [VERBOSE] Submitting propose transaction +3s
  aztec:sequencer:publisher [INFO] Published L2 block to L1 rollup contract gasPrice=1012568824 gasUsed=320673 transactionHash=0x3d32f3008b217efefc54e4746e4a7f0c68f301b43390525636fee55b3b77e6c1 calldataGas=17964 calldataSize=1764 sender=0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266 txCount=1 blockNumber=4 blockTimestamp=1730392893 noteEncryptedLogLength=525 noteEncryptedLogCount=1 encryptedLogLength=8 encryptedLogCount=0 unencryptedLogCount=0 unencryptedLogSize=8 eventName=rollup-published-to-l1 slotNumber=65 blockHash=0x27dd96175d17ea7506eb627693ae9e498c5ec94daa9e4a7e3417dad4fc02e3d8 +120ms
  aztec:sequencer [INFO] Submitted rollup block 4 with 1 transactions duration=92ms (Submitter: 0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266) +125ms
FAIL src/prover-coordination/e2e_prover_coordination.test.ts
  e2e_prover_coordination
    ✓ Sequencer selects best valid proving quote for each block (31966 ms)
    ✕ Can claim proving rights after a prune (27169 ms)

  ● e2e_prover_coordination › Can claim proving rights after a prune

    expect(received).toEqual(expected) // deep equality

    Expected: "success"
    Received: "pending"

      409 |     // the tx from epoch 2 is still valid since it references a proven block
      410 |     // this will be added back onto the chain
    > 411 |     expect(tx2AfterReorg.status).toEqual(TxStatus.SUCCESS);
          |                                  ^
      412 |     expect(tx2AfterReorg.blockNumber).toEqual(tx2BeforeReorg.blockNumber);
      413 |     expect(tx2AfterReorg.blockHash).not.toEqual(tx2BeforeReorg.blockHash);
      414 |

      at Object.toEqual (prover-coordination/e2e_prover_coordination.test.ts:411:34)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 passed, 2 total
@just-mitch just-mitch added P-high 🔥 Priority: high. Do this task next. T-bug Type: Bug. Something is broken. team-alpha labels Oct 31, 2024
@just-mitch just-mitch added this to the Sequencer & Prover Testnet milestone Oct 31, 2024
@just-mitch
Copy link
Collaborator Author

@alexghr has confirmed this is a known, fixable issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P-high 🔥 Priority: high. Do this task next. T-bug Type: Bug. Something is broken. team-alpha
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants