Skip to content

Commit

Permalink
chore: replace fork
Browse files Browse the repository at this point in the history
  • Loading branch information
0xHansLee committed Apr 5, 2024
1 parent e49ed98 commit 38827c5
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 10 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Incident summary

On March 12, 2024, at 15:16:25 UTC, a fork occurred at block number [8171899](https://kromascan.com/block/8171899).
On March 12, 2024, at 15:16:25 UTC, a wrong block was generated at block number [8171899](https://kromascan.com/block/8171899).

Some nodes encountered a `BAD BLOCK` error due to a hash discrepancy in that block compared to the block hash of the
sequencer. This discrepancy was caused by a bug introduced in `KromaZKTrie` in kroma-geth
Expand Down Expand Up @@ -33,7 +33,7 @@ mistakenly set as the root node, altering the state tree and resulting in a diff

This issue is first discovered when executing a specific transaction. As a result of execution of the
[transaction](https://kromascan.com/tx/0x50580775422fee57c8ec78dce4a3598e2246c12d7a73756f874dd117bed0ad72), the
structure of the tree changed, leading to the calculation of different state roots and thus causing a fork with
structure of the tree changed, leading to the calculation of different state roots and thus causing wrong block with
different block hashes.

# Recovery
Expand All @@ -44,10 +44,10 @@ continue generating correct blocks.

## Timeline (UTC)

- 2024-03-12 0616: fork occurred at 8171899
- 2024-03-12 0616: wrong block generated at 8171899
- 2024-03-12 0619: received alerts from some RPC nodes about discrepancies of latest block
- 2024-03-12 0620: started investigating the fork
- 2024-03-12 0649: announced fork occurrence on discord
- 2024-03-12 0620: started investigating the incident
- 2024-03-12 0649: announced incident occurrence on discord
- 2024-03-12 0814: requested canonical chain snapshot data from Wemade
- 2024-03-12 0907: announced rollback of Kroma mainnet on discord
- 2024-03-12 0942: set up a new sequencer using the snapshot data provided by Wemade
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Incident Summary

On April 1, 2024, at 02:51:15 UTC, a fork occurred at block number [9029744](https://kromascan.com/block/9029744).
On April 1, 2024, at 02:51:15 UTC, a wrong block was generated at block number [9029744](https://kromascan.com/block/9029744).

Some nodes ignored the batch due to a hash discrepancy in that block compared to the block hash of the sequencer. This
discrepancy was caused due to that the sequencer omitted a deposited transaction from L1.
Expand All @@ -25,7 +25,8 @@ RPC node that fell behind it had not yet synced, resulting in empty transaction
Actually, the block in question did contain [deposited transaction](https://kromascan.com/tx/0xf76e4f34645bc3e172909fb03311cd1770f9c574ce46bd1da3644f9cea82c0e3)
on L1, yet the sequencer proceeded to generate the block without processing this transaction, having received empty
transaction receipts from the trusted L1 RPC. As a consequence, starting from block 9029744, a different block was
generated, leading to a fork. This is the transaction not included in the block sequencer generated.
generated, leading to a generation of wrong block. This is the transaction not included in the block sequencer
generated.

# Recovery

Expand All @@ -34,10 +35,10 @@ transactions from block 9029744 to generate the block correctly.

## Timeline (UTC)

- 2024-04-01 0251: fork occurred at 9029744
- 2024-04-01 0251: wrong block generated at 9029744
- 2024-04-01 0258: notified that block generation was halted by Wemade
- 2024-04-01 0357: stop sequencer and RPC nodes
- 2024-04-01 0358: announced fork occurrence on discord, twitter, and other partners
- 2024-04-01 0358: announced incident occurrence on discord, twitter, and other partners
- 2024-04-01 0453: restarted a sequencer with backup snapshot
- 2024-04-01 0611: recovery of Kroma mainnet (RPC, P2P, validator) completed
- 2024-04-01 0618: announced the recovery of Kroma mainnet on partners
Expand All @@ -57,4 +58,4 @@ trusted source. It will be turned off to allow all Kroma nodes to validate trans
## Enhancement of monitoring system

We are going to strengthen the monitoring system by establishing a system to monitor blocks from trust partners,
enabling swift detection of forks and improving our ability to respond to incidents.
enabling swift detection of incidents and improving our ability to respond to incidents.

0 comments on commit 38827c5

Please sign in to comment.