docs: add postmortem of tx receipts validation skip fork #289
Conversation
WalkthroughThe recent updates involve two primary changes. Firstly, a minor modification to text casing in the incident summary header within the postmortem documentation. Secondly, a detailed incident report addressing a chain halt caused by a validation issue, leading to improvements in validation processes and monitoring systems to prevent similar incidents. Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
Actionable comments outside the diff hunks (2)
postmortems/2024-03-12-zktrie-incident.md (2)
Line range hint
19-22: Ensure the description of the upgrade process is clear and concise.- On Thu, Mar 07, 2024, at 05:00:00 UTC, an upgrade to kroma [`v1.3.2`](https://github.com/kroma-network/kroma/releases/tag/v1.3.2) and kroma-geth `v0.4.4` was conducted to enhance ZKTrie, which is to store the state of accounts and storage. + On March 7, 2024, at 05:00:00 UTC, we upgraded to kroma [`v1.3.2`](https://github.com/kroma-network/kroma/releases/tag/v1.3.2) and kroma-geth `v0.4.4` to enhance ZKTrie, our state storage solution.
Line range hint
82-85: Clarify the testing and deployment process for the bug fix.- The bug will be fixed in this [PR](https://github.com/kroma-network/go-ethereum/pull/81), and once merged, it will undergo sufficient test on the internal devnet and Kroma sepolia. + The bug fix, available in this [PR](https://github.com/kroma-network/go-ethereum/pull/81), will undergo extensive testing on the internal devnet and Kroma Sepolia upon merging.
There was a problem hiding this comment.
Actionable comments posted: 1
Review Status
Configuration used: CodeRabbit UI
Files selected for processing (1)
- postmortems/2024-04-01-tx-receipts-validation-skip-incident.md (1 hunks)
Additional comments not posted (6)
postmortems/2024-04-01-tx-receipts-validation-skip-incident.md (6)
1-1: The title accurately reflects the incident date and content.
3-10: The incident summary is clear and provides a concise overview of the issue.
12-17: The background section effectively sets the context for the incident.
32-35: The recovery process is clearly outlined.
37-46: The timeline section is detailed and informative, providing a clear sequence of events.
52-62: The future works section clearly outlines steps to prevent similar incidents.
| # Causes | ||
|
|
||
| Block head of our L1 RPC began to fall behind the canonical chain, triggering a switch to the fallback L1 RPC once it | ||
| deviated beyond a certain threshold. During the switch, sequencer attempted to retrieve transaction receipts from the L1 | ||
| RPC node that fell behind it had not yet synced, resulting in empty transaction receipt responses. | ||
|
|
||
| Actually, the block in question did contain [deposited transaction](https://kromascan.com/tx/0xf76e4f34645bc3e172909fb03311cd1770f9c574ce46bd1da3644f9cea82c0e3) | ||
| on L1, yet the sequencer proceeded to generate the block without processing this transaction, having received empty | ||
| transaction receipts from the trusted L1 RPC. As a consequence, starting from block 9029744, a different block was | ||
| generated, leading to a generation of wrong block. |
There was a problem hiding this comment.
Consider adding an article for grammatical correctness.
- Block head of our L1 RPC began to fall behind
+ The block head of our L1 RPC began to fall behindCommittable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| # Causes | |
| Block head of our L1 RPC began to fall behind the canonical chain, triggering a switch to the fallback L1 RPC once it | |
| deviated beyond a certain threshold. During the switch, sequencer attempted to retrieve transaction receipts from the L1 | |
| RPC node that fell behind it had not yet synced, resulting in empty transaction receipt responses. | |
| Actually, the block in question did contain [deposited transaction](https://kromascan.com/tx/0xf76e4f34645bc3e172909fb03311cd1770f9c574ce46bd1da3644f9cea82c0e3) | |
| on L1, yet the sequencer proceeded to generate the block without processing this transaction, having received empty | |
| transaction receipts from the trusted L1 RPC. As a consequence, starting from block 9029744, a different block was | |
| generated, leading to a generation of wrong block. | |
| # Causes | |
| The block head of our L1 RPC began to fall behind the canonical chain, triggering a switch to the fallback L1 RPC once it | |
| deviated beyond a certain threshold. During the switch, sequencer attempted to retrieve transaction receipts from the L1 | |
| RPC node that fell behind it had not yet synced, resulting in empty transaction receipt responses. | |
| Actually, the block in question did contain [deposited transaction](https://kromascan.com/tx/0xf76e4f34645bc3e172909fb03311cd1770f9c574ce46bd1da3644f9cea82c0e3) | |
| on L1, yet the sequencer proceeded to generate the block without processing this transaction, having received empty | |
| transaction receipts from the trusted L1 RPC. As a consequence, starting from block 9029744, a different block was | |
| generated, leading to a generation of wrong block. |
Summary by CodeRabbit