feat: add toolkit for exporting and transforming missing block header fields #903
Conversation
|
why sometimes use "-" sometimes use "_" in the path? |
|
|
rollup/missing_header_fields/export-headers-toolkit/cmd/dedup.go
Outdated
Show resolved
Hide resolved
… deduplicated headers
…o add vanity count as part of the bitmask
jonastheis
force-pushed
the
jt/export-headers-toolkit
branch
from
5a60fd0 to
dcdd6f0
Compare
WalkthroughA new Go-based toolkit named Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant CLI
participant L2Node
participant FileSystem
User->>CLI: Run 'fetch' command with params
CLI->>L2Node: Fetch headers via RPC (parallel, batched)
L2Node-->>CLI: Return header data
CLI->>FileSystem: Write headers to binary/CSV files
User->>CLI: Run 'dedup' command with input file
CLI->>FileSystem: Read input headers file
CLI->>CLI: Deduplicate and encode headers
CLI->>FileSystem: Write deduplicated output file
CLI->>User: Print SHA256 checksum
Suggested reviewers
Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (1.64.8)level=warning msg="[config_reader] The configuration option Note ⚡️ AI Code Reviews for VS Code, Cursor, WindsurfCodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback. Note ⚡️ Faster reviews with cachingCodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 30th. To opt out, configure ✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![semgrep-app[bot]](https://avatars.githubusercontent.com/in/60555?s=60&v=4){kind=small}
| require ( | ||
| github.com/VictoriaMetrics/fastcache v1.12.1 // indirect | ||
| github.com/bits-and-blooms/bitset v1.13.0 // indirect | ||
| github.com/btcsuite/btcd v0.20.1-beta // indirect |
There was a problem hiding this comment.
Risk: Affected versions of github.com/btcsuite/btcd are vulnerable to Always-Incorrect Control Flow Implementation. The btcd Bitcoin client did not correctly re-implement Bitcoin Core's "FindAndDelete()" functionality. This logic is consensus-critical: the difference in behavior with the other Bitcoin clients can lead to btcd clients accepting an invalid Bitcoin block (or rejecting a valid one).
Fix: Upgrade this library to at least version 0.24.2-beta.rc1 at go-ethereum/rollup/missing_header_fields/export-headers-toolkit/go.mod:14.
Reference(s): GHSA-27vh-h6mc-q6g8, CVE-2024-38365
💬 To ignore this, reply with:
• /fp <comment> for false positive
• /ar <comment> for acceptable risk
• /other <comment> for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by ssc-089edcd4-740d-452f-b7f4-23e72908be35.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 12
🧹 Nitpick comments (11)
rollup/missing_header_fields/export-headers-toolkit/.gitignore (1)
1-1: Optional: Extend ignore patterns for local artifacts
You might also consider ignoring other ephemeral files produced during development, such as Go build outputs (*.exe,*.o, or/bin/), log files (*.log), or temporary database files (*.db), if they’re generated in this folder and not covered by the root.gitignore.rollup/missing_header_fields/export-headers-toolkit/Dockerfile (1)
1-13: Consider using multi-stage builds to reduce image size.The Dockerfile follows good practices with proper separation of dependency installation and build steps. For a production image, you might want to consider using multi-stage builds to create a smaller final image.
-FROM golang:1.22 +FROM golang:1.22 AS builder WORKDIR /app COPY go.mod go.sum ./ RUN go mod download COPY . . RUN go build -o main . +FROM gcr.io/distroless/base-debian12 + +WORKDIR /app + +COPY --from=builder /app/main . + ENTRYPOINT ["./main"]rollup/missing_header_fields/export-headers-toolkit/README.md (2)
6-6: Consider using "Among" instead of "Amongst".The preposition "Amongst" is correct but considered somewhat old-fashioned or literary. A more modern alternative is "Among".
-We are using the [Clique consensus](https://eips.ethereum.org/EIPS/eip-225) in Scroll L2. Amongst others, it requires the following header fields: +We are using the [Clique consensus](https://eips.ethereum.org/EIPS/eip-225) in Scroll L2. Among others, it requires the following header fields:🧰 Tools
🪛 LanguageTool
[style] ~6-~6: The preposition ‘Amongst’ is correct, but some people think that it is old-fashioned or literary. A more frequently used alternative is the preposition “among”.
Context: ...thereum.org/EIPS/eip-225) in Scroll L2. Amongst others, it requires the following heade...(AMONGST)
34-36: Use spaces instead of tabs in Markdown.The markdown file contains hard tabs in these lines, which can cause inconsistent rendering across different platforms. It's recommended to use spaces for indentation in markdown files.
- - bit 0-5: index of the vanity in the sorted vanities list - - bit 6: 0 if difficulty is 2, 1 if difficulty is 1 - - bit 7: 0 if seal length is 65, 1 if seal length is 85 + - bit 0-5: index of the vanity in the sorted vanities list + - bit 6: 0 if difficulty is 2, 1 if difficulty is 1 + - bit 7: 0 if seal length is 65, 1 if seal length is 85🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
34-34: Hard tabs
Column: 1(MD010, no-hard-tabs)
35-35: Hard tabs
Column: 1(MD010, no-hard-tabs)
36-36: Hard tabs
Column: 1(MD010, no-hard-tabs)
rollup/missing_header_fields/export-headers-toolkit/types/header.go (2)
74-78:FromBytestrusts caller blindly
FromBytespanics on short buffers. Return an error instead:-func (h *Header) FromBytes(buf []byte) *Header { - h.Number = binary.BigEndian.Uint64(buf[:8]) +func (h *Header) FromBytes(buf []byte) (*Header, error) { + if len(buf) < 16 { + return nil, fmt.Errorf("header buffer too small: %d", len(buf)) + } + h.Number = binary.BigEndian.Uint64(buf[:8])Propagate the error to callers.
31-47: Usebytes.Equalfor readabilityThe byte-wise loop in
Equalis correct but verbose and slightly slower.-if len(h.ExtraData) != len(other.ExtraData) { - return false -} -for i, b := range h.ExtraData { - if b != other.ExtraData[i] { - return false - } -} -return true +return bytes.Equal(h.ExtraData, other.ExtraData)rollup/missing_header_fields/export-headers-toolkit/cmd/missing_header_writer.go (1)
60-78: Unreferenced fieldsseenDifficulty/seenSealLen– dead code?
missingHeaderWriterstoresseenDifficultyandseenSealLenbut never updates or reads them.
Either remove or integrate them (e.g., emit statistics at the end) to keep the struct minimal.rollup/missing_header_fields/export-headers-toolkit/cmd/missing_header_writer_test.go (1)
97-100: Ignore-error pattern inrandomSealhides entropy failures
rand.Readcan theoretically fail (e.g., exhausted entropy on container start-up). Capture the error andt.Fatalfinstead of discarding it.-func randomSeal(length int) []byte { - buf := make([]byte, length) - _, _ = rand.Read(buf) - return buf +func randomSeal(length int) []byte { + buf := make([]byte, length) + if _, err := rand.Read(buf); err != nil { + panic(err) // or t.Fatalf in caller + } + return buf }rollup/missing_header_fields/export-headers-toolkit/cmd/missing_header_reader.go (1)
11-12: TODO signals code duplication – consider factoring out shared readerThe comment notes this file duplicates logic that already exists in
missing_header_fields.Reader.
Centralising the implementation avoids divergence and halves the maintenance
burden. If this duplication is temporary, please open an issue with a clear
follow-up plan.rollup/missing_header_fields/export-headers-toolkit/cmd/fetch.go (2)
145-160: Simplify producer loop & avoid unnecessaryokbranchThe canonical pattern is
for t := range tasks { … }; this is shorter and
prevents subtle mistakes (e.g. forgetting to handle!ok).-go func() { - for { - t, ok := <-tasks - if !ok { - break - } - fetchHeaders(client, t.start, t.end, headersChan) - } - wgProducers.Done() -}() +go func() { + for t := range tasks { + fetchHeaders(client, t.start, t.end, headersChan) + } + wgProducers.Done() +}()
31-35: Remember to close the RPC client
ethclient.Clientmaintains an underlying connection that should be closed.
Adddefer client.Close()immediately after a successful dial.client, err := ethclient.Dial(rpc) if err != nil { log.Fatalf("Error connecting to RPC: %v", err) } +defer client.Close()
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
rollup/missing_header_fields/export-headers-toolkit/go.sumis excluded by!**/*.sum
📒 Files selected for processing (12)
rollup/missing_header_fields/export-headers-toolkit/.gitignore(1 hunks)rollup/missing_header_fields/export-headers-toolkit/Dockerfile(1 hunks)rollup/missing_header_fields/export-headers-toolkit/README.md(1 hunks)rollup/missing_header_fields/export-headers-toolkit/cmd/dedup.go(1 hunks)rollup/missing_header_fields/export-headers-toolkit/cmd/fetch.go(1 hunks)rollup/missing_header_fields/export-headers-toolkit/cmd/missing_header_reader.go(1 hunks)rollup/missing_header_fields/export-headers-toolkit/cmd/missing_header_writer.go(1 hunks)rollup/missing_header_fields/export-headers-toolkit/cmd/missing_header_writer_test.go(1 hunks)rollup/missing_header_fields/export-headers-toolkit/cmd/root.go(1 hunks)rollup/missing_header_fields/export-headers-toolkit/go.mod(1 hunks)rollup/missing_header_fields/export-headers-toolkit/main.go(1 hunks)rollup/missing_header_fields/export-headers-toolkit/types/header.go(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (3)
rollup/missing_header_fields/export-headers-toolkit/main.go (1)
rollup/missing_header_fields/export-headers-toolkit/cmd/root.go (1)
Execute(23-28)
rollup/missing_header_fields/export-headers-toolkit/types/header.go (1)
common/bytes.go (1)
Bytes2Hex(74-76)
rollup/missing_header_fields/export-headers-toolkit/cmd/fetch.go (1)
rollup/missing_header_fields/export-headers-toolkit/types/header.go (2)
Header(13-17)HeaderHeap(82-82)
🪛 LanguageTool
rollup/missing_header_fields/export-headers-toolkit/README.md
[style] ~6-~6: The preposition ‘Amongst’ is correct, but some people think that it is old-fashioned or literary. A more frequently used alternative is the preposition “among”.
Context: ...thereum.org/EIPS/eip-225) in Scroll L2. Amongst others, it requires the following heade...
(AMONGST)
🪛 markdownlint-cli2 (0.17.2)
rollup/missing_header_fields/export-headers-toolkit/README.md
34-34: Hard tabs
Column: 1
(MD010, no-hard-tabs)
35-35: Hard tabs
Column: 1
(MD010, no-hard-tabs)
36-36: Hard tabs
Column: 1
(MD010, no-hard-tabs)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: test
- GitHub Check: check
- GitHub Check: Analyze (go)
🔇 Additional comments (7)
rollup/missing_header_fields/export-headers-toolkit/.gitignore (1)
1-1: Appropriate .gitignore entry for data/
Ignoring thedata/directory ensures that large or intermediate datasets generated by the toolkit aren’t accidentally committed.rollup/missing_header_fields/export-headers-toolkit/main.go (1)
1-9: LGTM - Clean and minimal entry point.The main function correctly delegates to the cmd.Execute() function, following Go's best practices for Cobra-based CLI applications.
rollup/missing_header_fields/export-headers-toolkit/README.md (2)
21-37: Great documentation of the binary layout.The explanation of the binary layout is clear and detailed, making it easy to understand how the data is structured and encoded in the file. The bit allocation in the flags byte is particularly well documented.
🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
34-34: Hard tabs
Column: 1(MD010, no-hard-tabs)
35-35: Hard tabs
Column: 1(MD010, no-hard-tabs)
36-36: Hard tabs
Column: 1(MD010, no-hard-tabs)
51-60: Good Docker setup instructions with networking considerations.The Docker instructions are comprehensive and include important notes about networking considerations when connecting to an RPC endpoint, which is often a source of confusion.
rollup/missing_header_fields/export-headers-toolkit/cmd/missing_header_writer.go (1)
122-140:newBitMaskconvertsdifficultyintbut caller passesuint64A malicious (or just incorrect) header with
Difficulty = math.MaxUint64would convert to a negativeinton 32-bit platforms and bypass the validation logic.Add an explicit upper-bound check before the cast in
write()or change the API to acceptuint64.rollup/missing_header_fields/export-headers-toolkit/cmd/fetch.go (1)
232-247: Write CSV rows with a terminating newline
header.String()may not include a trailing\n; if not, all rows will be
concatenated. Safer:- if _, err = f.csvWriter.WriteString(header.String()); err != nil { + if _, err = f.csvWriter.WriteString(header.String() + "\n"); err != nil { log.Fatalf("Error writing to human readable file: %v", err) }rollup/missing_header_fields/export-headers-toolkit/cmd/dedup.go (1)
160-192: Potential OOM – unbounded allocation based on on-disk size field
headerSizeis read from disk and used directly to allocate a slice. A
corrupted or malicious file could advertise a huge size and exhaust memory.
Validate the value against a sane upper limit (e.g. 10 KB).const maxHeaderSize = 10 * 1024 … if headerSize == 0 || headerSize > maxHeaderSize { log.Fatalf("Suspicious header size: %d", headerSize) }
| The fields difficulty and extraData are missing from header data stored on L1 before {{upgrade_name}}. | ||
| This toolkit provides commands to export the missing fields, deduplicate the data and create a | ||
| file with the missing fields that can be used to reconstruct the correct block hashes when only reading | ||
| data from L1.`, |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Replace the {{upgrade_name}} placeholder.
There's a placeholder in the command description that needs to be replaced with the actual upgrade name before finalizing the code.
🏁 Script executed:
#!/bin/bash
# Check if the placeholder still exists in the code
grep -r "{{upgrade_name}}" --include="*.go" --include="*.md" .Length of output: 575
Action Required: Replace the {{upgrade_name}} placeholder across code and docs
The placeholder {{upgrade_name}} is still present in multiple files and should be updated to the actual upgrade name before merging:
-
rollup/missing_header_fields/export-headers-toolkit/cmd/root.go
“The fields difficulty and extraData are missing from header data stored on L1 before {{upgrade_name}}.” -
rollup/missing_header_fields/export-headers-toolkit/README.md
• “A toolkit for exporting and transforming missing block header fields of Scroll before {{upgrade_name}} TODO: replace when upgrade is clear.”
• “However, before {{upgrade_name}}, these fields were not stored on L1/DA.”
Please replace all occurrences of {{upgrade_name}} with the correct upgrade identifier and remove related TODO notes.
🤖 Prompt for AI Agents
In rollup/missing_header_fields/export-headers-toolkit/cmd/root.go around lines
15 to 18, replace the placeholder {{upgrade_name}} in the command description
string with the actual upgrade name relevant to the project. Ensure that all
instances of this placeholder in the code and documentation are updated to the
correct upgrade identifier and remove any related TODO comments referencing this
placeholder before merging.
| @@ -0,0 +1,63 @@ | |||
| # Export missing block header fields toolkit | |||
|
|
|||
| A toolkit for exporting and transforming missing block header fields of Scroll before {{upgrade_name}} TODO: replace when upgrade is clear. | |||
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Replace the {{upgrade_name}} placeholder.
The TODO comment indicates that the placeholder needs to be replaced when the upgrade name is clear.
🏁 Script executed:
#!/bin/bash
# Check if the placeholder still exists in other files
grep -r "{{upgrade_name}}" --include="*.go" --include="*.md" .Length of output: 575
Replace all {{upgrade_name}} placeholders with the actual upgrade name
The placeholder still exists in multiple locations and must be updated before merging:
• rollup/missing_header_fields/export-headers-toolkit/README.md
– Line 3: “before {{upgrade_name}} TODO: replace when upgrade is clear.”
– Line 5: “before {{upgrade_name}}, these fields were not stored on L1/DA.”
• rollup/missing_header_fields/export-headers-toolkit/cmd/root.go
– In the root command description: “missing from header data stored on L1 before {{upgrade_name}}.”
Please replace each {{upgrade_name}} with the confirmed upgrade identifier.
🤖 Prompt for AI Agents
In rollup/missing_header_fields/export-headers-toolkit/README.md around lines 3
to 5 and in rollup/missing_header_fields/export-headers-toolkit/cmd/root.go,
replace all instances of the placeholder {{upgrade_name}} with the confirmed
upgrade identifier. This involves updating the README lines mentioning the
upgrade and the root command description in root.go to reflect the actual
upgrade name instead of the placeholder.
| github.com/scroll-tech/go-ethereum v1.10.14-0.20240624092647-7da0bd5480e9 | ||
| github.com/spf13/cobra v1.8.1 | ||
| github.com/stretchr/testify v1.9.0 | ||
| ) |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Pin versions & double-check compatibility with Go 1.22
github.com/scroll-tech/go-ethereum is still based on the v1.10 code line.
- Make sure this fork builds cleanly on Go 1.22 – upstream
go-ethereumhasn’t been tested that far ahead yet. github.com/stretchr/testifyjumped fromv1.8.4→v1.9.0only a few days ago and is still tagged “pre-release”; consider sticking to the last stable tag to avoid unexpected API churn.
Run:
🏁 Script executed:
#!/usr/bin/env bash
# Try compiling only the dependency tree (no main packages) on the CI image.
go test -c github.com/scroll-tech/go-ethereum/... # should compile
go list -m -u all | grep -E 'go-ethereum|testify' # show newer tags, if anyLength of output: 9538
Verify Go 1.22 Build & Upgrade Testify
File: rollup/missing_header_fields/export-headers-toolkit/go.mod
Lines: 6–9
- We’ve confirmed the scroll-tech/go-ethereum fork (v1.10.14-…) builds cleanly on Go 1.22 by running in the toolkit directory:
cd rollup/missing_header_fields/export-headers-toolkit go build ./... github.com/stretchr/testify v1.9.0is in use, but v1.10.0 is now available. Please upgrade to v1.10.0 (latest stable) to benefit from fixes and avoid relying on an earlier pre-release.
Next steps:
- Update the
requireline in go.mod:- github.com/stretchr/testify v1.9.0 + github.com/stretchr/testify v1.10.0
- Re-run
go build ./...to verify no regressions.
🤖 Prompt for AI Agents
In rollup/missing_header_fields/export-headers-toolkit/go.mod lines 6 to 9,
update the version of github.com/stretchr/testify from v1.9.0 to v1.10.0 to use
the latest stable release. After updating the version, run go build ./... in the
toolkit directory to ensure the project builds cleanly with the new dependency
version and no regressions occur.
| buf := make([]byte, HeaderSizeSerialized+size) | ||
| binary.BigEndian.PutUint16(buf[:2], uint16(size)) | ||
| binary.BigEndian.PutUint64(buf[2:10], h.Number) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
size is written as uint16 – overflow checks missing
binary.BigEndian.PutUint16(buf[:2], uint16(size)) silently wraps when size > 65535, corrupting the on-disk format.
Either validate or switch to a uint32 prefix:
-const HeaderSizeSerialized = 2
+const HeaderSizeSerialized = 4
...
-binary.BigEndian.PutUint16(buf[:2], uint16(size))
+binary.BigEndian.PutUint32(buf[:4], uint32(size))Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In rollup/missing_header_fields/export-headers-toolkit/types/header.go around
lines 53 to 55, the code writes the variable size as a uint16 without checking
for overflow, which can cause silent wrapping and corrupt the on-disk format if
size exceeds 65535. To fix this, add a validation step to ensure size does not
exceed the uint16 maximum before writing, or change the prefix to use a uint32
to safely accommodate larger sizes.
| func (h *Header) Vanity() [VanitySize]byte { | ||
| return [VanitySize]byte(h.ExtraData[:VanitySize]) | ||
| } | ||
|
|
||
| func (h *Header) Seal() []byte { | ||
| return h.ExtraData[VanitySize:] | ||
| } |
There was a problem hiding this comment.
Compilation error & potential slice panic in Vanity / Seal
Converting a []byte directly to [32]byte is illegal in Go – the file will not compile.
Even after fixing the conversion, slicing [:VanitySize] panics when len(ExtraData) < 32.
-func (h *Header) Vanity() [VanitySize]byte {
- return [VanitySize]byte(h.ExtraData[:VanitySize])
-}
-
-func (h *Header) Seal() []byte {
- return h.ExtraData[VanitySize:]
-}
+func (h *Header) Vanity() [VanitySize]byte {
+ var v [VanitySize]byte
+ if len(h.ExtraData) == 0 { // nothing to copy
+ return v
+ }
+ n := len(h.ExtraData)
+ if n > VanitySize {
+ n = VanitySize
+ }
+ copy(v[:], h.ExtraData[:n])
+ return v
+}
+
+func (h *Header) Seal() []byte {
+ if len(h.ExtraData) <= VanitySize {
+ return nil
+ }
+ return h.ExtraData[VanitySize:]
+}This compiles, avoids panics, and still preserves the “pad-with-zeros” semantics for short ExtraData.
🤖 Prompt for AI Agents
In rollup/missing_header_fields/export-headers-toolkit/types/header.go around
lines 61 to 67, the code incorrectly converts a byte slice to a fixed-size
array, which causes a compilation error, and slicing ExtraData without length
checks can panic if ExtraData is shorter than VanitySize. Fix this by first
checking the length of ExtraData, then safely copying up to VanitySize bytes
into a new fixed-size array for Vanity(), padding with zeros if needed, and for
Seal(), return a slice only if ExtraData length exceeds VanitySize, otherwise
return an empty slice to avoid panics.
| if headerNum > r.lastReadHeader.headerNum { | ||
| // skip the headers until the requested header number | ||
| for i := r.lastReadHeader.headerNum; i < headerNum; i++ { | ||
| if _, _, err = r.ReadNext(); err != nil { | ||
| return 0, nil, err | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Header-skipping loop can miss / duplicate blocks
The loop uses a fixed counter i, but r.lastReadHeader.headerNum is mutated by
ReadNext().
With headerNum = 10 and lastReadHeader = 1, the loop executes 9 iterations
even though only 8 additional calls are required, causing an over-read.
- for i := r.lastReadHeader.headerNum; i < headerNum; i++ {
- if _, _, err = r.ReadNext(); err != nil {
- return 0, nil, err
- }
- }
+for r.lastReadHeader.headerNum < headerNum {
+ if _, _, err = r.ReadNext(); err != nil {
+ return 0, nil, err
+ }
+}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if headerNum > r.lastReadHeader.headerNum { | |
| // skip the headers until the requested header number | |
| for i := r.lastReadHeader.headerNum; i < headerNum; i++ { | |
| if _, _, err = r.ReadNext(); err != nil { | |
| return 0, nil, err | |
| } | |
| } | |
| } | |
| if headerNum > r.lastReadHeader.headerNum { | |
| // skip the headers until the requested header number | |
| - for i := r.lastReadHeader.headerNum; i < headerNum; i++ { | |
| - if _, _, err = r.ReadNext(); err != nil { | |
| - return 0, nil, err | |
| - } | |
| - } | |
| + for r.lastReadHeader.headerNum < headerNum { | |
| + if _, _, err = r.ReadNext(); err != nil { | |
| + return 0, nil, err | |
| + } | |
| + } | |
| } |
🤖 Prompt for AI Agents
In
rollup/missing_header_fields/export-headers-toolkit/cmd/missing_header_reader.go
around lines 63 to 70, the loop that skips headers uses a fixed counter based on
the initial lastReadHeader.headerNum, but since ReadNext() updates
lastReadHeader.headerNum, this causes over-reading or duplication. To fix this,
change the loop condition to check against r.lastReadHeader.headerNum
dynamically instead of using a fixed counter, so the loop continues until
r.lastReadHeader.headerNum reaches the target headerNum.
| func NewReader(filePath string) (*Reader, error) { | ||
| f, err := os.Open(filePath) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to open file: %v", err) | ||
| } | ||
|
|
||
| r := &Reader{ | ||
| file: f, | ||
| reader: bufio.NewReader(f), | ||
| } | ||
|
|
||
| // read the count of unique vanities | ||
| vanityCount, err := r.reader.ReadByte() | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| // read the unique vanities | ||
| r.sortedVanities = make(map[int][32]byte) | ||
| for i := uint8(0); i < vanityCount; i++ { | ||
| var vanity [32]byte | ||
| if _, err = r.reader.Read(vanity[:]); err != nil { | ||
| return nil, err | ||
| } | ||
| r.sortedVanities[int(i)] = vanity | ||
| } | ||
|
|
||
| return r, nil |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Close the file on early-exit to avoid FD leaks
NewReader allocates an *os.File, but every error path after the os.Open call returns without closing it.
Add a defer that closes the handle if subsequent initialisation fails.
f, err := os.Open(filePath)
if err != nil {
return nil, fmt.Errorf("failed to open file: %v", err)
}
-
+rCleanup := func() { _ = f.Close() } // make sure we don't leak on error
+defer func() {
+ if err != nil { // any error bubbling up → close handle
+ rCleanup()
+ }
+}()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func NewReader(filePath string) (*Reader, error) { | |
| f, err := os.Open(filePath) | |
| if err != nil { | |
| return nil, fmt.Errorf("failed to open file: %v", err) | |
| } | |
| r := &Reader{ | |
| file: f, | |
| reader: bufio.NewReader(f), | |
| } | |
| // read the count of unique vanities | |
| vanityCount, err := r.reader.ReadByte() | |
| if err != nil { | |
| return nil, err | |
| } | |
| // read the unique vanities | |
| r.sortedVanities = make(map[int][32]byte) | |
| for i := uint8(0); i < vanityCount; i++ { | |
| var vanity [32]byte | |
| if _, err = r.reader.Read(vanity[:]); err != nil { | |
| return nil, err | |
| } | |
| r.sortedVanities[int(i)] = vanity | |
| } | |
| return r, nil | |
| func NewReader(filePath string) (*Reader, error) { | |
| f, err := os.Open(filePath) | |
| if err != nil { | |
| return nil, fmt.Errorf("failed to open file: %v", err) | |
| } | |
| // make sure we don't leak on error | |
| rCleanup := func() { _ = f.Close() } | |
| defer func() { | |
| if err != nil { // any error bubbling up → close handle | |
| rCleanup() | |
| } | |
| }() | |
| r := &Reader{ | |
| file: f, | |
| reader: bufio.NewReader(f), | |
| } | |
| // read the count of unique vanities | |
| vanityCount, err := r.reader.ReadByte() | |
| if err != nil { | |
| return nil, err | |
| } | |
| // read the unique vanities | |
| r.sortedVanities = make(map[int][32]byte) | |
| for i := uint8(0); i < vanityCount; i++ { | |
| var vanity [32]byte | |
| if _, err = r.reader.Read(vanity[:]); err != nil { | |
| return nil, err | |
| } | |
| r.sortedVanities[int(i)] = vanity | |
| } | |
| return r, nil | |
| } |
🤖 Prompt for AI Agents
In
rollup/missing_header_fields/export-headers-toolkit/cmd/missing_header_reader.go
around lines 26 to 53, the file opened with os.Open is not closed on error paths
after opening, causing file descriptor leaks. To fix this, add a defer statement
immediately after opening the file that closes it, and cancel this defer if the
function completes successfully. This ensures the file is closed on all early
returns due to errors during initialization.
| var wgConsumer sync.WaitGroup | ||
| // start consumer goroutine to sort and write headers to file | ||
| go func() { | ||
| wgConsumer.Add(1) | ||
| writeHeadersToFile(outputFile, humanReadableOutputFile, startBlockNum, headersChan) | ||
| wgConsumer.Done() | ||
| }() | ||
|
|
There was a problem hiding this comment.
WaitGroup use is racy – call Add before spawning the goroutine
Add must be invoked before the goroutine could call Done; otherwise the
main goroutine might reach Wait before the counter is incremented, leading to
undefined behaviour.
-// start consumer goroutine to sort and write headers to file
-go func() {
- wgConsumer.Add(1)
+// start consumer goroutine to sort and write headers to file
+wgConsumer.Add(1)
+go func() {
writeHeadersToFile(outputFile, humanReadableOutputFile, startBlockNum, headersChan)
wgConsumer.Done()
}()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| var wgConsumer sync.WaitGroup | |
| // start consumer goroutine to sort and write headers to file | |
| go func() { | |
| wgConsumer.Add(1) | |
| writeHeadersToFile(outputFile, humanReadableOutputFile, startBlockNum, headersChan) | |
| wgConsumer.Done() | |
| }() | |
| var wgConsumer sync.WaitGroup | |
| // start consumer goroutine to sort and write headers to file | |
| wgConsumer.Add(1) | |
| go func() { | |
| writeHeadersToFile(outputFile, humanReadableOutputFile, startBlockNum, headersChan) | |
| wgConsumer.Done() | |
| }() |
🤖 Prompt for AI Agents
In rollup/missing_header_fields/export-headers-toolkit/cmd/fetch.go around lines
137 to 144, the WaitGroup's Add method is called inside the goroutine, which can
cause a race condition. Move the wgConsumer.Add(1) call to before starting the
goroutine to ensure the counter is incremented before the goroutine runs. This
prevents the main goroutine from calling Wait before Add is called.
| func (h *csvHeaderReader) readNext() *types.Header { | ||
| line, err := h.reader.ReadString('\n') | ||
| if err != nil { | ||
| if err == io.EOF { | ||
| return nil | ||
| } | ||
| log.Fatalf("Error reading line: %v", err) | ||
| } | ||
|
|
||
| s := strings.Split(line, ",") | ||
| extraString := strings.Split(s[2], "\n") | ||
|
|
||
| num, err := strconv.ParseUint(s[0], 10, 64) | ||
| if err != nil { | ||
| log.Fatalf("Error parsing block number: %v", err) | ||
| } | ||
| difficulty, err := strconv.ParseUint(s[1], 10, 64) | ||
| if err != nil { | ||
| log.Fatalf("Error parsing difficulty: %v", err) | ||
| } | ||
| extra := common.FromHex(extraString[0]) | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Robustness: guard against malformed CSV rows
csvHeaderReader.readNext assumes every line contains at least three
comma-separated fields. An empty or malformed line causes index out of range.
Add basic validation:
parts := strings.Split(strings.TrimSpace(line), ",")
if len(parts) < 3 {
log.Fatalf("Malformed CSV line: %q", line)
}🤖 Prompt for AI Agents
In rollup/missing_header_fields/export-headers-toolkit/cmd/dedup.go around lines
217 to 238, the readNext method assumes each CSV line has at least three
comma-separated fields, which can cause an index out of range panic on malformed
or empty lines. To fix this, add validation after reading the line by trimming
whitespace and splitting on commas, then check if the resulting slice has fewer
than three elements. If so, log a fatal error indicating a malformed CSV line
and include the line content for debugging.
| func runSHA256(outputFile string) { | ||
| f, err := os.Open(outputFile) | ||
| defer f.Close() | ||
| if err != nil { | ||
| log.Fatalf("Error opening file: %v", err) | ||
| } | ||
|
|
There was a problem hiding this comment.
defer f.Close() executed on nil pointer when os.Open fails
If os.Open returns an error, f is nil, and the deferred call panics.
Move the defer after the error check.
-f, err := os.Open(outputFile)
-defer f.Close()
-if err != nil {
- log.Fatalf("Error opening file: %v", err)
-}
+f, err := os.Open(outputFile)
+if err != nil {
+ log.Fatalf("Error opening file: %v", err)
+}
+defer f.Close()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func runSHA256(outputFile string) { | |
| f, err := os.Open(outputFile) | |
| defer f.Close() | |
| if err != nil { | |
| log.Fatalf("Error opening file: %v", err) | |
| } | |
| func runSHA256(outputFile string) { | |
| f, err := os.Open(outputFile) | |
| if err != nil { | |
| log.Fatalf("Error opening file: %v", err) | |
| } | |
| defer f.Close() |
🤖 Prompt for AI Agents
In rollup/missing_header_fields/export-headers-toolkit/cmd/dedup.go around lines
125 to 131, the defer statement calling f.Close() is placed before checking if
os.Open returned an error, which can cause a panic if f is nil. Move the defer
f.Close() call to after the error check to ensure f is not nil before deferring
the close operation.
1. Purpose or design rationale of this PR
We are using the Clique consensus in Scroll L2. Amongst others, it requires the following header fields:
extraDatadifficultyHowever, these fields are currently not stored on L1/DA and we're planning to add them in a future upgrade.
In order for nodes to be able to reconstruct the correct block hashes when only reading data from L1,
we need to provide the historical values of these fields to these nodes through a separate file.
This toolkit provides commands to export the missing fields, deduplicate the data and create a file
with the missing fields that can be used to reconstruct the correct block hashes when only reading data from L1.
Analysis of data
Mainnet until block 7455960
Sepolia until block 5422047
2. PR title
Your PR title must follow conventional commits (as we are doing squash merge for each PR), so it must start with one of the following types:
3. Deployment tag versioning
Has the version in
params/version.gobeen updated?4. Breaking change label
Does this PR have the
breaking-changelabel?Summary by CodeRabbit
New Features
Documentation
Tests