Verkle tree-based state with overlay transition

Squash the main verkle PR ahead of rebase don't call Bytes() in GetTreeKey (ethereum#137) trie: avoid endianness conversion in GetTreeKey (ethereum#140) * trie/utils: add concrete expected value in trie key generation test Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * mod: update to latest go-verkle Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * trie/utils: avoid endianness conversions Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * apply review changes & update to official go-verkle version Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> upgrade go-verkle to CoW version and get TestProcessVerkle to build (ethereum#138) updating ci to use self-hosted machine (ethereum#143) fix: storage offset in non-header group + reuse of value buffer (ethereum#145) dedup call to ChunkifyCode, same as replay branch (ethereum#156) * dedup call to ChunkifyCode, same as replay branch * fix some linter issues fix code offset in tree update (ethereum#157) fix REVERT in state processor test execution (ethereum#158) * fix code offset in tree update * fix REVERT in test execution save on key hashing: lump code size update with first code chunk group (ethereum#159) fix code chunk key calculation and storage key calculation (ethereum#161) * fix codeKey calculation * Remove * fix storageOffset * fix the fix to the fix to the offset fix * Remove copy/pasted, unused code in test * fix linter --------- Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com> fix: infinite loop when calling extcodecopy on empty code (ethereum#151) upgrade to latest go-verkle fix: only update code in the tree if it's dirty (ethereum#174) fix: read-touch the code size and Keccak of the origin (ethereum#175) List of changes for converting a sepolia database (ethereum#182) * naive conversion rebased on top of beverly hills * changes for the sepolia shadow fork conversion * fixes to please the linter * fixes to please the linter Unified point cache (ethereum#180) * Unified point cache * Use cache for Try*Account * alter Trie interface to use caching for slots (ethereum#181) * alter Trie interface to use caching for slots * fix: use a lock to protect the point cache (ethereum#185) * use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (ethereum#184) * mod: update to fastest go-verkle version today Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * trie/verkle: use new batch serialization api Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com> * fix: TryDelete signature in unit tests --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com> trie/utils: fix potential overflow (ethereum#191) * trie/utils: fix potential overflow Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * trie/utils: receive storage key as a byte slice Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * revert formatter changes Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * trie/utils: fix mod 256 Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> trie/utils: fix incorrect bigint assignment (ethereum#193) Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> upgrade precomp link to fix CI fix: add missing code size&keccak leaves in empty accounts (ethereum#192) fixes to use the latest go-verkle@master (ethereum#197) * fixes to use the latest go-verkle@master * linter fixes * linter fixes for tests * fix: use jsign's go-verkle fix refactor: remove unused (*StateDB).GetXLittleEndian methods (ethereum#204) fix gas accounting issue in state_processor_test.go (ethereum#207) update go-verkle not to use StatelessNode anymore (ethereum#206) * update go-verkle not to use StatelessNode anymore * update go-verkle to latest refactor: move verkle gas accounting to its own block in TransitionDB (ethereum#208) fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements use the cachingDB instead of a custom VerkleDB (ethereum#209) * use the cachingDB instead of a custom VerkleDB * fix stack trace in LES remove holiman from CODEOWNERS as he gets too many emails read from tree in state object if the snapshot is nil (ethereum#205) add missing error checks for the root node type (ethereum#214) implement OpenStorageTrie for verkle trees (ethereum#210) * implement OpenStorageTrie for verkle trees * add a few comments for future maintenance * fix linter issue fix: copy balance leaf to new buffer in TryGetAccount (ethereum#217) implement some heretofore unimplemented iterator methods (ethereum#219) params: move verkle params to their own file (ethereum#228) fix: proper number of chunk evals (ethereum#215) overlay transition (ethereum#244) * overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com> Include base -> overlay key-values migration logic (ethereum#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (ethereum#202) fixes for performing the overlay transition (ethereum#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (ethereum#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (ethereum#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (ethereum#222) * fixes for pbss replay (ethereum#227) * fixes for pbss replay * trie/verkle: use capped batch size (ethereum#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * don't keep more than 32 state root conversions in RAM (ethereum#230) --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * re-enable snapshot (ethereum#231) * re-enable cancun block / snapshot (ethereum#226) * clear storage conversion key upon translating account (ethereum#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com> * fix: self-deadlock with translated root map mutex (ethereum#236) * return compressed commitment as root commitment (ethereum#237) --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com> --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com> --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (ethereum#239) Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> core: print state root every 100 blocks (ethereum#240) Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> fix: only Commit the account trie (ethereum#242) fixes to get TestProcessVerkle to work with the overlay branch (ethereum#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (ethereum#247) * code cleanup (ethereum#248) * fix: don't error on a missing conversion.txt (ethereum#249) * Overlay Tree preimages exporting and usage (ethereum#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * cleanup Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * review feedback Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * fix: reduce the PR footprint (ethereum#250) * fix: don't fail when preimages.bin is missing (ethereum#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com> review changes remove replay-specific code
rjl493456442 · Aug 28, 2023 · f1ae3e8 · f1ae3e8
1 parent df64a21
commit f1ae3e8
Show file tree

Hide file tree

Showing 57 changed files with 3,963 additions and 58 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -6,7 +6,7 @@ accounts/scwallet               @gballet
 accounts/abi                    @gballet @MariusVanDerWijden
 cmd/clef                        @holiman
 consensus                       @karalabe
-core/                           @karalabe @holiman @rjl493456442
+core/                           @karalabe @rjl493456442
 eth/                            @karalabe @holiman @rjl493456442
 eth/catalyst/                   @gballet
 eth/tracers/                    @s1na

diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml
@@ -0,0 +1,48 @@
+name: Go lint and test
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master, verkle-trie-proof-in-block-rebased, verkle-trie-post-merge, beverly-hills-head, 'verkle/replay-change-with-tree-group-tryupdate' ]
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: self-hosted
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.18
+      - name: Build
+        run: go build -v ./...
+
+  lint:
+    runs-on: self-hosted
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.18
+      - name: Download golangci-lint
+        run: wget -O- -nv https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s latest
+      - name: Lint
+        run: ./bin/golangci-lint run
+      - name: Vet
+        run: go vet
+
+  test:
+    runs-on: self-hosted
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.18
+      - name: Download precomputed points
+        run: wget -nv https://github.com/gballet/go-verkle/releases/download/banderwagonv3/precomp -Otrie/utils/precomp
+      - name: Test
+        run: go test ./...
diff --git a/cmd/geth/chaincmd.go b/cmd/geth/chaincmd.go
@@ -150,6 +150,17 @@ It's deprecated, please use "geth db import" instead.
 		Description: `
 The export-preimages command exports hash preimages to an RLP encoded stream.
 It's deprecated, please use "geth db export" instead.
+`,
+	}
+	exportOverlayPreimagesCommand = &cli.Command{
+		Action:    exportOverlayPreimages,
+		Name:      "export-overlay-preimages",
+		Usage:     "Export the preimage in overlay tree migration order",
+		ArgsUsage: "<dumpfile>",
+		Flags:     flags.Merge([]cli.Flag{utils.TreeRootFlag}, utils.DatabasePathFlags),
+		Description: `
+The export-overlay-preimages command exports hash preimages to a flat file, in exactly
+the expected order for the overlay tree migration.
 `,
 	}
 	dumpCommand = &cli.Command{
@@ -407,6 +418,33 @@ func exportPreimages(ctx *cli.Context) error {
 	return nil
 }
 
+// exportOverlayPreimages dumps the preimage data to a flat file.
+func exportOverlayPreimages(ctx *cli.Context) error {
+	if ctx.Args().Len() < 1 {
+		utils.Fatalf("This command requires an argument.")
+	}
+	stack, _ := makeConfigNode(ctx)
+	defer stack.Close()
+
+	chain, _ := utils.MakeChain(ctx, stack, true)
+
+	var root common.Hash
+	if ctx.String(utils.TreeRootFlag.Name) != "" {
+		rootBytes := common.FromHex(ctx.String(utils.StartKeyFlag.Name))
+		if len(rootBytes) != common.HashLength {
+			return fmt.Errorf("invalid root hash length")
+		}
+		root = common.BytesToHash(rootBytes)
+	}
+
+	start := time.Now()
+	if err := utils.ExportOverlayPreimages(chain, ctx.Args().First(), root); err != nil {
+		utils.Fatalf("Export error: %v\n", err)
+	}
+	fmt.Printf("Export done in %v\n", time.Since(start))
+	return nil
+}
+
 func parseDumpConfig(ctx *cli.Context, stack *node.Node) (*state.DumpConfig, ethdb.Database, common.Hash, error) {
 	db := utils.MakeChainDatabase(ctx, stack, true)
 	var header *types.Header

diff --git a/cmd/geth/main.go b/cmd/geth/main.go
@@ -210,6 +210,7 @@ func init() {
 		exportCommand,
 		importPreimagesCommand,
 		exportPreimagesCommand,
+		exportOverlayPreimagesCommand,
 		removedbCommand,
 		dumpCommand,
 		dumpGenesisCommand,