This repository has been archived by the owner on May 21, 2024. It is now read-only.
Statediffing service #4
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* port statediff from https://github.com/jpmorganchase/quorum/blob/9b7fd9af8082795eeeb6863d9746f12b82dd5078/statediff/statediff.go; minor fixes * integrating state diff extracting, building, and persisting into geth processes * work towards persisting created statediffs in ipfs; based off github.com/vulcanize/eth-block-extractor * Add a state diff service * Remove diff extractor from blockchain * Update imports * Move statediff on/off check to geth cmd config * Update starting state diff service * Add debugging logs for creating diff * Add statediff extractor and builder tests and small refactoring * Start to write statediff to a CSV * Restructure statediff directory * Pull CSV publishing methods into their own file * Reformatting due to go fmt * Add gomega to vendor dir * Remove testing focuses * Update statediff tests to use golang test pkg instead of ginkgo - builder_test - extractor_test - publisher_test * Use hexutil.Encode instead of deprecated common.ToHex * Remove OldValue from DiffBigInt and DiffUint64 fields * Update builder test * Remove old storage value from updated accounts * Remove old values from created/deleted accounts * Update publisher to account for only storing current account values * Update service loop and fetching previous block * Update testing - remove statediff ginkgo test suite file - move mocks to their own dir * Updates per go fmt * Updates to tests * Pass statediff mode and path in through cli * Return filename from publisher * Remove some duplication in builder * Remove code field from state diff output this is the contract byte code, and it can still be obtained by querying the db by the codeHash * Consolidate acct diff structs for updated & updated/deleted accts * Include block number in csv filename * Clean up error logging * Cleanup formatting, spelling, etc * Address PR comments * Add contract address and storage value to csv * Refactor accumulating account row in csv publisher * Add DiffStorage struct * Add storage key to csv * Address PR comments * Fix publisher to include rows for accounts that don't have store updates * Update builder test after merging in release/1.8 * Update test contract to include storage on contract intialization - so that we're able to test that storage diffing works for created and deleted accounts (not just updated accounts). * Factor out a common trie iterator method in builder
* Open a trie from the in-memory database * Use a node's LeafKey as an identifier instead of the address It was proving difficult to find look the address up from a given path with a full node (sometimes the value wouldn't exist in the disk db). So, instead, for now we are using the node's LeafKey with is a Keccak256 hash of the address, so if we know the address we can figure out which LeafKey it matches up to. * Make sure that statediff has been processed before pruning * Use blockchain stateCache.OpenTrie for storage diffs * Clean up log lines and remove unnecessary fields from builder * Apply go fmt changes * Add a sleep to the blockchain test * Address PR comments * Address PR comments
…and paths (still need to make this optional)
…ing state diff payloads over an rpc websocket subscription
…r loop (may be missing something here)
… statediff processing is halted/paused until there is at least one subscriber listening for the results
* travis: Enable ARM support * Include fixes from 20039 * Add a trace to debug the invalid lookup issue * Try increasing the timeout to see if the arm test passes * Investigate the resolver issue * Increase arm64 timeout for clique test * increase timeout in tests for arm64 * Only test the failing tests * Review feedback: don't export epsilon * Remove investigation tricks+include fjl's feeback * Revert the retry ahead of using the mock resolver * Fix rebase errors
This adds all dashboard changes from the last couple months. We're about to remove the dashboard, but decided that we should get all the recent work in first in case anyone wants to pick up this project later on. * cmd, dashboard, eth, p2p: send peer info to the dashboard * dashboard: update npm packages, improve UI, rebase * dashboard, p2p: remove println, change doc * cmd, dashboard, eth, p2p: cleanup after review * dashboard: send current block to the dashboard client
TestEmptyWork* occasionally fails due to timeout. Increase the timeout.
This PR implements the LES server RPC API. Methods for server capacity, client balance and client priority management are provided.
…0244) * accounts/abi/bind, cmd/abigen: implement alias for abigen * accounts/abi/bind: minor fixes * accounts/abi/bind: address comments * cmd/abigen: address comments * accounts/abi/bind: print error log when identifier collision * accounts/abi/bind: address comments * accounts/abi/bind: address comment
* p2p/discover: slow down lookups on empty table * p2p/discover: wake from slowdown sleep when table is closed
* core: fix staticcheck warnings * fix goimports
The original idea behind this change was to remove a use of the deprecated CancelRequest method. Simply removing it would've been an option, but I couldn't resist and did a bit of a refactoring instead. All remote sealing code was contained in a single giant function. Remote sealing is now extracted into its own object, remoteSealer.
This change: - removes the PostChainEvents method on core.BlockChain. - sorts 'removed log' events by block number. - fire the NewChainHead event if we inject a canonical block into the chain even if the entire insertion is not successful. - guarantees correct event ordering in all cases.
) * consensus/ethash, params: implement eip-2384: bump difficulty bomb * params: EIP 2384 compat checks * consensus, params: add Muir Glacier block number (mainnet,ropsten) + official name * core/forkid: forkid tests for muir glacier * params/config: address review concerns * params, core/forkid: review nitpicks * cmd/geth,eth,les: add override option for muir glacier * params: nit fix
- Remove vendor dir - Removing unused geth cmd config - Apply govet updates
elizabethengelman
force-pushed
the
statediffing-service
branch
from
4a00c66 to
ed2bc05
Compare
yaoandrew
pushed a commit
that referenced
this pull request
Jul 7, 2021
* eth/protocols: persist received state segments * core: initial implementation * core/state/snapshot: add tests * core, eth: updates * eth/protocols/snapshot: count flat state size * core/state: add metrics * core/state/snapshot: skip unnecessary deletion * core/state/snapshot: rename * core/state/snapshot: use the global batch * core/state/snapshot: add logs and fix wiping * core/state/snapshot: fix * core/state/snapshot: save generation progress even if the batch is empty * core/state/snapshot: fixes * core/state/snapshot: fix initial account range length * core/state/snapshot: fix initial account range * eth/protocols/snap: store flat states during the healing * eth/protocols/snap: print logs * core/state/snapshot: refactor (#4) * core/state/snapshot: refactor * core/state/snapshot: tiny fix and polish Co-authored-by: rjl493456442 <garyrong0905@gmail.com> * core, eth: fixes * core, eth: fix healing writer * core, trie, eth: fix paths * eth/protocols/snap: fix encoding * eth, core: add debug log * core/state/generate: release iterator asap (#5) core/state/snapshot: less copy core/state/snapshot: revert split loop core/state/snapshot: handle storage becoming empty, improve test robustness core/state: test modified codehash core/state/snapshot: polish * core/state/snapshot: optimize stats counter * core, eth: add metric * core/state/snapshot: update comments * core/state/snapshot: improve tests * core/state/snapshot: replace secure trie with standard trie * core/state/snapshot: wrap return as the struct * core/state/snapshot: skip wiping correct states * core/state/snapshot: updates * core/state/snapshot: fixes * core/state/snapshot: fix panic due to reference flaw in closure * core/state/snapshot: fix errors in state generation logic + fix log output * core/state/snapshot: remove an error case * core/state/snapshot: fix condition-check for exhausted snap state * core/state/snapshot: use stackTrie for small tries * core/state/snapshot: don't resolve small storage tries in vain * core/state/snapshot: properly clean up storage of deleted accounts * core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks * core/state/snapshot: fix error (+testcase) * core/state/snapshot: clean up tests a bit * core/state/snapshot: work in progress on better tests * core/state/snapshot: polish code * core/state/snapshot: fix trie iteration abortion trigger * core/state/snapshot: fixes flaws * core/state/snapshot: remove panic * core/state/snapshot: fix abort * core/state/snapshot: more tests (plus failing testcase) * core/state/snapshot: more testcases + fix for failing test * core/state/snapshot: testcase for malformed data * core/state/snapshot: some test nitpicks * core/state/snapshot: improvements to logging * core/state/snapshot: testcase to demo error in abortion * core/state/snapshot: fix abortion * cmd/geth: make verify-state report the root * trie: fix failing test * core/state/snapshot: add timer metrics * core/state/snapshot: fix metrics * core/state/snapshot: udpate tests * eth/protocols/snap: write snapshot account even if code or state is needed * core/state/snapshot: fix diskmore check * core/state/snapshot: review fixes * core/state/snapshot: improve error message * cmd/geth: rename 'error' to 'err' in logs * core/state/snapshot: fix some review concerns * core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync * core: add error log * core/state/snapshot: use proper timers for metrics collection * core/state/snapshot: address some review concerns * eth/protocols/snap: improved log message * eth/protocols/snap: fix heal logs to condense infos * core/state/snapshot: wait for generator termination before restarting * core/state/snapshot: revert timers to counters to track total time Co-authored-by: Martin Holst Swende <martin@swende.se> Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update: https://github.com/makerdao/go-ethereum/releases/tag/v1.9.9-rc1 corresponds to this branch/PR
On the previous version of this repo, we had the statediffing branch hanging out as a PR without merging it into
master. I'm thinking that it may be easier to try to get this merged intomastersooner rather than later, so was planning to try to get #2 merged into this branch first, and then working to get this ready to merge into master. I'd be curious to hear if folks think that's a good idea!This branch includes: