Supply delta live tracer #29347

ziogaschr · 2024-03-26T09:56:48Z

This PR is based on #24723 by @karalabe / @holiman (and the successor #25662) and uses the live chain tracing with hooks implementation #29189 by @s1na.

This PR adds a live tracer which stores the supply delta of each block during import of a block.
For finding the supply for the whole chain, a fresh sync has to be started.

The tracer output for a single block is:

{
  "issuance": {
    "genesisAlloc": "0x0",
    "reward": "0x0",
    "withdrawals": "0x0"
  },
  "burn": {
    "eip1559": "0x0",
    "blob": "0x0",
    "misc": "0x0"
  },
  "blockNumber": 1,
  "hash": "0x54177b2adb754306d0d267f82537551c467201bf0c8509e60b77229b41c99e3a",
  "parentHash": "0xaf41e72f748de317965454508c749f7e14dc4fe444cd07bca4c981c7e952364d"
}

For using this tracer the CLI flag --vmtrace supply has to be set. Additional options for the tracer can be set as --vmtrace.config '{"path": "/tmp/supply_logs/"}'.

Currently the following options are supported:

path        Path to the directory where the tracer logs will be stored
maxSize     MaxSize is the maximum size in megabytes of the tracer log file before it gets rotated. It defaults to 100 megabytes.

The output of this tracer is stored in log rotated JSONL files in the configured directory, those exported files have to be analysed with another tool.

Example tracer output:

{"delta":320000001000000000000000000,"reward":0,"withdrawals":0,"burn":0,"blockNumber":0,"hash":"0x25a5cc106eea7138acab33231d7160d69cb777ee0c2c553fcddf5138993e6dd9","parentHash":"0x0000000000000000000000000000000000000000000000000000000000000000"}
{"delta":2000000000000000000,"reward":2000000000000000000,"withdrawals":0,"burn":0,"blockNumber":1,"hash":"0x696d95da6726a67afd5be2a37d3883e9be8008491b30d5bd1069ea5922fa2a41","parentHash":"0x25a5cc106eea7138acab33231d7160d69cb777ee0c2c553fcddf5138993e6dd9"}
{"delta":2000000000000000000,"reward":2000000000000000000,"withdrawals":0,"burn":0,"blockNumber":2,"hash":"0x89b977ffab1052f5f6f778e4680354a8a1a1e0f623db9b2f20beba9f36f68721","parentHash":"0x696d95da6726a67afd5be2a37d3883e9be8008491b30d5bd1069ea5922fa2a41"}
{"delta":2000000000000000000,"reward":2000000000000000000,"withdrawals":0,"burn":0,"blockNumber":3,"hash":"0x1970908e724ac37efbb0b3bf9f77ebdf04e1ccb3f9da1463bec00252cb3a701f","parentHash":"0x89b977ffab1052f5f6f778e4680354a8a1a1e0f623db9b2f20beba9f36f68721"}

This reverts commit bc983e2.

fjl · 2024-03-28T11:12:44Z

core/tracing/hooks.go

@@ -214,6 +214,44 @@ const (
 	BalanceDecreaseSelfdestructBurn BalanceChangeReason = 14
 )

+// String returns a string representation of the reason.
+func (r BalanceChangeReason) String() string {


Please use https://pkg.go.dev/golang.org/x/tools/cmd/stringer to generate this method.

@fjl is this what you meant? 9e21e2d

https://pkg.go.dev/golang.org/x/tools/cmd/stringer

eth/tracers/live/supply.go

We try to solve the following error: ``` --- FAIL: TestSupplySelfdestruct (1.56s) testing.go:1231: TempDir RemoveAll cleanup: remove C:\Users\appveyor\AppData\Local\Temp\1\TestSupplySelfdestruct3803137642\001\supply.jsonl: The process cannot access the file because it is being used by another process. ```

Was expecting lumberjack.Rotate() to close the file

core/blockchain.go

eth/tracers/live/supply.go

Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>

s1na

LGTM but I was co-author :) I'll wait for others to chime in as well

ziogaschr · 2024-04-03T11:10:49Z

core/tracing/hooks.go

-	OnSkippedBlock   SkippedBlockHook
-	OnGenesisBlock   GenesisBlockHook
+	OnBlockchainInit      BlockchainInitHook
+	OnBlockchainTerminate BlockchainTerminateHook


@s1na what do you think of this name over "OnBlockchainEnd"?

s1na · 2024-04-16T12:33:52Z

Linking the reader here for visibility: https://github.com/ziogaschr/supply-tracer-parser

core/blockchain.go

holiman · 2024-04-23T12:50:19Z

eth/tracers/internal/tracetest/supply_test.go

+	Delta       *big.Int `json:"delta"`
+	Reward      *big.Int `json:"reward"`
+	Withdrawals *big.Int `json:"withdrawals"`
+	Burn        *big.Int `json:"burn"`


big.Int marshals as json numbers. Which means that we're kind of limited to 53 bits, because of javascript. So better if we marshal it as hex-strings (or dec-strings for that matter).

So Delta is a signed number. The best thing I can think of is to add a new field for the sign.

I like your idea @s1na, as supporting Negative numbers for hex isn't the nicest thing and also the consumers of the JSON will have to find a library that supports negative numbers as well.

Another suggestion might be if we use two different fields like DeltaInflation and DeltaDeflation (we have to think of better names) (thanks to @meowsbits for this suggestion)

What do you think of this? ziogaschr@3ac74bd#r142142290

@holiman @s1na I pushed some commits which change the output to the following after our discussion with @s1na.
We dropped the delta, which could have negative numbers and we will let the consumer to do the calculations on their side.

{ "issuance": { "genesisAlloc": "0x0", // omitempty "reward": "0x0", "withdrawals": "0x0" }, "burn": { "1559": "0x0", "blob": "0x0", "misc": "0x0" }, "blockNumber": 1, "hash": "0x54177b2adb754306d0d267f82537551c467201bf0c8509e60b77229b41c99e3a", "parentHash": "0xaf41e72f748de317965454508c749f7e14dc4fe444cd07bca4c981c7e952364d" }

holiman · 2024-04-25T06:16:48Z

This needs a rebase

# Conflicts: # core/tracing/hooks.go

The new output becomes: ``` { "issuance": { "reward": 0, "withdrawals": 0 }, "burn": { "1559": 0, "blob": 0, "misc": 0 }, "blockNumber": 1, "hash": "0x", "parentHash": "0x" } ```

eth/tracers/live/supply.go

s1na

Looks good to me! 🙌 🙌

Pazibad · 2024-06-07T16:35:25Z

0x34a9c05b638020a07bb153bf624c8763bf8b4a86

Introduces the first built-in live tracer. The supply tracer tracks ETH supply changes across blocks and writes the output to disk. This will need to be enabled through CLI using the `--vmtrace supply` flag. Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>

ziogaschr and others added 7 commits March 26, 2024 11:30

add BalanceChangeReason String() for better logging

c686e30

add supply delta live tracer

a4117a8

change []byte to json.RawMessage in test

55a9f40

fix temp path on Windows by converting backslashes to slashes

5db2083

allow test to use different file for tracer output

bc983e2

Revert "allow test to use different file for tracer output"

3a6cf0a

This reverts commit bc983e2.

try alternative tempdir

53f6152

s1na mentioned this pull request Mar 27, 2024

Supply live tracer s1na/go-ethereum#21

Closed

fjl reviewed Mar 28, 2024

View reviewed changes

ziogaschr added 2 commits March 28, 2024 14:00

use stringer to generate BalanceChangeReason String method

9e21e2d

https://pkg.go.dev/golang.org/x/tools/cmd/stringer

fix failing test

a6cabe1

s1na reviewed Mar 28, 2024

View reviewed changes

eth/tracers/live/supply.go Outdated Show resolved Hide resolved

ziogaschr added 9 commits March 28, 2024 20:02

make supply type private

df69332

make SupplyInfo private

fb0160b

typo interalTxsHandler->internalTxsHandler

6e7f5bc

try a fix for “TempDir RemoveAll cleanup”

91260a7

try a fix for test to pass on Windows

cba97ef

fix test by closing logger file

4708a19

Was expecting lumberjack.Rotate() to close the file

add OnTerminate in Hooks which triggered on blockchain Stop

00ef42b

remove comment

de1c41d

s1na reviewed Mar 31, 2024

View reviewed changes

core/blockchain.go Outdated Show resolved Hide resolved

s1na reviewed Mar 31, 2024

View reviewed changes

eth/tracers/live/supply.go Outdated Show resolved Hide resolved

ziogaschr and others added 2 commits March 31, 2024 16:55

Update core/blockchain.go

200f496

Co-authored-by: Sina M <1591639+s1na@users.noreply.github.com>

log any error during log file close

f65a19a

ziogaschr marked this pull request as ready for review March 31, 2024 20:06

ziogaschr requested review from karalabe, holiman and rjl493456442 as code owners March 31, 2024 20:06

refactor logging

e67c3bf

s1na added the status:triage label Apr 2, 2024

s1na approved these changes Apr 2, 2024

View reviewed changes

rename Hook “OnTerminate” to “OnBlockchainTerminate”

5ba5254

ziogaschr commented Apr 3, 2024

View reviewed changes

holiman reviewed Apr 23, 2024

View reviewed changes

core/blockchain.go Outdated Show resolved Hide resolved

holiman reviewed Apr 23, 2024

View reviewed changes

s1na removed the status:triage label Apr 23, 2024

ziogaschr added 6 commits April 25, 2024 09:34

Merge branch 'master' into supply-delta-live-tracer

a6541d2

# Conflicts: # core/tracing/hooks.go

s/OnBlockchainTerminate/OnClose

a2639b2

supply test compares output as JSON

dec5d3b

change output JSON schema

b07ef33

The new output becomes: ``` { "issuance": { "reward": 0, "withdrawals": 0 }, "burn": { "1559": 0, "blob": 0, "misc": 0 }, "blockNumber": 1, "hash": "0x", "parentHash": "0x" } ```

output hex values instead of numbers

e4a2c82

Merge branch 'master' into supply-delta-live-tracer

b12c669

s1na reviewed May 30, 2024

View reviewed changes

eth/tracers/live/supply.go Outdated Show resolved Hide resolved

ziogaschr added 2 commits May 31, 2024 15:00

omitempty fields from supply tracer output

3ed1bd8

keep logic of removing empty fields in a single place for tech debt

405ceae

s1na approved these changes May 31, 2024

View reviewed changes

s1na merged commit c9e0b31 into ethereum:master Jun 3, 2024
3 checks passed

s1na added this to the 1.14.4 milestone Jun 3, 2024

ziogaschr mentioned this pull request Jun 4, 2024

Update schema ziogaschr/supply-tracer-parser#2

Merged

This was referenced Jun 5, 2024

ethereum 1.14.4 Homebrew/homebrew-core#173743

Merged

ethereum 1.14.5 Homebrew/homebrew-core#173874

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Supply delta live tracer #29347

Supply delta live tracer #29347

ziogaschr commented Mar 26, 2024 •

edited

Loading

fjl Mar 28, 2024

ziogaschr Mar 28, 2024

s1na left a comment

ziogaschr Apr 3, 2024

s1na commented Apr 16, 2024

holiman Apr 23, 2024

s1na May 14, 2024

ziogaschr May 17, 2024

ziogaschr May 17, 2024 •

edited

Loading

ziogaschr May 30, 2024

holiman commented Apr 25, 2024

s1na left a comment

Pazibad commented Jun 7, 2024

Supply delta live tracer #29347

Supply delta live tracer #29347

Conversation

ziogaschr commented Mar 26, 2024 • edited Loading

fjl Mar 28, 2024

Choose a reason for hiding this comment

ziogaschr Mar 28, 2024

Choose a reason for hiding this comment

s1na left a comment

Choose a reason for hiding this comment

ziogaschr Apr 3, 2024

Choose a reason for hiding this comment

s1na commented Apr 16, 2024

holiman Apr 23, 2024

Choose a reason for hiding this comment

s1na May 14, 2024

Choose a reason for hiding this comment

ziogaschr May 17, 2024

Choose a reason for hiding this comment

ziogaschr May 17, 2024 • edited Loading

Choose a reason for hiding this comment

ziogaschr May 30, 2024

Choose a reason for hiding this comment

holiman commented Apr 25, 2024

s1na left a comment

Choose a reason for hiding this comment

Pazibad commented Jun 7, 2024

ziogaschr commented Mar 26, 2024 •

edited

Loading

ziogaschr May 17, 2024 •

edited

Loading