ERC: State Snapshot JSON format

[cdetrio]

This is a pre-draft ERC (Ethereum Request for Comments) to gather feedback on standardizing the format for JSON state snapshots. Currently, two clients (pyethereum and Parity) support exporting JSON state snapshots in two different formats. For reference, an example of each is included below.

In comparing the two formats to a single standard, there are major and minor aspects to consider. The major ones:

• Should a state snapshot include any header data of previous blocks?

  Pyethereum includes some header data from previous blocks (prev_headers), as well some header data of recent uncles (recent_uncles). Parity includes no header data.

• Should a state snapshot include any block header data or metadata?

  Pyethereum includes the following data: block_number, block_coinbase, refunds, timestamp, gas_used, txindex, block_difficulty, bloom, gas_limit. Parity includes none.

• Should state snapshots cover intermediate states?

  Pyethereum uses tx_index to indicate that the snapshot is of an intermediate state.

On the third point, I'll suggest that intermediate snapshots be JSON Patch extensions of the state snapshot.

Minor aspects to consider include encodings (hex or decimal), hash fields (code_hash and storage_root, which can be derived by hashing the code and storage, respectively), and null/empty fields.

---

Here is an excerpt of a Pyethereum state snapshot, at block #100000 (full snapshot):

{
  "block_number":"100000",
  "alloc":{
    "381d12209f3d12ca9b10b47eeddb206c4e790137":{
      "nonce":"2",
      "balance":"0",
      "storage":{},
      "code":"0x"
    },
    "845eda2dc559f882d393ffe560a5a6989f7a46e5":{
      "nonce":"2",
      "balance":"997681245562021000",
      "storage":{},
      "code":"0x"
    },
    "412c794424c331d8794edf9aa54b4fdaece5f4fd":{
      "nonce":"0",
      "balance":"0",
      "storage":{
        "f6e37d4d23fa855633f695803387ea81d1727f70770953e3499787d2eb3ad073":"07d0",
        "9e35f2b103e0ec5be2c5d37376f9b921de7b8dbbdfbc1ef0bb9ec5ad07059ed7":"1f40"
      },
      "code":"0x60606040526000357c01000000000000000000000000000000000000000000000000000000009004806390b98a1114610044578063bbd39ac01461007157610042565b005b61005b6004803590602001803590602001506100b3565b6040518082815260200191505060405180910390f35b610082600480359060200150610098565b6040518082815260200191505060405180910390f35b60006000506020528060005260406000206000915090505481565b600081600060005060003373ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000206000505410156100f557600090506101e9565b81600060005060003373ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060008282825054039250508190555081600060005060008573ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000206000828282505401925050819055507f16cdf1707799c6655baac6e210f52b94b7cec08adcaf9ede7dfe8649da926146338484604051808473ffffffffffffffffffffffffffffffffffffffff1681526020018373ffffffffffffffffffffffffffffffffffffffff168152602001828152602001935050505060405180910390a1600190506101e9565b9291505056"
    },
    .....
  },
  "block_coinbase":"0xe6a7a1d47ff21b6321162aea7c6cb457d5476bca",
  "refunds":"0",
  "prev_headers":[
    {
      "hash":"0x91c90676cab257a59cd956d7cb0bceb9b1a71d79755c23c7277a0697ccfaf8c4",
      "timestamp":"1439799138",
      "gas_used":"0",
      "number":"100000",
      "difficulty":"3845539970657",
      "uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "gas_limit":"3141592"
    },
    {
      "hash":"0xfbafb4b7b6f6789338d15ff046f40dc608a42b1a33b093e109c6d7a36cd76f61",
      "timestamp":"1439799105",
      "gas_used":"0",
      "number":"99999",
      "difficulty":"3847418593016",
      "uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "gas_limit":"3141592"
    },
    {
      "hash":"0x576fd45e598c9f86835f50fe2c6e6d11df2d4c4b01f19e4241b7e793d852f9e4",
      "timestamp":"1439799094",
      "gas_used":"42000",
      "number":"99998",
      "difficulty":"3845540887505",
      "uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "gas_limit":"3141592"
    },
    .....
    {
      "hash":"0x116713528175f43bd957139f51e819fac969113a714273aebb0597e08ec3b72c",
      "timestamp":"1439794957",
      "gas_used":"0",
      "number":"99746",
      "difficulty":"3819457392639",
      "uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "gas_limit":"3141592"
    },
    {
      "hash":"0xafd82cb026c1ea57e105ec96042457740f4a3422a6c04343a2651d364b3b8d33",
      "timestamp":"1439794937",
      "gas_used":"0",
      "number":"99745",
      "difficulty":"3821323273143",
      "uncles_hash":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "gas_limit":"3141592"
    }
  ],
  "timestamp":"1439799138",
  "gas_used":"0",
  "txindex":"0",
  "block_difficulty":"3845539970657",
  "recent_uncles":{
    "99999":[],
    "99998":[],
    "99995":[],
    "99997":[],
    "99996":[],
    "100000":[]
  },
  "bloom":"0",
  "gas_limit":"3141592"
}

---

Here is an excerpt of a Parity state snapshot:

{
  "state":{
    "0x00000000000000000000000000000000000125ca":{
      "balance":"0x",
      "nonce":"0x"
    },
    "0x0000000000000000000000000000000000024bec":{
      "balance":"0x",
      "nonce":"0x"
    },
    "0x1067dcb77e3390563819a94907adb53a402dd707":{
      "balance":"0x260407a553800",
      "nonce":"0x2"
    },
    "0x843fd22c88d59e57ae1856a871a5d95e95b0a656":{
      "balance":"0x2fbf9bd9c800",
      "nonce":"0x1"
    },
    "0x580886a6fc371d9485933fafbac4ac24e7927315":{
      "balance":"0x",
      "nonce":"0x",
      "code_hash":"0x4092c8a5fb6487e931a5679ab058788c99bdb84fb49fdbf8cd5dc131e8a2f19f",
      "code":"606060405236156100405760e060020a600035046335faa416811461004d57806355a373d61461006e5780638da5cb5b14610080578063b69ef8a814610092575b6101026000610107610096565b61017760008054819033600160a060020a039081169116146101c857610002565b610190600154600160a060020a031681565b610190600054600160a060020a031681565b6101ad5b6000600160009054906101000a9004600160a060020a0316600160a060020a03166370a08231306040518260e060020a0281526004018082600160a060020a031681526020019150506020604051808303816000876161da5a03f1156100025750506040515191505090565b005b50565b604080516001546000805460e060020a63a9059cbb028452600160a060020a03908116600485015260248401869052935194955092169263a9059cbb926044808401936020939290839003909101908290876161da5a03f115610002575050604051511515905061010457610002565b6040805192835260208301919091528051918290030190f35b60408051600160a060020a03929092168252519081900360200190f35b60408051918252519081900360200190f35b600091505b9091565b6101d0610096565b600154600080546040805160e060020a63a9059cbb028152600160a060020a03928316600482015260248101869052905194955092169263a9059cbb9260448181019360209392839003909101908290876161da5a03f115610002575050604051511590506101bf57600191506101c456",
      "storage_root":"0x1fc09808178a80141b792a6263da8ec0525c369c31052f23dbf390b8b60fa2b1",
      "storage":{
        "0x0000000000000000000000000000000000000000000000000000000000000000":"0x0000000000000000000000008b3b3b624c3c0397d3da8fd861512393d51dcbac",
        "0x0000000000000000000000000000000000000000000000000000000000000001":"0x000000000000000000000000bb9bc244d798123fde783fcc1c72d3bb8c189413"
      }
    },
    "0xa1313a80575477e2662a59fbb98372efcd65bf4c":{
      "balance":"0x",
      "nonce":"0x1"
    },
    "0x1ae3d6240e6e158b65c3057d1020355c14c92178":{
      "balance":"0x",
      "nonce":"0x",
      "code_hash":"0x4092c8a5fb6487e931a5679ab058788c99bdb84fb49fdbf8cd5dc131e8a2f19f",
      "code":"606060405236156100405760e060020a600035046335faa416811461004d57806355a373d61461006e5780638da5cb5b14610080578063b69ef8a814610092575b6101026000610107610096565b61017760008054819033600160a060020a039081169116146101c857610002565b610190600154600160a060020a031681565b610190600054600160a060020a031681565b6101ad5b6000600160009054906101000a9004600160a060020a0316600160a060020a03166370a08231306040518260e060020a0281526004018082600160a060020a031681526020019150506020604051808303816000876161da5a03f1156100025750506040515191505090565b005b50565b604080516001546000805460e060020a63a9059cbb028452600160a060020a03908116600485015260248401869052935194955092169263a9059cbb926044808401936020939290839003909101908290876161da5a03f115610002575050604051511515905061010457610002565b6040805192835260208301919091528051918290030190f35b60408051600160a060020a03929092168252519081900360200190f35b60408051918252519081900360200190f35b600091505b9091565b6101d0610096565b600154600080546040805160e060020a63a9059cbb028152600160a060020a03928316600482015260248101869052905194955092169263a9059cbb9260448181019360209392839003909101908290876161da5a03f115610002575050604051511590506101bf57600191506101c456",
      "storage_root":"0x1fc09808178a80141b792a6263da8ec0525c369c31052f23dbf390b8b60fa2b1",
      "storage":{
        "0x0000000000000000000000000000000000000000000000000000000000000000":"0x0000000000000000000000008b3b3b624c3c0397d3da8fd861512393d51dcbac",
        "0x0000000000000000000000000000000000000000000000000000000000000001":"0x000000000000000000000000bb9bc244d798123fde783fcc1c72d3bb8c189413"
      }
    },
    "0x2c5bf6a0156e6f19671007cd6ce32ecd6111f2d1":{
      "balance":"0x6fede29ef3000",
      "nonce":"0x1"
    },
    "0xe6115b13f9795f7e956502d5074567dab945ce6b":{
      "balance":"0x152d02c7e14af6800000",
      "nonce":"0x"
    },
    "0x799c8f030135d3a0d9f804266636aa5e33c5608c":{
      "balance":"0x6fede29ef3000",
      "nonce":"0x1"
    },
    "0xa8741ea717952dcd7512ff652e40d02c609f7caf":{
      "balance":"0xfda435a8",
      "nonce":"0x7"
    },
    "0x92f14b5b51d3ef911fe81651d7bf04d8a494f65e":{
      "balance":"0x",
      "nonce":"0x",
      "code_hash":"0x9b0854b07c8f3f67ddd0f3e9b3b02b8879a6e9aa6ed2f7ba6a987e060cce8d04",
      "code":"366000803760206000366000736ab9dd83108698b9ca8d03af3c7eb91c0e54c3fc60325a03f41560015760206000f3",
      "storage_root":"0x55366cfd2df759d08d18aea43cc5482b7faec467adca2ee15866ec5d8aea430c",
      "storage":{
        "0x0000000000000000000000000000000000000000000000000000000000000005":"0x000000000000000000000000c0ce22e65c5789158213f5c327d1f2c5d5c761ba",
        "0x0000000000000000000000000000000000000000000000000000000000000000":"0x0000000000000000000000000000000000000000000000000000000000000002",
        "0xd5d9831dc909e45ff4b2c233e9d38b35fce7558e94e42fcc467d6bace510e3bf":"0x0000000000000000000000000000000000000000000000000000000000000001",
        "0x0000000000000000000000000000000000000000000000000000000000000107":"0x0000000000000000000000000000000000000000000000000000000000004285",
        "0xb78bc6bac7fd8d4c3579ece970ea32c61872375a674a318a507376a1bfeda2df":"0x0000000000000000000000000000000000000000000000000000000000000003",
        "0x0000000000000000000000000000000000000000000000000000000000000004":"0x000000000000000000000000d854f48a37551e0849db7757cacabb3dac1e3e00",
        "0x0000000000000000000000000000000000000000000000000000000000000001":"0x0000000000000000000000000000000000000000000000000000000000000003",
        "0x0000000000000000000000000000000000000000000000000000000000000003":"0x000000000000000000000000711a8720b458700cc3512e9950c18d745b41dac9",
        "0xbd860fee2a0cbe10878863255e3c707461110dbe60e529014acc41a81aa71236":"0x0000000000000000000000000000000000000000000000000000000000000002"
      }
    },
    "0xcd6504391c4e755ccbaf235c02209f7ce2635e57":{
      "balance":"0x17d4",
      "nonce":"0x2"
    },
    "0x653b9df2568d8b3009467a570f1e54ba06150abf":{
      "balance":"0x1a84893680a8a",
      "nonce":"0x16"
    },
    "0x11669eb622d14f726dfa8d78e5553ad1c57b1873":{
      "balance":"0x470de4df820001",
      "nonce":"0x"
    },
    "0x97da73e5cb1da305b4c160e212da9a4ddcb4510e":{
      "balance":"0xb1a2bc2ec50000",
      "nonce":"0x"
    },
    .....
  }
}

[danfinlay]

Thanks for pushing this forward, @cdetrio.

Since a state snapshot can have many different uses, I think the best strategy is to have a format that is as extensible as possible, while permitting small, simple snapshots as well.

I would recommend making prev_headers and recent_uncles, and these other fields to be optional, so that applications that require them can benefit from them, while not requiring implementation from non-participating clients.

An example where optional fields would shine is in an RPC test suite (;)), where some RPC features might involve uncles, while others might only refer to the current state. Defining the blockchain state for such tests would ideally be as concise as possible, for readability, and reducing side effects.

[cdetrio]

For background, pyethereum's JSON state snapshot feature was documented here, and Parity's was mentioned in the v1.5.0 release notes.

[janx]

I want to suggest another aspect: memory friendly.

Snapshot export for pyethapp constructs the whole json in memory, which comsumes more than serveral GBs memory if there're millions of accounts in state. This problem could be easily avoided if the snapshot format supports efficient update.

[rphmeier]

While the JSON snapshots are useful for performing analysis of the state and perhaps hand-editing tests, JSON is definitely not a compact enough representation for something as large as Ethereum's state. In terms of use-cases where file size and avoiding data redundancy is desired, neither of the JSON formats described here is really suitable. Instead, something like Parity's RLP state snapshots should be used.

Although Parity supports JSON export, the state snapshot format for Parity Warp Sync (also implemented in EthereumJ) is fully documented at https://github.com/paritytech/parity/wiki/Warp-Sync-Snapshot-Format, never re-includes duplicate code, and is designed to allow out-of-order restoration of chunks. It uses RLP, which is already a core part of most Ethereum clients and is the de facto standard for serialization within this space.

Parity includes no header data

With the JSON state export (it is called state export), sure, but RLP snapshots can include header, transaction, uncle, and receipt data for a configurable amount recent blocks (albeit also in compressed form).

Reading and restoring from this data should be no more difficult than doing so with JSON, provided appropriate libraries. Doing analysis on this data shouldn't be much worse, either.

[rphmeier]

That said, it is important to note that both proposed formats here store trie key pre-images w.r.t. accounts and contract storage, which is definitely more human readable, although it does impose a much storage overhead on the node to keep track of those pre-images in order to create snapshots later.

Should state snapshots cover intermediate states?

Doubly-so here: having snapshots cover intermediate states requires nodes to actually store intermediate states, and there already isn't much reason to. With EIP98 implemented, intermediate state snapshots would be much more annoying to verify than currently.

[janx]

How about a combination of the two? Metadata in a json file and accounts data in another RLP file.

[github-actions]

There has been no activity on this issue for two months. It will be closed in a week if no further activity occurs. If you would like to move this EIP forward, please respond to any outstanding feedback or add a comment indicating that you have addressed all required feedback and are ready for a review.

[github-actions]

This issue was closed due to inactivity. If you are still pursuing it, feel free to reopen it and respond to any feedback or request a review in a comment.