Feature - expose state and chaindata via ipfs-http-api

[kumavis]

This is a recommendation for an MVP of Ethereum-IPFS integration by exposing all hash-referenced ethereum objects (state trie and chaindata) via the ipfs-http-api spec. Lookups are performed by hash, results are returned as binary with no metadata.

Requirements

http server that implements the "block.get" method of the ipfs-http-api
example uri:

http://localhost:5001/api/v0/block/get?arg=z43AaGF5tmkT9SEX6urrhwpEW5ZSaACY73Vw357ZXTsur2fR8BM

Request handling flow

1. expose route: http://localhost:5001/api/v0/block/get?arg=${contentId}

2. parse request url for contentId

3. parse contentId into { type, hash } via multiformats/rust-cid

4. lookup type by hash, return result as binary

where type is any of:

Type	Name	Code
eth-block	Ethereum Block (block header)	0x90
eth-block-list	Ethereum Block List (block ommers)	0x91
eth-tx-trie	Ethereum Transaction Trie Node	0x92
eth-tx	Ethereum Transaction	0x93
eth-tx-receipt-trie	Ethereum Transaction Receipt Trie Node	0x94
eth-tx-receipt	Ethereum Transaction Receipt	0x95
eth-state-trie	Ethereum State Trie Node	0x96
eth-storage-trie	Ethereum Contract Storage Trie Node	0x98

spec references

ipfs-http-api

spec repo: https://github.com/ipfs/http-api-spec
block.get spec (note spec doc has not been updated to use content-id (cid) instead of multihash)
https://github.com/ipfs/http-api-spec/blob/master/apiary.apib#L807-L921

content-id (CID)

multibase encoded ( see multibase )
format: <cid version><data type><multihash>
spec repo: https://github.com/ipld/cid
rust implementation: https://github.com/ipld/rust-cid

multibase

format: <varint-base-encoding-code><base-encoded-data>
spect repo: https://github.com/multiformats/multibase
cargo package: https://crates.io/crates/multibase

multihash

format: <varint hash function code><varint digest size in bytes><digest>
spec repo: https://github.com/multiformats/multihash
rust implementation: https://github.com/multiformats/rust-multihash
cargo package: https://crates.io/crates/multihash

[arkpar]

eth-tx-trie, eth-tx-receipt-trie, eth-tx-receipt would require additional database indexing

[kumavis]

fine to skip eth-tx-trie, eth-tx-receipt-trie, eth-tx-receipt
really only need eth-block, eth-tx, eth-state-trie, eth-storage-trie

[kumavis]

This has been awesome to work with, thank you

A couple notes:

• blocks should be block headers only, not block bodies
• please expose contract evm code by codeHash when multicodec format is raw-bin 0x55

[kumavis]

currently eth-storage-trie look ups are not supported #6043

[kumavis]

trying to lookup the state trie nodes for a contract's storage via the eth-storage-trie CID type fails with CID codec not supported

it does work with eth-state-trie

ethereum's IPLD resolvers differentiate between block->account and account->storage tries to help resolve the leaf types

cc @maciejhirsz

[kumavis]

also seems that while "raw" ipld type has a handler, https://github.com/paritytech/parity/blob/master/ipfs/src/route.rs#L65
codeHash 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470
encoded as zSZ9SCdE8YLTbDksiQAUZofsygev1f7JAZES18LvtrXYBE2rj
results in CID parsing failed%

UPDATE: the "raw" IPLD type was an issue with js incorrectly generating CIDs
multiformats/js-multicodec#16

[rmulhol]

What's the rationale for not including the block body as a type in the spec?

[kumavis]

@rmulhol All values returned by IPFS lookups must match their hash. Block hashes do not reference the block bodies, just the block headers. Block bodies aren't referenced by hash anywhere, or we could add another IPLD type for block-bodies.

IPFS is working on their selector spec, which would allow you to make queries over multiple hash-referenced objects. e.g. Get block by hash and all transactions + transaction receipts below them. It is not ready yet as it requires significant rewrites to IPFS internals to perform the query on the remote peer instead of walking the dag over the network.

[folsen]

@kumavis I don't really have enough context for the motivation here to be honest. Why would you want to store data on both network? The blocks are retrievable through the ethereum network, why copy data to the IPFS network as well?

@maciejhirsz What's the status of our work on this issue? You had a PR here, is there anything more we want to do here or can we close this issue?

[ghost]

Why would you want to store data on both network?

@folsen We prefer to understand that rather than storing data, what the IPFS/libp2p does, is making data available through its protocol.

While there are available ways in eth/63 to ask for state and storage trie nodes, it is needed for a client to be able to connect to a network of devp2p clients, preferably to ports 30303 or 30304, making it difficult to browser peers, which are usually constrained to ports 80 and 443. A network of extended js-ipfs clients connected in libp2p, sharing this information would boost the dynamics of sharing ethereum data.

And even with the current solution of opening in parity an IPFS RPC (as it is working today in 2018.06.16), you would find that the API is restricted to one trie node per request, making traversals for accounts a business of 6 to 8 requests, and even harder for getting the complete storage trie of a smart contract.

Bottom line: We feel that having tools to making available the key-value store of state+storage ethereum trie nodes on libp2p for browser consumption will be a major accelerator for the emergence of light clients running in such applications.

I am more than happy to be proved wrong, and find out that there is a very simple way to execute a call (RPC, or calling parity on a different thread while running) where I can get, from the running parity database a walk through the trie in n-levels from an arbitrary root. We can work a bridge from there and share our results 😄

Best, Herman

---

Edit: The call in eth/63 is called GetNodeDataMsg, code 0x0d.

Edit 2: Also, aware that there exists a parity export state command. Would be cool to have an addendum to get trie nodes in hex, with an arbitrary root and depth as arguments.

[folsen]

rather than storing data, what the IPFS/libp2p does, is making data available through its protocol

This I can agree with, and Parity has started some significant work towards making Parity nodes talk libp2p.

We feel that having tools to making available the key-value store of state+storage ethereum trie nodes on libp2p for browser consumption will be a major accelerator for the emergence of light clients running in such applications.

I'm not sure how this makes light clients better/more possible? A light client needs all headers and that's really the only thing it needs, if you had an ethereum node that spoke libp2p that's all you'd need to run a light client in a browser. Unless you're proposing some new light client protocol that uses more than just header data, then I'm not sure what IPFS gives you.

[adria0]

Closing the issue due to its stale state