Eth blocks missing IPLD links

[olizilla]

When we transform Eth blocks to objects, it's links aren't expanded out to '/': Buffer style IPLD links, they are just properties that point to Buffers, so when they are run through the IPLD explorer, it can't show any of their linked nodes.

Currently:

If the IPLD stucture was the same as that suggested here in go-ipld-eth https://github.com/ipfs/go-ipld-eth/blob/4199044661a07667f13ec1ca4d228b3741b75b09/plugin/README.md#usage-and-examples

{
    "bloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "coinbase": "0x4bb96091ee9d802ed039c4d1a5f6216f90f81b01",
    "difficulty": 11966502474733,
    "extra": "0xd783010400844765746887676f312e352e31856c696e7578",
    "gaslimit": 3141592,
    "gasused": 21000,
    "mixdigest": "0x2565992ba4dbd7ab3bb08d1da34051ae1d90c79bc637a21aa2f51f6380bf5f6a",
    "nonce": "0xf7a14147c2320b2d",
    "number": 997522,
    "parent": {
        "/": "z43AaGF24mjRxbn7A13gec2PjF5XZ1WXXCyhKCyxzYVBcxp3JuG"
    },
    "receipts": {
        "/": "z44vkPhjt2DpRokuesTzi6BKDriQKFEwe4Pvm6HLAK3YWiHDzrR"
    },
    "root": {
        "/": "z45oqTRunK259j6Te1e3FsB27RJfDJop4XgbAbY39rwLmfoVWX4"
    },
    "time": 1455362245,
    "tx": {
        "/": "z443fKyLvyDQBBQRGMNnPb8oPhPerbdwUX2QsQCUKqte1hy4kwD"
    },
    "uncles": {
        "/": "z43c7o73GVAMgEbpaNnaruD3ZbF4T2bqHZgFfyWqCejibzvJk41"
    }
}

...where the props are expanded out to IPLD links, we get all the auto linking magic for free

[Mr0grog]

Seems like part of the problem is that the CBOR resolver returns an IPLD-friendly representation when you call its util.deserialize() (which calls the CBOR decoder, which is set up to replace CID-tagged bytestrings with the {"/": "<cid>"} structure)…

…but here in js-ipld-ethereum, the util.deserialize() function only returns the Ethereum-native representation:

js-ipld-ethereum/util/createUtil.js

Lines 6 to 8 in 0e4e0be

	function createUtil (multicodec, EthObjClass) {
	return {
	deserialize: asyncify((serialized) => new EthObjClass(serialized)),

…and the transformation to an IPLD-friendly representation only happens during path traversal when reading into the object. Actual mapping:

js-ipld-ethereum/eth-block/index.js

Lines 9 to 32 in 0e4e0be

	function mapFromEthObj (ethObj, options, callback) {
	const paths = []
	// external links
	paths.push({
	path: 'parent',
	value: { '/': cidFromHash('eth-block', ethObj.parentHash).toBaseEncodedString() }
	})
	paths.push({
	path: 'ommers',
	value: { '/': cidFromHash('eth-block-list', ethObj.uncleHash).toBaseEncodedString() }
	})
	paths.push({
	path: 'transactions',
	value: { '/': cidFromHash('eth-tx-trie', ethObj.transactionsTrie).toBaseEncodedString() }
	})
	paths.push({
	path: 'transactionReceipts',
	value: { '/': cidFromHash('eth-tx-receipt-trie', ethObj.receiptTrie).toBaseEncodedString() }
	})
	paths.push({
	path: 'state',
	value: { '/': cidFromHash('eth-state-trie', ethObj.stateRoot).toBaseEncodedString() }
	})

Where it’s called during traversal (resolve() → resolveFromEthObject() → mapFromEthObject()):

js-ipld-ethereum/util/createResolver.js

Lines 59 to 74 in 0e4e0be

	function resolve (binaryBlob, path, callback) {
	waterfall([
	(cb) => util.deserialize(binaryBlob, cb),
	(ethObj, cb) => resolveFromEthObject(ethObj, path, cb)
	], callback)
	}
	function resolveFromEthObject (ethObj, path, callback) {
	// root
	if (!path || path === '/') {
	const result = { value: ethObj, remainderPath: '' }
	return callback(null, result)
	}
	// check tree results
	mapFromEthObject(ethObj, {}, (err, paths) => {

So that means what you get with ipld.get(<CID of an Eth object>) is the Ethereum library’s representation of the object, not the set of paths the IPLD resolver will traverse for you to get values (big surprise to me!).

It seems like this library would have to be restructured in a pretty big way to change that so util.deserialize() gets you a transformed representation… OR IPLD Explorer is doing the wrong thing by calling ipld.get(cid,...) (which returns the result of util.deserialize()) and should instead be calling ipld.treeStream(cid,...) and then calling ipld.get(cid, path, ...) with each item from the tree stream to build up a representation.

[Mr0grog]

Looks like the Bitcoin and ZCash resolves both follow the same approach (util.deserialize() gets you the underlying library’s parsed representation, not a representation of the paths the IPLD resolver will let you traverse).

[Mr0grog]

(Also, apologies if any of the above was just restating the obvious for folks here. I was surprised that a JS ipld.get(cid) gets a very different sort of result from what a Go coreapi.DagAPI.Get(context, cidPath) gets you.)

[olizilla]

That is super useful info for me @Mr0grog !

[Mr0grog]

Minor update: it looks like the js-ipld-git resolver does not follow this pattern; the deserialized representation matches the paths you can traverse with it: https://github.com/ipld/js-ipld-git/blob/3dac2488dbb3aafe96561b913dd130ccc91384e0/src/util/commit.js#L37-L84

So the disconnect between those exists in the resolvers for Ethereum, Bitcoin, and ZCash, but not in PB, CBOR, and Git.

[vmx]

Thanks @olizilla and @Mr0grog for digging into this. It should indeed return a proper link. Please note that what "proper link" means is currently in flux. @mikeal come up with the idea of making the resolvers return JavaScript Objects and not JSON, i.e. it won't be {"somefield": {"/": "basencoded-string"}} but {"somefield": CIDObject}. I'll open corresponding issues, once at least one format switch to the new "using a CID" which can then follow lead.

Update: Reference to the CID discussions: ipld/ipld#44

[Mr0grog]

See also ipld/js-ipld#141, based on follow-on discussions about this on Friday, 2018-08-03.

[Mr0grog]

OK, to bring this around to concrete, actionable issues, IPLD Explorer is actually doing the right thing here, but there are several bugs in js-ipld and the various resolver modules:

• ipld.get(cid) returns the wrong thing (ipld.get() should never give the direct result of resolver.util.deserialize() js-ipld#142)
• js-ipld-ethereum (and other resolvers) return the wrong thing (or fail!) with resolver.resolve(cid, '/', callback) (Implementation of nested objects interface-ipld-format#34)
• Some questions about the format of links in particular, but I think this is less relevant since IPLD Explorer is already handling both styles we are discussing unifying (dag-json VS dag-cbor ipld#44)

@olizilla all that adds up to:

• You are technically doing the right thing; it doesn’t work because of bugs in js-ipld-ethereum and js-ipld
• You can wait for the first two issues above to get fixed or you can work around it for now by using ipld.treeStream(cid) to get a list of paths and ipld.get(cid, path) to resolve each of those paths and compose the object you want to display from that.

@vmx should we treat this issue as a tracking bug for this module’s side of ipld/interface-ipld-format#34 or should we close it?

[vmx]

I'm happy to leave this issue open until the IPLD side (most notable ipld/interface-ipld-format#34) is fixed.