Design: IPLD Explorer

[daviddias]

An IPLD explorer could be hugely useful for letting people play with IPLD and IPFS to build an intuition of how it fits together.

WebUI has a DAG (IPLD) explorer. It takes an IPFS hash (also due a rename to CID) and lists out all the links or data that in the object the CID points to. In the case of a directory, you can see a list of the files it contains, and each filename is a link to it's IPLD object, allowing you drill down, navigating through the links. Where a hash points to data, the raw data is rendered in plain text.

The UI/UX should be improved but the idea is interesting, and could become a vital tool for helping people to understand IPFS and IPLD.

Project description: https://github.com/protocol/pm-ipfs-gui/blob/master/proj-descriptions/IPLD_EXPLORER.md

The current state of things is described here: https://github.com/ipfs-shipyard/pm-ipfs-gui/blob/master/research/README.md#dag--ipld-explorer

[akrych]

I think we can agree here that it should be only in WebUI :)

[hacdias]

Yup @akrych

[olizilla]

Yep, I think so too, but that depends on what we think each app should be responsible for... #41

[lidel]

Something worth keeping in mind while designing interface for exploring this is that we have two types of low level graphs.

trickle-dag [TD] is a merkle-dag [MD] in the definition sense but when we think about merkle-dag we default to thinking about balanced tree DAG. This isn't the case in with trickle-dag that has other structure (list-of-lists). – ipfs-inactive/faq#218 (comment)

Examples of the same content represented as MD and TD below:

"IPLD Inspector", merkle-dag, balanced tree

Sample resource: /ipfs/QmcPZGXSyaQSnDVEtJrgtp2EyxhENiBdpGa8pfs5rMBCWr (Inspect via: OldWebUI / IPLD Explorer)
Screenshot of Old WebUI:

"IPLD Inspector", trickle-dag, list of lists

Sample resource:/ipfs/QmZ5tQsSSdKS5Lez75KDfNxZ4usEYjHML5RBpPM6Hn2Z4n (Inspect via Old WebUI / IPLD Explorer)

Screenshot of Old WebUI:

---

tl;dr we can't assume DAG is always a balanced tree

May not be that important for non-graphical representations (such as the one in current WebUI), but I think it is worth noting and providing relevant samples (above) in case we start experimenting with drawing trees at some point :)

[olizilla]

Very much agree. The IPLD Explorer needs to deal with arbitrary graphs. Also, as came up in irc today (webui creating a TD graph by default but not informing the user), we need to be super clear to the user when adding files what sort of graph we're building for them under the hood.

[lidel]

FYI I've updated David's doc with some additional notes on ipfs/dataviz prior art with D3 and GraphViz:

See: IPLD_EXPLORER.md#explorer-4-graphical-version

"IPLD Graph Explorer"

A screenshot of prior art as an inspiration:

[olizilla]

https://www.hyperledger.org/projects/explorer is a web app / dashboard for navigating around hyperledger based blockchains...

https://www.youtube.com/watch?v=mI6-qRagn_Q

I don't think our IPLD explorer should look anything like that, but it's a relevant data point.

[olizilla]

Graph db explorers seem like a useful reference point...
http://www.popotojs.com/ & https://vimeo.com/107537564 is a interesting demo of the "connected blobs" style of graph visualisation.

These full-on, wobbly animated, self-reorganising blob visualisation, often fall more on the side of "fun eye-candy demo" than on the "useful tool" side of things. I think you've got to have an intuitive way of querying the graph and seeing the change to the tree it produces for it to move more to the useful tool side of the spectrum

https://linkurio.us/

http://www.popotojs.com/examples.html

Right now, the WebUI offers a simpler "Here are the IPLD links in the node for that CID", and clicking on them takes you through to the same page, but now showing the next set of links

Explore a cid	Drill into a data node

[olizilla]

This is demo of the https://github.com/ipfs/dataviz
https://v04x.ipfs.io/ipfs/QmX5smVTZfF8p1VC8Y3VtjGqjvDVPWvyBk24JgvnMwHtjC/viz#QmX5smVTZfF8p1VC8Y3VtjGqjvDVPWvyBk24JgvnMwHtjC

Right now it only shows files and offers no mechanism for querying, so it's most useful as a demo of what's possible right now. Our explorer needs to offer the user a way to comprehend the data presented, and be able to show info about links to things that aren't files. Where an IPLD node links to a thing, we can figure out some info about what we might find if we go and fetch the data that the CID link points to... we can unpack the multicodec which'll tell us that the data we'd get is one of these formats: https://github.com/multiformats/multicodec/blob/master/table.csv

This CID will become more useful in time once CID v1 are rolled out, but for vast majority of existing v0 CIDs it'll just tell us that it's dag-pb so we need other ways to give the user a clue as to what sort of data the hash links to. If we already have the all the data for the hash in our local repo, we can try to mime-sniff enough of it to have a good guess as to what sort of file it is.

The main point is we'll have to deal with "opaque" v0 CIDs that point to data we don't have yet, and are fetched on demand, as well as CIDs that we can be more informative about as we have the data, and also fancy new v1 CIDs that give a clue as to what to expect from the data behind the hash even if you haven't fetched it yet.

[daviddias]

The IPLD Team should also be involved in the design of the Dag explorer as they might have to polish some of the APIs (or might be changing the existing ones). See their OKRs at ipld/ipld#31.

[olizilla]

We can show the user a compact version of the information contained in the CID itself. We can show that immediately while the ipfs request to fetch the data is in progress.

See: http://cid-utils.ipfs.team/#QmYPNmahJAvkMTU6tDx5zvhEkoLzEFeTDz6azDCSNqzKkW

We could limit it to the human readable version of the CID. It'd be great to show the relationship between the sections of original hash and the unpacked human readable version.

[olizilla]

@vmx @pgte have you ever envisaged what a really good IPLD explorer app might look like? What concepts must it explain to do a good job of helping new users understand IPLD? What actions should it allow. We're using this issue to gather ideas on the rebuild of the DAG page on ipfs webui that currently ships with go-ipfs.

IPLD is new to everyone, so we're pulling together screenshots of anything related to help @akrych and the rest of us pin down what a really useful IPLD explorer would look like and offer. Any thoughts you have on the subject would be greatly appreciated.

[olizilla]

@alanshaw made a command line tool for exploring IPLD graphs

https://github.com/tableflip/ipld-explorer-cli

[olizilla]

The multihash repo has a good visual primer https://github.com/multiformats/multihash#visual-examples we should use a similar mechanism to visually associate sections of a CID with the human readable version.

[lidel]

A neat color-coded legend from that primer:

[olizilla]

@akrych this is a nice intro to trees and graphs https://medium.com/basecs/a-gentle-introduction-to-graph-theory-77969829ead8

We say "graph" alot but we're building a "tree " explorer. In this world a tree is just a graph with some special properties. IPLD graphs will be directed: the connecting lines can only point in 1 direction, from the root towards the leaf. A node (or circle in the diagrams can only point to it's children, the circles below it).

Not only do child nodes not know about their direct parents, but they don't know about any of their ancestors. That means IPLD graphs will never contain loop or a "cycle"... which in graph world is called acyclic. So IPLD graphs are "Directed (no backsies) Acyclic (no loops) Graphs" or DAGs for short.

Directed Acyclic Graphs look a bit like simple tree drawings, and we tend to draw graphs like that with the root node at the top, and all the children beneath it, as per the diagram above.

A CID (ipfs hash) like /ipfs/QmHash is the address for a root node in an IPLD graph. It indicates a place to start your explorations from. When you add a directory of files to your IPFS repo, each file gets a CID. Those CIDs are the addresses of the child nodes in our tree diagram when looked at an in IPLD explorer. The directory is processed last of all, and it gets a CID that is the address of our IPLD root node, and it points to all the files in the directory. In the IPLD graph, the directory node will contain links that point to all the files nodes, but the file nodes won't contain any links to the directory. A CID address points to a node in the IPLD graph, and each node can contain a list of links that are the CID addresses of it's child nodes, if it has any.

Large files get broken up in to chunks, so an IPLD node that represents a single file, like a video, may be a root node for a long list of links to chunks of video, which is one way we can end up with having many layers in the tree.

IPLD provides the tools to create any tree shaped graph. All the graphs will be "directed and acyclic", but they won't all come from directories of files.

[olizilla]

From conversations today, we need to be careful to show the right amount of information in the ipld graph view by default... where we have 174 links we don't need to show all 174 to give the idea. We offer paging by clicking on the "more here" and an expanded view, for those times you really really want to see them all

[vmx]

@olizilla I don't know how that could be applied to tries, but showing only a subset reminded me of https://leaflet.github.io/Leaflet.markercluster/example/marker-clustering-realworld.388.html

[olizilla]

I've been talking about the concept of a mini-map for our graph... so we have the 1 or 2 previous graph layers in view in the interactive graph, and then a mini-map to give the user a sense of the overall graph shape, and to give context to how far in and what path you've taken into the graph so far, a bit like the old mini-map from sublime for helping you spot shapes in the code and jump to different sections but with graphs ✨📈✨

[olizilla]

@akrych for graph node types we can start with the list of IPLD formats from here, line 407 onwards https://github.com/multiformats/multicodec/blob/888cc045de9ea668ec2b098c00173a5eb886b0f8/table.csv#L406

We can group related ones together with a common icon / colour which gives us an initial list of

• raw
• dag-pb
• dag-cbor
• git-raw
• eth-*
• bitcoin-*
• zcash-*
• stellar-*
• torrent-*
• ed25519-pub

As a design hint, dag-pb and dag-cbor nodes should get a similar colour or icon and be similar to ipfs styles as they are formats that originate from the ipfs/ipld team, while the others (git, eth, zcash, stellar, torrents) represent data structures from other tech teams with their own branding / colours.

Of note, we can't tell if a link is pointing to a node that represents a file or a directory (or a shard of a large file, or a shard of a directory with more than 174 files) until we go and fetch it. So we might show a node as being dag-pb for a while and then once we fetch it, we can add more info to show that it is a dag-pd node that represents a file or part of a file... that would give us an extra 4 combos to consider

• dag-pb unixfs file
• dag-pb unixfs file shard
• dag-pb unixfs directory
• dag-pb unixfs directory shard

and in the future dag-cb nodes may be used to represent files and directories, so we should be prepared for nodes like

• dag-cbor unixfs file
• dag-cbor unixfs file shard
• dag-cbor unixfs directory
• dag-cbor unixfs directory shard

this points to us needing a design that lets's us easily mix and match the states dag-pb vs dag-cbor and file vs directory, and complete vs shard, as we'll often get nodes of any combination of those three concepts. We could simplify it by keeping the "file vs dir" and "shard vs complete" ideas in just as labels in the data area and not add them to the graph at all, and stick with just using icons / colors for their multicodec type from the initial list.

[olizilla]

@akrych We can break a CID down into

CID info e.g.

• base: base58btc
• version:1
• codec: raw

Multihash info e.g.

• hash function: sha2-256
• size: 256 bits
• hash value: 30fbdf640d1a8cfe6bca34571b44a19aa5de97470a7bc84901c9d908ce3ccb02

I've worked up a simple version of the multihash visual from the slides https://github.com/multiformats/multihash#visual-examples

Common sha2-256 flavour

Rare but possible blake2b-512 flavour

[Kubuxu]

@olizilla as a note, under the MULITHASH header, I found mixing of decimal and hex representations a bit confusing.

[olizilla]

@Kubuxu thanks for the heads up, that wasn't intentional. Fixed now:

[olizilla]

IPLD Explorer page progress

[olizilla]

Introducing: Graph Crumb!

It shows the original source node and the path you've drilled down into to get to the node you're looking at, and politely indicates the node containment and path boundaries along the way.

[olizilla]

Notes...

https://databricks.com/blog/2015/06/22/understanding-your-spark-application-through-visualization.html

DAGViz: A DAG Visualization Tool for Analyzing Task-Parallel Program Traces
http://ccl.cse.nd.edu/research/papers/dagviz-vpa-2015.pdf

[olizilla]

https://www.researchgate.net/publication/5878342_Exploring_Multiple_Trees_through_DAG_Representations

[vmx]

@olizilla I stumbled upon https://www.graphviz.org/Documentation/TSE93.pdf in case you want to get get deep into the algorithmic side of drawing DAGs.

[olizilla]

News just in! ETH IN IPLD EXPLORER!

[hacdias]

That's amazing Oli!

[vmx]

Another nice visualization that could work well with: DAGs https://observablehq.com/@nitaku/tangled-tree-visualization-ii Thanks @rvagg for the link!

[jessicaschilling]

Closing this issue, but referencing it in ipld/explore.ipld.io#55 -- effort to add context to IPLD Explorer both for less experienced users and for cross-linking to other tools and docs materials.