Client-side Javascript DAG visualization / explorer

[harlantwood]

It seems like it would be immensely cool to have a visualization of data stored in ipfs.

[Edit: To clarify, what I'm imagining is an interactive node/link visualization/browser running in a JavaScript app, either in the webui or a separate application.]

1. Does this exist already somewhere? If not, is the webui the right place to add it? I think this may ultimately want to be it's own application.
2. What kind of visualization? I lean toward graph viz (nodes and edges). Other good options include tree-shaped viz.
3. Which JS viz framework? My top 3 picks are:

• http://d3js.org/
• http://sigmajs.org/
• http://famo.us/

If we were doing graph-shaped viz, I'd lean toward Sigma, because it creates beautiful graphs pretty much out of the box, and renders to webgl (when available) for use with large data sets.

If I get time I'll hack on it a bit; meanwhile any feedback very welcome.

[whyrusleeping]

https://github.com/ipfs/go-ipfs/blob/master/bin/graphmd

[harlantwood]

@whyrusleeping I think this script is awesome. To clarify, what I'm thinking of is kind of an extension of this functionality -- an interactive node/link visualization/browser running in JavaScript.

If this was running in the webui, or in a similar application, where would we imagine it getting data from? From the node api? Still figuring out the moving parts around here...

[cryptix]

This is perfect! I think the webui needs dag viewer, if only a basic one - especially once more exotic data formats start to emerge (we already have the geoip stuff, append only logs anyone?).

The webui gets its data through node-ipfs-api.

For the standalone case, you might like to experiment with node-ipfsd-ctl which can spin up the go implementation and passes you ipfs-api clients (temporary right now, persisted by passing a dir just needs to be documented iirc? cc @krl)

[harlantwood]

TL;DR: After really a lot of research, I recommend a D3 expandable tree layout to visualize the IPFS DAG data. Additional data would be fetched either on each tree node expansion, or (preferably IMO) pre-fetched, at least for a reasonable number of next possible tree nodes.

For those who may be interested, some of my research notes follow.

Force Directed Graphs

I was hoping for a graph-shaped solution, rather than tree-shaped, and Force Directed Graphs are the best way I've seen to automatically arrange a graph in a readable layout.

D3 Force Directed Graphs

http://bl.ocks.org/MoritzStefaner/1377729 | Force-based label placement
http://bl.ocks.org/mbostock/1129492 | Bounded Force Layout
http://bl.ocks.org/christophermanning/1625629 | Chicago Lobbyists Force-Directed Graph Visualization
http://bl.ocks.org/mbostock/4600693 | Curved Links
http://bl.ocks.org/couchand/6420534 | Better force layout node selection
http://xliberation.com/googlecharts/d3nodefocustagsite.html | d3.js from Excel - ramblings.mcpher.com
https://elegans.herokuapp.com/# | Elegans
http://www.cograph.co/wjzeng/document/1916 | Cograph

Arbor.js Force Directed Graphs

http://arborjs.org/halfviz/#/date-with-destiny
http://nooshu.com/explore/arbor-visual-thesaurus/

Springy.js Force Directed Graphs

http://getspringy.com/

D3 Hand Curated Graph

http://www.nytimes.com/interactive/2013/02/20/movies/among-the-oscar-contenders-a-host-of-connections.html?_r=0 | Among the Oscar Contenders, a Host of Connections - Interactive Feature - NYTimes.com
http://bost.ocks.org/mike/example/ | Explains the creation of the Oscar graph above

Sigma.js Graphs

http://wikiproject.oii.ox.ac.uk/networks/ | Co-editing among Wikipedians in English
http://neosigma.herokuapp.com/index.html | Neo4j and Sigma.js Example

D3 Trees

http://bl.ocks.org/mbostock/4063530 | Circle Packing
http://mbostock.github.io/d3/talk/20111018/treemap.html
http://mbostock.github.io/d3/talk/20111018/tree.html

[harlantwood]

Thanks for the info @cryptix 😄

[harlantwood]

@jbenet @cryptix @whyrusleeping I am interested in what you think of something like http://mbostock.github.io/d3/talk/20111018/tree.html to visualize IPFS DAGs before going much further.

[whyrusleeping]

@harlantwood that expanding tree is awesome. i would love that!

[jbenet]

Sorry for delay will take a look in a bit

—
Sent from Mailbox

On Thu, May 28, 2015 at 1:16 PM, Jeromy Johnson notifications@github.com
wrote:

@harlantwood that expanding tree is awesome. i would love that!

Reply to this email directly or view it on GitHub:
#56 (comment)

[harlantwood]

Cool, glad you like @whyrusleeping. Just had a good call with @fractastical of Swarm... Once this and other data visualizations of IPFS are up and running, it would be awesome to start reflecting the blockchains in from Swarm, Etherium, Counterparty, Bitcoin, etc... IMO IPFS could be an ideal permanent place to store and visualize all this immutable data.

[jbenet]

@harlantwood very much agreed-- ok looking now. sorry for delay

[jbenet]

TL;DR: After really a lot of research, I recommend a D3 expandable tree layout to visualize the IPFS DAG data. Additional data would be fetched either on each tree node expansion, or (preferably IMO) pre-fetched, at least for a reasonable number of next possible tree nodes.

After looking at all of those, and remembering some myself, Entirely agreed. i think "expandable tree layout" is one of the best out there today.

btw @harlantwood also if you haven't seen these you'll love them:

• http://vcg.informatik.uni-rostock.de/~hs162/treeposter/poster.html
• http://keshif.me/demo/treevis.html

Regardless, i think we should have an intermediate format, between merkledag and these visualizations, that makes it easy to plug into all the graphs. So we could visualize the same data in a bunch of different graphs.

Basically something like:

ipfs export <root> | graphmd --format json >output.json

• this export doesnt exist, but we could use the car format we're making portentially, or just some concatenated ipfs object get output.
• the graphmd tool today doesn't do this, it only uses dot syntax, but we can extend it to do some predefined json format for ease of use.
• then we can just have a script that generates a webpage from the data

> mkdir foo
> mv output.json foo/.
> graph-md-site foo
> ls foo
index.html
js/
css/
output.json
> ipfs add -r -q foo | tail -n0
<hash>
> http://gateway.ipfs.io/ipfs/<hash>  # view it with different viewers.

How does this sound @harlantwood @krl ?

[harlantwood]

@jbenet this is very cool. Let's call what you have described here [option 1].

What I love most about [option 1] is that we are serving visualizations of IPFS through IPFS itself, as static HTML/JS.

Proposing an [option 2]:

Background: the IPFS project has this fantastically cool JS example visualizer which can load in examples from other IPFS nodes:

• http://gateway.ipfs.io/ipfs/QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D/example#/ipfs/QmTDMoVqvyBkNMRhzvukTDznntByUNDwyNdSfV8dZ3VKRC/readme.md
• http://gateway.ipfs.io/ipfs/QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D/example#/ipfs/QmThrNbvLj7afQZhxH72m5Nn1qiVn3eMKWFYV49Zp2mv9B/git/readme.md

Sooo... we could create a visualizer in the same way, which could be accessed at http://gateway.ipfs.io/ipfs/[HASH_OF_VISUALIZER]/visualize#/ipfs/[HASH_OF_TREE_TO_VISUALIZE]

The cool thing is that once the visualizer was up and working, you could plug any IPFS directory in as the HASH_OF_TREE_TO_VISUALIZE -- so only updates to the visualizer need be published to IPFS, instead of having to publish for each directory you want to visualize (as in [option 1]).

Now to how it would work: (caveat: I am new to IPFS) --

[option 2a]

Right now the IPFS gateway serves up HTML for any directory object, including links like:

  <li><a href="/ipfs/QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D/home">home</a> - 3947 bytes</li>
  <li><a href="/ipfs/QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D/lib">lib</a> - 268261 bytes</li>

A naive approach would be to construct a 1-level tree from this directory info, and either prefetch or fetch upon click the next level of the tree, by loading and interpreting the objects/directories linked to.

[option 2b]

We could add functionality to the IPFS gateway to serve directories either as HTML or as JSON, depending on the Accept content type request header. That is, keep serving the HTML index page by default, unless the HTTP request asks for JSON, which the viz JS app would. We can do this easily from the client side, example here.

This JSON response could be depth 1 by default, and (perhaps) have an optional depth parameter that you could pass as a query string, eg ?depth=3.

Interested in your thoughts on options 1 vs 2, and 2a vs 2b -- @whyrusleeping @jbenet @cryptix @krl.

[harlantwood]

After discussing briefly at the Sprint meeting today, I'll pursue [option 2] (hopefully 2b, though I may need help on the go gateway side for that).

In any case, this no longer looks like part of the webui repo, though the webui could certainly link to it / use it.

My suggestion: make a new repo using a lot of the same ideas as ipfs/examples -- perhaps ipfs/dataviz.

If this seems clearly awesome, go ahead and make the repo... if not sure, I'll just start it in my account and we can push it to the IPFS org when its working.

[jbenet]

@harlantwood

We could add functionality to the IPFS gateway to serve directories either as HTML or as JSON, depending on the Accept content type request header.

i want to expose the Read part of the API's object/get calls on the default gateway. that way js can grab raw objects :)

maybe there's a separate case where we want to allow things to plug transforms into the gateway that transform objects server side before serving them (based on header, query, or path)

This JSON response could be depth 1 by default, and (perhaps) have an optional depth parameter that you could pass as a query string, eg ?depth=3.

yeah, i think this is nice. this may make sense as a separate API route though. (and we can expose readonly api parts via the gateway).

After discussing briefly at the Sprint meeting today, I'll pursue [option 2](hopefully 2b, though I may need help on the go gateway side for that).

Sounds good! yeah we can make an issue in go-ipfs to "Expose read-only API via Gateway"

My suggestion: make a new repo using a lot of the same ideas as ipfs/examples -- perhaps ipfs/dataviz.

yeah for sure. done! https://github.com/ipfs/dataviz

[harlantwood]

Awesome. Hopefully will have some time to work on dataviz tomorrow.

btw @harlantwood also if you haven't seen these you'll love them...

I love them.

[harlantwood]

Closing this thread, for the latest see:

• DAG visualized as D3 tree ipfs-shipyard/dataviz#1
• Expose read-only API via Gateway kubo#1322