feat: support DAG API #534

daviddias · 2017-03-09T16:13:52Z

Will need changes that come from ipfs/kubo#3771 (comment)

atfornes · 2017-05-11T12:07:07Z

@diasdavid, could you please share what have to be done to complete the DAG API?

you mentioned that different formats have to be supported at #549, is it related with https://github.com/ipld/interface-ipld-format implementations?

if the tasks are relatively easy I can try to help with this in the JavaScript part.

Thanks

daviddias · 2017-05-11T17:21:57Z

Right now, we just need these tasks to be done: ipfs/kubo#3771 (comment), essentially, the http-api needs to support what we need in order to complete the JS. Would you like to take a look at that? :)

atfornes · 2017-05-26T16:38:06Z

@diasdavid, I am afraid I am not familiar enough with Go language to help with the http-api. Is there something else I can do to help pushing forward the solution for this issue?

atfornes · 2017-05-29T12:56:09Z

@diasdavid, what about implementing it in js-ipfs http API? does it makes sense? https://github.com/ipfs/js-ipfs/tree/master/src/http-api

magik6k · 2017-08-12T19:40:46Z

So with fixes/additions in ipfs/kubo#4133 current status is that:

dag.put works on both sides both for dag-cbor and dag-pb
dag.get doesn't because:
- We need to resolve top level block in path, get it, resolve remaining path on JS side
- For that either ipfs resolve must be extended to return IPLD path remainder or something like ipfs dag resolve must be added
- We then need to block get the resolved block and parse it on JS side
- Then return subnode under whatever remainder path it was within a block.

@diasdavid Is my thinking here correct or do you think this can be solved in a simpler way?

magik6k · 2017-08-13T18:57:01Z

So I rebased this on current master and implemented get as as dag resolve+block get. Everything seems to work on both (go-ipfs / js-api) sides, test failures here appear to be related more to the tests itself or deeper IPLD stuff.

daviddias · 2017-08-15T12:35:10Z

src/dag/get.js

+          dagPB.resolver.resolve(blk, path, cb)
+        }
+      }
+    ], callback)


@magik6k this is an interesting approach. It misses any other format that is not dag-cbor and or dag-pb. I also believe that it won't pass every test currently present in https://github.com/ipfs/interface-ipfs-core/blob/master/src/dag.js, which is a requisite to get things merged in js-ipfs-api.

I had understood that you were working on implementing the dag api on go-ipfs, is there anything blocking you there from finishing the implementation?

go-ipfs side is done. Now that i think of it, is there any reason the code for put which was here earlier didn't just pass the data as json to go-ipfs? Passing json directly instead of serializing it here would allow to support every encoding go side supports without additional work here.

The same is the case for get with the difference that using go-ipfs dag get doesn't return remainderPath(I don't think there is any case where it wouldn't be empty). Does the value passed to callback in get implement any API or is it much like just like parsing json from ipfs dag get?

Passing json directly instead of serializing it here would allow to support every encoding go side supports without additional work here.

Using json for all the serialization has a big drawback described here: https://github.com/ipfs/interface-ipfs-core/issues/81#issuecomment-277271941

tl;dr; JSON doesn't support Buffers, which makes it really hard to distinguish a Buffer from a String, you can see that we do a cast to Buffer every single time in object get just because we know that Data in the protobuf is always binary. This is the tl;dr; but please do read the whole thread :)

The same is the case for get with the difference that using go-ipfs dag get doesn't return remainderPath(I don't think there is any case where it wouldn't be empty)

remainderPath should return any path that the node wasn't able to resolve either because it couldn't find more nodes or because it didn't have the right parser for it.

daviddias · 2017-08-15T12:35:39Z

src/dag/put.js

+      if (format === 'dag-pb') {
+        dagPB.util.serialize(dagNode, finalize)
+      }
+    }


Same drawbacks as my comment above https://github.com/ipfs/js-ipfs-api/pull/534/files#r133178508

koalalorenzo · 2018-02-10T17:08:14Z

Do we have any update on this? ETA? :)
Do you need any help for this PR?

vmx · 2018-02-12T11:39:59Z

@koalalorenzo I don't have an ETA for this. I currently working on getting the CLI to support the dag command (ipfs/js-ipfs#1212).

In regards to help: sure, any help is welcome. After a quick look at this PR, I think it shouldn't use the IPLD formats directly, but instead use the API from https://github.com/ipld/js-ipld. This way you needed care format specific things yourself, but have support for all.

@diasdavid Do you agree that using js-ipld would make sense?

pcardune · 2018-02-15T23:36:29Z

I got here looking for a way to put an IPLD serialized object into an IPFS node while that IPFS node was running as a daemon. I'm aware that this can now be done with ipfs dag put < somedata.json, but I'm looking for something that doesn't require spawning subprocesses.

I figured out how to use ipld-js to store the data while the IPFS node is not running: https://gist.github.com/pcardune/411c93eb4dd7d15a642b3750a4bd8cfa. But then you have to start the IPFS node to make that data available publicly.

I'm guessing that this PR would solve my problem, but since it's unclear if and when this PR is going to get merged, I'm wondering if there is another way to accomplish this?

cc @jaycenhorton

daviddias · 2018-02-16T11:48:41Z

@diasdavid Do you agree that using js-ipld would make sense?

I do :) That was my realization on #534 (comment), that we can use js-ipld and use the ipfsAPI.block as the block service that js-ipld uses. This way we don't have any serialization problems.

vmx · 2018-02-16T17:53:24Z

@koalalorenzo @pcardune @jaycenhorton I'm working on this now. I thought it's a piece of cake, but it turned out that it's harder than I thought (too many bugs and outdated code on the way). Though I'll have a look again next week.

vmx · 2018-02-22T00:24:27Z

A quick update: I run into more troubles than expected. I'll link the issues here in case someone wants to know more.

daviddias · 2018-02-22T07:12:43Z

@vmx AFAIU you won't get the nice dag API -- as in https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/DAG.md -- from go-ipfs http-api soon due to how things are wired inside go-ipfs (@Stebalien can confirm). You will need to implement the dag API by bringing js-ipld and mounting it on top of the Block API, making the resolve happen clientside.

vmx · 2018-02-22T12:54:51Z

@diasdavid ipfs/kubo#4728 (comment) sounds promising.

At least it should be possible to do it "manually" in JS land. But currently even that is not really possible. If you request <cid>/Data you will get an error and not the CID with Data as remainder path back. Just to be clear that totally makes sense, the JS implementations do the same for paths they don't know. So let's just keep the fingers crossed that there will be a js-ipld compatible representation of Protocol Buffer blocks in the future. Until then it will just not be supported by js-ipfs-api.

* Implement dag.get * Handle dag put response, enable CIDs for dag get and fix dag get request param

The js-ipld formats API changed, they now take a raw data block rather than an IPFS block.

There was a refactoring, now the usual way of doing a HTTP request is through SendOneFile.

vmx · 2018-04-03T13:53:20Z

Many people have worked on this. I've rebased the commits and added my own ones. This is ready for review, so @diasdavid and @magik6k please review it.

daviddias · 2018-04-03T13:58:43Z

src/dag/put.js

+      }
+      if (format === 'dag-pb') {
+        dagPB.util.serialize(dagNode, finalize)
+      }


What about all the other types?

I'd use js-ipld here

Later. This PR is on for almost a year.

The plan surely is to use js-ipld. But that's not straight forward as js-ipld currently only works with local repositories and not with remote ones (that's what I wanted to mention in the JS Dev call but couldn't remember).

Fair enough.

I'd import git here too as it's the one other (well) supported codec on go-ipfs side (and would be really handy to have for https://github.com/ipfs-shipyard/IGiS which can't use window.ipfs from api now)

Is it OK if I I open an issue for it and do it in a separate PR? That' also something newcomers can work on.

This PR won't let users resolve through multiple IPLD formats.

@diasdavid Sounds like a test for that is missing in interface-ipfs-core then.

It will work once js-ipld is used.

Summary of this thread: everything mentioned here is solved if js-ipld is used. Hence I propose merging this change as is currently is and open a new issue (I'll do that) to use js-ipld instead.

Sounds good my IPLD captain! 🚢

daviddias added the in progress label Mar 9, 2017

daviddias requested review from whyrusleeping and removed request for whyrusleeping March 9, 2017 16:14

daviddias assigned daviddias and whyrusleeping Mar 9, 2017

This was referenced Mar 10, 2017

DAG API ipfs/kubo#3771

Closed

feat(dag): basics (put, get) #520

Closed

daviddias added ready and removed in progress labels Apr 5, 2017

magik6k mentioned this pull request May 4, 2017

Implement dag.get #549

Closed

atfornes mentioned this pull request Jun 20, 2017

Feat/dag api #568

Merged

daviddias added backlog and removed ready labels Jun 20, 2017

This was referenced Jul 8, 2017

CIDs with dag-cbor formats as V0 addresses multiformats/js-cid#35

Closed

Support for /api/v0/dns/ #501

Closed

magik6k mentioned this pull request Aug 12, 2017

Improve dag API ipfs/kubo#4133

Merged

magik6k force-pushed the feat/dag-api branch 2 times, most recently from 8e6d61b to 1caafb8 Compare August 13, 2017 18:51

daviddias commented Aug 15, 2017

View reviewed changes

daviddias mentioned this pull request Aug 23, 2017

🦑 IPLD deep dive calls ⚓️ ipfs/team-mgmt#484

Closed

daviddias changed the title ~~feat: DAG API~~ feat: support DAG API Oct 9, 2017

daviddias mentioned this pull request Oct 9, 2017

add DAG api ? #600

Closed

daviddias removed the backlog label Oct 13, 2017

daviddias assigned vmx Jan 12, 2018

daviddias added the difficulty:hard label Jan 25, 2018

vmx added ready and removed in progress labels Feb 5, 2018

daviddias and others added 7 commits March 28, 2018 14:02

feat: dag.put

450c581

Feat(dag): DAG api (#568)

7a8b030

* Implement dag.get * Handle dag put response, enable CIDs for dag get and fix dag get request param

feat(dag): update option names to reflect go-ipfs API

2e0c4db

feat(dag): proper get implementation

af9fc47

feat(dag): rebase, use waterfall for put

fce3138

fix(dag): path logic for DAG get was wrong

b7e95c2

fix(dag): js-ipld format resolver take the raw block

d98f8e1

The js-ipld formats API changed, they now take a raw data block rather than an IPFS block.

vmx force-pushed the feat/dag-api branch from 1caafb8 to 182014c Compare March 29, 2018 10:05

ghost added in progress and removed ready labels Mar 29, 2018

fix(dag): use SendOneFile for dag put

49dd9a7

There was a refactoring, now the usual way of doing a HTTP request is through SendOneFile.

vmx force-pushed the feat/dag-api branch from 182014c to 49dd9a7 Compare March 29, 2018 10:52

daviddias commented Apr 3, 2018

View reviewed changes

daviddias merged commit 9c37213 into master Apr 5, 2018

daviddias deleted the feat/dag-api branch April 5, 2018 15:02

ghost removed the in progress label Apr 5, 2018

vmx mentioned this pull request Apr 12, 2018

Use js-ipld for DAG API #738

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support DAG API #534

feat: support DAG API #534

daviddias commented Mar 9, 2017

atfornes commented May 11, 2017

daviddias commented May 11, 2017

atfornes commented May 26, 2017

atfornes commented May 29, 2017

magik6k commented Aug 12, 2017

magik6k commented Aug 13, 2017

daviddias Aug 15, 2017

magik6k Aug 15, 2017

daviddias Aug 15, 2017

daviddias Aug 15, 2017

koalalorenzo commented Feb 10, 2018

vmx commented Feb 12, 2018

pcardune commented Feb 15, 2018 •

edited

Loading

daviddias commented Feb 16, 2018

vmx commented Feb 16, 2018

vmx commented Feb 22, 2018

daviddias commented Feb 22, 2018

vmx commented Feb 22, 2018

vmx commented Apr 3, 2018

daviddias Apr 3, 2018

magik6k Apr 3, 2018

vmx Apr 3, 2018

magik6k Apr 3, 2018

vmx Apr 3, 2018

daviddias Apr 3, 2018

vmx Apr 3, 2018

vmx Apr 5, 2018

daviddias Apr 5, 2018

feat: support DAG API #534

feat: support DAG API #534

Conversation

daviddias commented Mar 9, 2017

atfornes commented May 11, 2017

daviddias commented May 11, 2017

atfornes commented May 26, 2017

atfornes commented May 29, 2017

magik6k commented Aug 12, 2017

magik6k commented Aug 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koalalorenzo commented Feb 10, 2018

vmx commented Feb 12, 2018

pcardune commented Feb 15, 2018 • edited Loading

daviddias commented Feb 16, 2018

vmx commented Feb 16, 2018

vmx commented Feb 22, 2018

daviddias commented Feb 22, 2018

vmx commented Feb 22, 2018

vmx commented Apr 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcardune commented Feb 15, 2018 •

edited

Loading