Awesome Endeavour: DHT Part II #856

daviddias · 2017-05-19T14:54:58Z

This PR is the second part of the DHT integration in js-ipfs

Remaining tasks

Main work tracker: libp2p/js-libp2p-kad-dht#1

Needs the following PRs:

Blocked PRs:

daviddias · 2017-07-20T05:45:19Z

Peer and Content Routing (DHT) js-ipfs-bitswap#142

daviddias · 2017-08-25T15:25:03Z

It might actually be good to enable the DHT if it proves to not break things and then figure out the interop of the DHT with go-ipfs as we go.

daviddias · 2017-08-26T14:31:57Z

Updated deps including all the latest changes in Bitswap. DHT tests in JS land are fine. Still no love between JS and Go 😢

daviddias · 2017-08-26T14:32:07Z

CI is still not happy as well


  1) interface-ipfs-core tests .dht callback API .findpeer finds other peers:
     Uncaught AssertionError: expected [Error] to not exist
      at nodeA.dht.findpeer (node_modules/interface-ipfs-core/src/dht.js:90:32)
      at self._libp2pNode.peerRouting.findPeer (src/core/components/dht.js:81:18)
      at node_modules/async/internal/once.js:12:16
      at next (node_modules/async/waterfall.js:21:29)
      at node_modules/async/internal/onlyOnce.js:12:16
      at waterfall (node_modules/libp2p-kad-dht/src/index.js:449:20)
      at nextTask (node_modules/async/waterfall.js:16:14)
      at next (node_modules/async/waterfall.js:23:9)
      at node_modules/async/internal/onlyOnce.js:12:16
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

  2) interface-ipfs-core tests .dht callback API .query returns the other node in the query:
     Uncaught AssertionError: expected [] to include 'QmS72d5n39h2zWXHQFFQs2Xub7GADbYimayvJYkocJv98D'
      at nodeA.dht.query (node_modules/interface-ipfs-core/src/dht.js:139:47)
      at self._libp2pNode._dht.getClosestPeers (src/core/components/dht.js:157:9)
      at q.run (node_modules/libp2p-kad-dht/src/index.js:172:18)
      at Query.run (node_modules/libp2p-kad-dht/src/query.js:47:14)
      at utils.convertBuffer (node_modules/libp2p-kad-dht/src/index.js:166:9)
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

hoffmabc · 2018-02-16T15:28:19Z

Is this PR still up to date?

daviddias · 2018-02-16T17:11:03Z

@hoffmabc it is as with regards to status, DHT is still a WIP. Lot's of work happening on js-libp2p land itself -- https://github.com/libp2p/js-libp2p --. Currently focused on libp2p/js-libp2p#159

alanshaw · 2018-04-11T11:39:31Z

Is there any prior docs I can read about the technical challenges involved in go <-> js DHT interop?

daviddias · 2018-04-30T19:18:41Z

Let's unblock this endeavour with the delegated routing as soon as 0.29.0 gets released.

alanshaw

Additionally the CLI command handlers may soon need to be updated to use getIpfs instead of ipfs. See #1860 for details.

alanshaw · 2019-02-05T16:23:17Z

README.md

@@ -319,7 +314,6 @@ Enable and configure experimental features.
 - `pubsub` (boolean): Enable libp2p pub-sub. (Default: `false`)
 - `ipnsPubsub` (boolean): Enable pub-sub on IPNS. (Default: `false`)
 - `sharding` (boolean): Enable directory sharding. Directories that have many child objects will be represented by multiple DAG nodes instead of just one. It can improve lookup performance when a directory has several thousand files or more. (Default: `false`)
- `dht` (boolean): Enable KadDHT. **This is currently not interoperable with `go-ipfs`.**


OK, so, I'm half inclined to just enable it in the browser by default, expose ourselves to the issues and get them fixed asap. That said, since this is such a big feature perhaps not having it enabled by default in the browser initially will give us some scope to test it out first and fix the bigger issues, so that we don't expose all our users to the bugs and make everyone mad.

We do need a way to enable/disable it though - if the dht config option is being removed from EXPERIMENTAL, is it being added to the top level?

alanshaw · 2019-02-05T16:32:13Z

src/cli/commands/dht/find-peer.js

+      const peers = await ipfs.dht.findPeer(peerID)
+      const addresses = peers.multiaddrs.toArray().map((ma) => ma.toString())
+
+      print(addresses)


print is just console.log which will give you output like:

[ '/ip4/127.0.0.1/tcp/4001', '/ip4/169.254.83.88/tcp/4001', '/ip4/192.168.1.95/tcp/4001' ]

In comparison go-ipfs outputs:

$ ipfs dht findpeer QmZMxNdpMkewiVZLMRxaNxUeZpDUb34pWjZ1kZvsd16Zic /ip6/2001:19f0:5:66d1::/tcp/4001 /ip4/149.28.228.162/tcp/33154 /ip4/45.77.150.82/tcp/4001 /ip6/::1/tcp/4001 /ip6/2001:19f0:5:5d40:5400:1ff:fec1:ce94/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip4/149.28.228.162/tcp/4001 /ip4/127.0.0.1/tcp/8081/ws

alanshaw · 2019-02-05T16:33:50Z

src/cli/commands/dht/provide.js

+  },
+
+  handler ({ ipfs, key, resolve }) {
+    // TODO add recursive option


Please pass this option to ipfs.dht.provide so it can throw and inform the user the option is not implemented yet.

alanshaw · 2019-02-05T16:39:42Z

src/core/components/libp2p.js

@@ -76,6 +76,9 @@ function defaultBundle ({ datastore, peerInfo, peerBook, options, config }) {
        }
      },
      dht: {
+        kBucketSize: get(options, 'dht.kBucketSize', 20),
+        enabled: get(options, 'dht.enabled', true) && !(get(options, 'local', false)),


"local" is now "offline"?

alanshaw · 2019-02-05T16:40:47Z

src/core/components/pin-set.js

-const some = require('async/some')
-const eachOf = require('async/eachOf')
+const someSeries = require('async/someSeries')
+const eachOfSeries = require('async/eachOfSeries')


What does this change do?

Using some is making a lot of parallel dht find and connect requests.

So, everything will get queried even if we find the target node, as the find takes longer than initiating the requests.

In the long run, we should refactor the pin-set, in order to try to parallelize some operations, but this is the safest option currently.

alanshaw · 2019-02-05T16:41:47Z

src/http/api/resources/dht.js

+      arg: Joi.string().required()
+    }).unknown()
+  },
+  handler: (request, reply) => {


These handlers now need to change for Hapi 18 - please shout if you have any questions!

daviddias added awesome endeavour status/in-progress In progress labels May 19, 2017

daviddias mentioned this pull request Jun 13, 2017

Site refactor libp2p/website#52

Merged

33 tasks

daviddias added status/ready Ready to be worked and removed status/in-progress In progress labels Jul 3, 2017

This was referenced Jul 5, 2017

Uploaded file via Spawned Node not accessible via public gateway #886

Closed

Implement IPNS #209

Closed

daviddias mentioned this pull request Aug 25, 2017

The file added by node.js can't be get/cat on other nodes #957

Closed

daviddias force-pushed the feat/dht-part-ii branch from 6a29c16 to df424e5 Compare August 26, 2017 14:03

ReisenB mentioned this pull request Sep 15, 2017

State of IPFS DistributedMemetics/DM#1

Open

daviddias added P0 Critical: Tackled by core team ASAP status/ready Ready to be worked and removed status/ready Ready to be worked labels Oct 17, 2017

daviddias mentioned this pull request Nov 11, 2017

Talking to js-ipfs richardschneider/net-ipfs-http-client#19

Open

RangerMauve mentioned this pull request Jan 9, 2018

pub/sub - publish / subscribe ipfs/notes#64

Open

3 tasks

dryajov mentioned this pull request Mar 26, 2018

OKR 2018 Q2 #1281

Merged

neuronsupport mentioned this pull request Apr 15, 2018

WebTorrent DHT integration webtorrent/webtorrent#1152

Open

daviddias mentioned this pull request Apr 23, 2018

feat: ping #1299

Merged

2 tasks

fazo96 mentioned this pull request May 2, 2018

peers don't connect automatically through circuit, but they do through websocket-star rendezvous server #1309

Closed

vasco-santos mentioned this pull request May 8, 2018

Merge branch 'master' of linonetwo/ipfs-browser-gateway ipfs-shipyard/service-worker-gateway-2019-poc#1

Closed

daviddias self-assigned this May 30, 2018

daviddias force-pushed the feat/dht-part-ii branch from df424e5 to 5347ffd Compare May 30, 2018 09:30

vasco-santos force-pushed the feat/dht-part-ii branch from b0acc71 to c4cacf8 Compare February 1, 2019 11:24

alanshaw suggested changes Feb 5, 2019

View reviewed changes

vasco-santos force-pushed the feat/dht-part-ii branch 2 times, most recently from 999ebf2 to 2b548be Compare February 6, 2019 11:31

alanshaw mentioned this pull request Feb 6, 2019

dht functionality #1857

Closed

vasco-santos force-pushed the feat/dht-part-ii branch 2 times, most recently from 9a4083e to c16bdc4 Compare February 6, 2019 15:32

Alan Shaw and others added 9 commits February 6, 2019 15:41

chore: increase silent timeout even more

53a3f32

feat: dht ready

28f5641

test: enable dht core test

8ecb7b5

feat: add dht cli

f9cd739

chore: enable dht by default

eadd7d5

fix: code review

4453b55

fix: code review

2a76f98

fix: ci tests

045e305

fix: code review

2956637

vasco-santos force-pushed the feat/dht-part-ii branch from c16bdc4 to 2956637 Compare February 6, 2019 15:46

alanshaw approved these changes Feb 6, 2019

View reviewed changes

Merge branch 'master' into feat/dht-part-ii

2651787

ghost assigned alanshaw Feb 7, 2019

vasco-santos force-pushed the feat/dht-part-ii branch 4 times, most recently from 0c73e99 to 147b100 Compare February 7, 2019 22:32

fix: make ci happy again

02d1704

vasco-santos force-pushed the feat/dht-part-ii branch from 147b100 to 02d1704 Compare February 7, 2019 23:20

vasco-santos mentioned this pull request Feb 8, 2019

feat: limit connections number #1872

Merged

4 tasks

alanshaw merged commit 77a0957 into master Feb 8, 2019

alanshaw deleted the feat/dht-part-ii branch February 8, 2019 11:01

ghost removed the status/in-progress In progress label Feb 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Awesome Endeavour: DHT Part II #856

Awesome Endeavour: DHT Part II #856

daviddias commented May 19, 2017 •

edited by vasco-santos

Loading

daviddias commented Jul 20, 2017 •

edited

Loading

daviddias commented Aug 25, 2017

daviddias commented Aug 26, 2017

daviddias commented Aug 26, 2017

hoffmabc commented Feb 16, 2018

daviddias commented Feb 16, 2018

alanshaw commented Apr 11, 2018

daviddias commented Apr 30, 2018

alanshaw left a comment

alanshaw Feb 5, 2019

alanshaw Feb 5, 2019

alanshaw Feb 5, 2019

alanshaw Feb 5, 2019

alanshaw Feb 5, 2019

vasco-santos Feb 6, 2019

alanshaw Feb 5, 2019

Awesome Endeavour: DHT Part II #856

Awesome Endeavour: DHT Part II #856

Conversation

daviddias commented May 19, 2017 • edited by vasco-santos Loading

Remaining tasks

Needs the following PRs:

Blocked PRs:

daviddias commented Jul 20, 2017 • edited Loading

daviddias commented Aug 25, 2017

daviddias commented Aug 26, 2017

daviddias commented Aug 26, 2017

hoffmabc commented Feb 16, 2018

daviddias commented Feb 16, 2018

alanshaw commented Apr 11, 2018

daviddias commented Apr 30, 2018

alanshaw left a comment

Choose a reason for hiding this comment

alanshaw Feb 5, 2019

Choose a reason for hiding this comment

alanshaw Feb 5, 2019

Choose a reason for hiding this comment

alanshaw Feb 5, 2019

Choose a reason for hiding this comment

alanshaw Feb 5, 2019

Choose a reason for hiding this comment

alanshaw Feb 5, 2019

Choose a reason for hiding this comment

vasco-santos Feb 6, 2019

Choose a reason for hiding this comment

alanshaw Feb 5, 2019

Choose a reason for hiding this comment

daviddias commented May 19, 2017 •

edited by vasco-santos

Loading

daviddias commented Jul 20, 2017 •

edited

Loading