Add minimal Kademlia DHT spec #108

raulk · 2018-11-07T18:49:37Z

~~This was a complicated birth.~~

~~TODO:~~

~~Specify that we haven't implemented PINGs.~~
Deviations in bucket behaviour from baseline Kademlia. Since we don't use PINGs, we don't test the least recently seen peer. We evict it blindly, thus causing a high degree of bucket thrashing and not observing the heuristic "the longer a peer has been around, the more likely it is to stay around".
~~Revisit RPC messages section. Copy protobufs and explain each field meticulously.~~
~~Resolve all feedback below.~~

Update by @mxinden on 2021-05-14.

This pull request adds a minimal specification for the libp2p Kademlia DHT protocol. It is a minimal version and thus while striving for correctness it does not strive for completeness. The version proposed in this pull request sets a base layer specification to be expanded in the future.

Areas worth expanding in future pull requests:

Client mode (non publicly reachable nodes not advertising the kademlia protocol ID) as well as its implications to FIND_NODE (see https://github.com/libp2p/specs/pull/108/files#r657950178).
Detailed record validation
Default value adjustments and reasoning
Detailed request handling
Expanded k-bucket routing table management
...

jhiesey

Looks pretty good, just a few comments. Do you want me to add the WIP (jhiesey) sections before or after merging this?

kad-dht/README.md

vasco-santos

Seems a good initial version of this spec! Thanks @raulk

Left just a comment

kad-dht/README.md

tomaka · 2018-11-14T21:15:26Z

We should also mention how the substreams that use the protocol behave.
Is it expected to open one substream per request? Or should implementations make an effort to only use one at a time and send requests one by one? Should endpoints close their substream after a successful response, or can they continue sending more requests?
(note: I actually have these questions right now for Rust, as I have no clue)

kad-dht/README.md

tomaka · 2018-11-14T21:20:50Z

Kademlia is by far the protocol for which I've suffered the most when implementing into Rust, because of all the differences between the well-documented Kademlia algorithm and the way it is implemented in libp2p. I think we should focus more on the latter and not copy-paste what people can already find on Google.

tomaka · 2018-11-14T21:21:57Z

Also, writing the Kademlia specs is a tremendous task. I don't expect a complete spec to be less than ~2k lines, and expecting a single person to write them is very optimistic.

raulk · 2018-11-14T23:45:56Z

@tomaka

Kademlia is by far the protocol for which I've suffered the most when implementing into Rust, because of all the differences between the well-documented Kademlia algorithm and the way it is implemented in libp2p. I think we should focus more on the latter and not copy-paste what people can already find on Google.

That was the spirit of this spec: to focus on differential areas vs. regurgitating how Kademlia works (as conveyed in the spec intro). Hence it covers provider records, public key records, conflict resolution, peer correction, etc. which is specific to libp2p Kad DHT.

I don't expect a complete spec to be less than ~2k lines

Could you enumerate what other aspects are worth covering? Aspects that are unique in libp2p. We don't want to clone the Kademlia and friends literature.

Regarding all the data model comments, there's a request in the doc for @jhiesey to replace these descriptions with the protobuf.

tomaka · 2018-11-15T10:15:49Z

Could you enumerate what other aspects are worth covering? Aspects that are unique in libp2p. We don't want to clone the Kademlia and friends literature.

Well, the 2k would include actually covering Kademlia, which I think we should do anyway, just not urgently.

I think there should be more explanation as to what actually happens on the network, rather than just dumping a protobuf definition file.
For example, which fields need to be field for which kind of RPC query? The format of the fields (bytes doesn't mean much)? Should nodes relay multiaddresses in their raw format, or produce error/ignore the ones it can't decode? Is there a limit to the number of multiaddresses that a peerinfo should contain?
That's just from the top of my head.
Also, the reason why I didn't implement record store in rust-libp2p at the time is that the Record definition was way too vague to be helpful. I'd expect more help from a specs.

raulk · 2018-11-17T11:00:45Z

@tomaka I agree that the RPC section needs redoing. The idea is to copy the protobufs, as these are normative for interoperability, and explain how each field is used and serialised (especially for the bytes types, which can be anything). @jhiesey, are you planning to tackle this?

I do recognise you've implemented this from scratch, and therefore your feedback is valuable. However, in all honesty, I don't see the value in reinventing the wheel and re-specifying the Kademlia baseline in this spec. I rather make it pre-required reading (like I've done), and build on top of it.

In a nutshell, I think of this spec as a diff between baseline Kademlia and our implementation, which:

deviates from Kademlia in some aspects, e.g. the way we manage buckets, the lack of PINGs, etc. (this needs specifying!)
cherry-picks ideas from Coral, mainlineDHT, etc.

Maybe you can compile a list of all the areas you tripped over, and we can make sure to cover them?

Also, Kademlia is abstract, in the sense that it doesn't specify wire messages, RPCs, timeouts, etc. So our spec should define those aspects very clearly.

tomaka · 2018-11-28T09:44:35Z

cc the second part of this comment: #111 (comment)

raulk · 2018-12-17T17:58:33Z

@jhiesey tagged you in the comments that need your attention; let's try to push this through! ;-)

jhiesey · 2018-12-21T01:05:56Z

@raulk sorry for not getting to this before now! Will work on this today.

kad-dht/README.md

jhiesey · 2018-12-21T05:44:25Z

I've addressed the issues I see. Left some more comments too.

jhiesey · 2019-01-10T21:38:43Z

What's the status on this @raulk? Anything I can do to help?

kad-dht/README.md

jhiesey · 2019-01-11T00:21:31Z

After reading @anacrolix 's feedback on this and my refactor proposal, I think we should simplify this DHT node spec substantially and move a bunch of the hairier stuff into separate discovery modules with their own specs.

Mikerah · 2019-02-06T03:54:11Z

I noticed that some of the links the the bibliography were behind Springer's paywall. It would be awesome to provide the links to these papers from the author's website for example. I think this would increase the accessibility of the spec.

petar · 2021-06-01T23:41:49Z

kad-dht/README.md

+
+### Kademlia routing table
+
+The data structure backing this system is a k-bucket routing table, closely


It is not clear to me what is this document a spec of? This paragraph describes our implementation, not the spec of the protocol (which is implementation-agnostic). E.g. it is an implementation detail that we use kbucket data structure. Other places in our stack use XOR-tries instead (which is generally recommended). So, what is this a spec of? Or is this document a description of an implementation?

A protocol spec would be worded differently. It would say: An implementation of a Kademlia node must maintain K peers with shared prefix of length L, for every L.

So, what is this a spec of?

Good point. This should be a specification of the protocol. Not a description of a specific implementation.

E.g. it is an implementation detail that we use kbucket data structure.

Thanks. 9355a8f refines this section, treating the data structure as an implementation detail. The section now contains your generic description using key prefix lengths.

Other places in our stack use XOR-tries instead (which is generally recommended).

I was not aware of XOR-tries. I will take a deeper look. With 9355a8f the specification suggests XOR-tries as one possible data structure.

petar · 2021-06-02T16:26:29Z

kad-dht/README.md

+The libp2p Kademlia DHT offers the following types of routing operations:
+
+- **Peer routing** - _Finding_ the closest nodes to a given key (`FIND_NODE`).
+- **Value routing** - _Putting_ a value to the nodes closest to the value's key


This is not a routing operation (as the section title suggests). Also the description is not accurate.
PUT_VALUE puts a value on the node it is called on. (Not on the "closest node to the key").

Changed via 072360f. Let me know what you think @petar.

petar · 2021-06-02T16:26:50Z

kad-dht/README.md

+- **Value routing** - _Putting_ a value to the nodes closest to the value's key
+  (`PUT_VALUE`) and _getting_ a value by its key from the nodes closest to that
+  key (`GET_VALUE`).
+- **Content routing** - _Adding_ oneself to the list of providers for a given


Same comment as above.

Changed via 072360f. Let me know what you think @petar.

petar · 2021-06-02T16:29:34Z

kad-dht/README.md

+### Peer routing
+
+The below is one possible algorithm to find nodes closest to a given key on
+the DHT. Implementations may diverge from this base algorithm as long as they


"Implementations may diverge from this base algorithm": so what's the point of describing this algorithm if our implementation is different and other implementations don't have to follow this?

The motivation for including this basic algorithm is to enable readers to write a minimal implementation compatible with the existing networks (e.g. IPFS and Polkadot) without having to read e.g. the go-libp2p or rust-libp2p source code.

That said, I don't feel strongly about this. If you prefer I am happy to remove it. In my opinion we should add descriptions of the more advanced algorithms from go-libp2p in future pull requests.

Sounds good. I would just elaborate slightly, something like: "may diverge as long as they adhere to the wire format and make progress towards the target key"

Added in b074091.

petar · 2021-06-02T16:30:26Z

kad-dht/README.md

+
+We keep track of the set of peers we've already queried (`Pq`) and the set of
+next query candidates sorted by distance from `Key` in ascending order (`Pn`).
+At initialization `Pn` is seeded with the `α` peers from our routing table we


Our implementation, is seeded with K peers.

Thanks. Adjusted via c4d4b53.

petar · 2021-06-02T16:32:54Z

kad-dht/README.md

+Then we loop:
+
+1. > The lookup terminates when the initiator has queried and gotten responses
+   from the k (see [#replication-parameter-k]) closest nodes it has seen.


In our implementation, a lookup actually terminates when a given percent (e.g. 80%) of the beta (another parameter) peers closest to the target have been contacted.

Good to know. That might be something worth exploring for rust-libp2p as well.

As mentioned above, I would suggest keeping this sample algorithm minimal for now, extending it with various optimization descriptions in future pull requests. Again, I don't feel strongly about this.

That sounds fine. What you've written will work. It'll just wait quite a bit before it is ready to terminate, but it is correct.

petar · 2021-06-02T16:33:55Z

kad-dht/README.md

+
+   The lookup might terminate early in case the local node queried all known
+   nodes, with the number of nodes being smaller than `k`.
+2. Pick as many peers from the candidate peers (`Pn`) as the `α` concurrency


When do you do the picking? Our implementation does this after a prior request completes and the state is updated.

I am not sure I follow. Steps 1 - 4 are executed in a loop, thus picking happens at the very start as well as each time a response is received (3) and the termination criterion (1) is still false. As far as I can tell, this is inline with what you describe above. Does that make sense @petar?

You are right... I overlooked. This looks fine.

mxinden

Thanks @petar for the review! Very much appreciated.

I have addressed all your comments. Would you mind taking another look?

mxinden · 2021-06-03T12:52:55Z

kad-dht/README.md

+### Peer routing
+
+The below is one possible algorithm to find nodes closest to a given key on
+the DHT. Implementations may diverge from this base algorithm as long as they


The motivation for including this basic algorithm is to enable readers to write a minimal implementation compatible with the existing networks (e.g. IPFS and Polkadot) without having to read e.g. the go-libp2p or rust-libp2p source code.

That said, I don't feel strongly about this. If you prefer I am happy to remove it. In my opinion we should add descriptions of the more advanced algorithms from go-libp2p in future pull requests.

mxinden · 2021-06-03T12:54:32Z

kad-dht/README.md

+
+We keep track of the set of peers we've already queried (`Pq`) and the set of
+next query candidates sorted by distance from `Key` in ascending order (`Pn`).
+At initialization `Pn` is seeded with the `α` peers from our routing table we


Thanks. Adjusted via c4d4b53.

mxinden · 2021-06-03T12:58:48Z

kad-dht/README.md

+Then we loop:
+
+1. > The lookup terminates when the initiator has queried and gotten responses
+   from the k (see [#replication-parameter-k]) closest nodes it has seen.


Good to know. That might be something worth exploring for rust-libp2p as well.

As mentioned above, I would suggest keeping this sample algorithm minimal for now, extending it with various optimization descriptions in future pull requests. Again, I don't feel strongly about this.

mxinden · 2021-06-03T13:01:42Z

kad-dht/README.md

+
+   The lookup might terminate early in case the local node queried all known
+   nodes, with the number of nodes being smaller than `k`.
+2. Pick as many peers from the candidate peers (`Pn`) as the `α` concurrency


I am not sure I follow. Steps 1 - 4 are executed in a loop, thus picking happens at the very start as well as each time a response is received (3) and the termination criterion (1) is still false. As far as I can tell, this is inline with what you describe above. Does that make sense @petar?

mxinden · 2021-06-03T13:06:50Z

kad-dht/README.md

+
+### Kademlia routing table
+
+The data structure backing this system is a k-bucket routing table, closely


So, what is this a spec of?

Good point. This should be a specification of the protocol. Not a description of a specific implementation.

E.g. it is an implementation detail that we use kbucket data structure.

Thanks. 9355a8f refines this section, treating the data structure as an implementation detail. The section now contains your generic description using key prefix lengths.

Other places in our stack use XOR-tries instead (which is generally recommended).

I was not aware of XOR-tries. I will take a deeper look. With 9355a8f the specification suggests XOR-tries as one possible data structure.

mxinden · 2021-06-03T13:07:31Z

kad-dht/README.md

+The libp2p Kademlia DHT offers the following types of routing operations:
+
+- **Peer routing** - _Finding_ the closest nodes to a given key (`FIND_NODE`).
+- **Value routing** - _Putting_ a value to the nodes closest to the value's key


Changed via 072360f. Let me know what you think @petar.

mxinden · 2021-06-03T13:07:36Z

kad-dht/README.md

+- **Value routing** - _Putting_ a value to the nodes closest to the value's key
+  (`PUT_VALUE`) and _getting_ a value by its key from the nodes closest to that
+  key (`GET_VALUE`).
+- **Content routing** - _Adding_ oneself to the list of providers for a given


Changed via 072360f. Let me know what you think @petar.

mxinden · 2021-06-08T12:50:59Z

@petar friendly ping. Would you mind taking another look?

@aschmahmann could you give this pull request a review as well?

petar · 2021-06-08T13:36:26Z

@mxinden Looks much better. Thank you!

aschmahmann

Thanks, this is much improved. Left a few comments/questions.

aschmahmann · 2021-06-24T13:16:22Z

kad-dht/README.md

+  - Getting providers for a given key from the nodes closest to that key via
+    `GET_PROVIDERS`.


Not sure if worth pointing out here that GET_PROVIDERS and GET_VALUE both have FIND_NODE semantics too (i.e. they return values and/or closest peers to the target) and so we can optimize by not needing to do FIND_NODE's first.

Not sure if worth pointing out here that GET_PROVIDERS and GET_VALUE both have FIND_NODE semantics too (i.e. they return values and/or closest peers to the target)

This is detailed in the corresponding GET_PROVIDER and GET_VALUE message descriptions in ## RPC messages:

* `GET_VALUE`: In the request `key` is an unstructured array of bytes. If `key` is a public key (begins with `/pk/`) and the key is known, the response has `record` set to that key. Otherwise, `record` is set to the value for the given key (if found in the datastore) and `closerPeers` is set to the `k` closest peers. * `GET_PROVIDERS`: In the request `key` is set to a CID. The target node returns the closest known `providerPeers` (if any) and the `k` closest known `closerPeers`.

What do you think @aschmahmann? Is this good enough?

and so we can optimize by not needing to do FIND_NODE's first.

I am not quite sure why one should do a FIND_NODE before a GET_PROVIDER or GET_VALUE in the first place? As far as I know this is neither suggested in the Kademlia paper nor in this specification (e.g. see #### Content provider discovery section). Am I missing something @aschmahmann?

Is this good enough?

Probably?

Am I missing something @aschmahmann?

Nope, I guess it's just an obvious sort of thing. It could have been designed that GET_VALUE only returned the value/error instead of also sending back closest peers but this way is more efficient.

aschmahmann · 2021-06-24T13:31:23Z

kad-dht/README.md

+- **Peer routing**
+
+  - Finding the closest nodes to a given key via `FIND_NODE`.


I'd be a bit careful with the wording here. FIND_NODE is unfortunately an overloaded function that does two things:

Tells you the closest DHT server peers to the target

Tells you about a peer you explicitly asked for even if they are not a DHT server

The second thing is frequently referred to as peer routing since it's helping route you to a peer. I'm not sure what to call the first, but it's routing you towards the target key in the peer-space (not quite the kademlia/xor space since the peerIDs need to be SHA256'd first).

Maybe just call this section FIND_NODE and define both behaviors.

If your curious this issue (libp2p/go-libp2p-kad-dht#584) describes my frustration with special casing around FIND_NODE and ADD/GET_PROVIDERS

I'd be a bit careful with the wording here. FIND_NODE is unfortunately an overloaded function that does two things:

1. Tells you the closest DHT server peers to the target 2. Tells you about a peer you explicitly asked for even if they are not a DHT server

Let me paraphrase the above to make sure I correctly understand your argument:

On FIND_NODE a DHT server returns the closest other DHT servers to the given target.

On FIND_NODE a DHT server returns a single DHT client if and only if the DHT client's peer ID matches the target.

In my eyes documenting this distinction without documenting the DHT client / server mode extension (see your comment on rust-libp2p) would be inconsistent.

I would prefer to tackle both in follow-up pull requests. What do you think @aschmahmann?

I added your comment to the list at the top of the pull request to keep track of it.

On FIND_NODE a DHT server returns a single DHT client if and only if the DHT client's peer ID matches the target.

Not quite, if we happen to know about a peer whose peerID is an exact match for the FIND_NODE query key then we add them to our list of servers to return. https://github.com/libp2p/go-libp2p-kad-dht/blob/6fff2a3b391f73da7a1c7558e27725ebe730e7ba/handlers.go#L256

In my eyes documenting this distinction without documenting the DHT client / server mode extension (see your comment on rust-libp2p) would be inconsistent.

I guess that's fair. If all nodes were servers this distinction wouldn't really matter. A follow up PR is fine.

Note: client mode isn't just for unreachable peers, it's generally recommended for "low quality" peers. This could mean low power, low bandwidth, low network uptime, etc. Basically peers that if they were servers would do more harm then good probably shouldn't be servers but may still want to make queries.

aschmahmann · 2021-06-24T14:02:17Z

kad-dht/README.md

+to _put_ and _get_ the public keys of nodes. Node public keys are stored in
+records under the `/pk` namespace. That is, the entry `/pk/<peerID>` will store
+the public key of peer `peerID`.


Why are we referencing the /pk space as part of the libp2p spec, is this just an example?

Two things:

This namespace shouldn't be required

It's effectively deprecated in IPFS

For context the main reason why it was deprecated was that it straddled this weird space between:

Your key is too big to include in whatever other system actually needs your key

Your key is small enough a DHT server will store it for you

Eventually IPFS decided that the distance between those two was effectively zero and started including the keys that were previously deemed too large (i.e. RSA) in IPNS records (the place they needed them). They also switched to Ed25519 keys by default since they're much smaller so they don't even need to embed them anymore.

Thanks both for the hint and the background. Removed with a065aac.

aschmahmann · 2021-06-24T14:05:42Z

kad-dht/README.md

+When _gettting_ a value in the DHT, the implementor should collect at least `Q`
+(quorum) responses from distinct nodes to check for consistency before returning
+an answer.


The point here is basically to determine if/when it's safe to shortcut your query and there are multiple ways to determine reliability.

A plain quorum is just one way. For example, you could also choose to adjust your view based on how close to the target the peer who gave you the content is.

I am in favor for this specification to be extended with more advanced procedures, though I would like to keep it simple in this iteration only mentioning the simple Quorum style. Is this ok for you @aschmahmann? If so, would you still like the wording to be adjusted?

I liked your wording "Implementations may diverge from this base algorithm as long as they adhere
to the wire format and make progress towards the target key." in that it indicated what we care about vs what we don't.

If you think it's obvious that all of this quorum stuff is optional client behavior and people can do whatever they want then feel free to leave it as is. If you think it's not really obvious, then it might be worth adding some wording around how when getting values you can abort whenever you feel confident you have the latest value, for example using a quorum.

324f915 extends the section, stressing the fact that quorum is one mechanism among many to determine when a query is finished. Let me know what you think.

aschmahmann · 2021-06-24T14:11:53Z

kad-dht/README.md

+   terminate early and return `best`. In either case we notify the peers holding
+   an outdated value (`Po`) of the best value we discovered, by sending
+   `PUT_VALUE(Key, best)` messages.


You probably also want to send the PUT_VALUE to the closest peers who should've had it. i.e. you're sending to closest peers - peers who already have the latest

Great catch, thanks! Adjusted in 20b3b73.

aschmahmann · 2021-06-24T16:54:25Z

kad-dht/README.md

+For performance reasons, a node may prune expired advertisements only
+periodically, e.g. every hour.


This is generic to all Puts/Gets. Your implementation can do whatever as long as you only return valid data.

Your network may have implied and/or explicit expirations times per record type.

This is an artifact of past versions of this pull request. Thinking about it some more, I am in favor of removing it. To me it is implementation specific. 1dcb218 removes it.

@aschmahmann let me know if you disagree.

aschmahmann · 2021-06-24T18:50:08Z

kad-dht/README.md

+On every run, we generate a random peer ID and we look it up via the process
+defined in [peer routing](#peer-routing). Peers encountered throughout the
+search are inserted in the routing table, as per usual business.
+
+This process is repeated as many times per run as configuration parameter
+`QueryCount` (default: 1). Every repetition is subject to a `QueryTimeout`
+(default: 10 seconds), which upon firing, aborts the run.


This isn't what go-libp2p does. We can slap on the usual "here's one way to do it disclaimer".

Unfortunately, this way is kind of bad because the bigger the network is the harder it is to find peers closer to you like this. At the very least you want to search for yourself in the network.

I can give the brief approach taken in go-libp2p-kad-dht if you're interested.

👍 c755a41 includes self lookup.

This isn't what go-libp2p does.

Neither does rust-libp2p. It first does a lookup for the local key and then tries to probabilistically generate a key for each bucket.

I can give the brief approach taken in go-libp2p-kad-dht if you're interested.

Yes.

aschmahmann · 2021-06-24T18:52:57Z

kad-dht/README.md

+    // Note: These fields were removed from the Record message
+    //
+    // Hash of the authors public key
+    // optional string author = 3;
+    // A PKI signature for the key+value+author
+    // optional bytes signature = 4;


Any reason to keep these in, maybe we should just denote the numbers as reserved/dead?

I kept them in to give some background on what these numbers were originally used for. In my eyes, with the note at the top // Note: These fields were removed from the Record message this shouldn't confuse anyone.

Not a strong opinion. Would you prefer this to be changed?

Your call. Was thinking it might make the long spec a little shorter was all.

aschmahmann · 2021-06-24T18:57:39Z

kad-dht/README.md

+  node to be found. `closerPeers` is set in the response; for an exact match
+  exactly one `Peer` is returned; otherwise `k` closest `Peer`s are returned.


What do you by exact match here?

As in the ID of the peer matches the target key in the original FIND_NODE request. Does that make sense @aschmahmann?

I'm not really sure why we don't always return the k closest peers but by https://github.com/libp2p/go-libp2p-kad-dht/blob/6fff2a3b391f73da7a1c7558e27725ebe730e7ba/handlers.go#L256

the only time we return a single peer is when Peer A asks Peer B "do you know peer B" and they say "yes, it's me". If Peer A asks Peer B "do you know peer C" they will say "yes here's there address along with k peers closer to them"

Thanks for looking this up. dab4549 simplifies the specification, always returning the k closest peers. I don't think we should document / require the peculiarity of go-libp2p-kad-dht returning a single peer when oneself was looked up. Sounds good?

I think that's right, or if it's important we can catch it in the follow up PR. We should investigate historical context here and see if a change is merited in go-libp2p-kad-dht.

kad-dht/README.md

mxinden

Thanks @aschmahmann for the thorough review, especially with the additional background.

I replied to all of your comments. Would you mind taking another look?

kad-dht/README.md

mxinden · 2021-06-25T09:03:27Z

kad-dht/README.md

+- **Peer routing**
+
+  - Finding the closest nodes to a given key via `FIND_NODE`.


I'd be a bit careful with the wording here. FIND_NODE is unfortunately an overloaded function that does two things:

1. Tells you the closest DHT server peers to the target 2. Tells you about a peer you explicitly asked for even if they are not a DHT server

Let me paraphrase the above to make sure I correctly understand your argument:

On FIND_NODE a DHT server returns the closest other DHT servers to the given target.

On FIND_NODE a DHT server returns a single DHT client if and only if the DHT client's peer ID matches the target.

In my eyes documenting this distinction without documenting the DHT client / server mode extension (see your comment on rust-libp2p) would be inconsistent.

I would prefer to tackle both in follow-up pull requests. What do you think @aschmahmann?

I added your comment to the list at the top of the pull request to keep track of it.

mxinden · 2021-06-25T09:11:58Z

kad-dht/README.md

+  - Getting providers for a given key from the nodes closest to that key via
+    `GET_PROVIDERS`.


Not sure if worth pointing out here that GET_PROVIDERS and GET_VALUE both have FIND_NODE semantics too (i.e. they return values and/or closest peers to the target)

This is detailed in the corresponding GET_PROVIDER and GET_VALUE message descriptions in ## RPC messages:

* `GET_VALUE`: In the request `key` is an unstructured array of bytes. If `key` is a public key (begins with `/pk/`) and the key is known, the response has `record` set to that key. Otherwise, `record` is set to the value for the given key (if found in the datastore) and `closerPeers` is set to the `k` closest peers. * `GET_PROVIDERS`: In the request `key` is set to a CID. The target node returns the closest known `providerPeers` (if any) and the `k` closest known `closerPeers`.

What do you think @aschmahmann? Is this good enough?

and so we can optimize by not needing to do FIND_NODE's first.

I am not quite sure why one should do a FIND_NODE before a GET_PROVIDER or GET_VALUE in the first place? As far as I know this is neither suggested in the Kademlia paper nor in this specification (e.g. see #### Content provider discovery section). Am I missing something @aschmahmann?

mxinden · 2021-06-25T09:18:20Z

kad-dht/README.md

+to _put_ and _get_ the public keys of nodes. Node public keys are stored in
+records under the `/pk` namespace. That is, the entry `/pk/<peerID>` will store
+the public key of peer `peerID`.


Thanks both for the hint and the background. Removed with a065aac.

mxinden · 2021-06-25T09:21:43Z

kad-dht/README.md

+When _gettting_ a value in the DHT, the implementor should collect at least `Q`
+(quorum) responses from distinct nodes to check for consistency before returning
+an answer.


I am in favor for this specification to be extended with more advanced procedures, though I would like to keep it simple in this iteration only mentioning the simple Quorum style. Is this ok for you @aschmahmann? If so, would you still like the wording to be adjusted?

mxinden · 2021-06-25T09:49:35Z

kad-dht/README.md

+Each peer that receives the `ADD_PROVIDER` RPC should validate that the
+received `PeerInfo` matches the sender's `peerID`, and if it does, that peer
+must store the `PeerInfo` in its datastore.


Again, appreciate the additional background. 6ec65b5 loosens the requirement.

mxinden · 2021-06-25T09:52:31Z

kad-dht/README.md

+For performance reasons, a node may prune expired advertisements only
+periodically, e.g. every hour.


This is an artifact of past versions of this pull request. Thinking about it some more, I am in favor of removing it. To me it is implementation specific. 1dcb218 removes it.

@aschmahmann let me know if you disagree.

mxinden · 2021-06-25T10:04:21Z

kad-dht/README.md

+On every run, we generate a random peer ID and we look it up via the process
+defined in [peer routing](#peer-routing). Peers encountered throughout the
+search are inserted in the routing table, as per usual business.
+
+This process is repeated as many times per run as configuration parameter
+`QueryCount` (default: 1). Every repetition is subject to a `QueryTimeout`
+(default: 10 seconds), which upon firing, aborts the run.


👍 c755a41 includes self lookup.

This isn't what go-libp2p does.

Neither does rust-libp2p. It first does a lookup for the local key and then tries to probabilistically generate a key for each bucket.

I can give the brief approach taken in go-libp2p-kad-dht if you're interested.

Yes.

mxinden · 2021-06-25T10:06:09Z

kad-dht/README.md

+    // Note: These fields were removed from the Record message
+    //
+    // Hash of the authors public key
+    // optional string author = 3;
+    // A PKI signature for the key+value+author
+    // optional bytes signature = 4;


I kept them in to give some background on what these numbers were originally used for. In my eyes, with the note at the top // Note: These fields were removed from the Record message this shouldn't confuse anyone.

Not a strong opinion. Would you prefer this to be changed?

mxinden · 2021-06-25T10:08:43Z

kad-dht/README.md

+  node to be found. `closerPeers` is set in the response; for an exact match
+  exactly one `Peer` is returned; otherwise `k` closest `Peer`s are returned.


As in the ID of the peer matches the target key in the original FIND_NODE request. Does that make sense @aschmahmann?

aschmahmann

Added one more comment, and I'm sure there are nits to be had in places but this is way better than nothing and given we are planning a follow up PR to iron out more details of the spec (e.g. support for clients) I'm happy here. Thanks for all the hard work 😄

My 2c are that as the specs evolve, including making an IPFS DHT spec, we can start to separate out the segments on, basic wire formats/rules ("the spec"), how to build compatible networks ("how to use the spec"), and how client implementations might work ("how to work with spec compliant networks").

aschmahmann · 2021-06-25T18:01:30Z

kad-dht/README.md

+The bootstrap process is responsible for keeping the routing table filled and
+healthy throughout time. It runs once on startup, then periodically with a
+configurable frequency (default: 5 minutes).
+
+On every run, we generate a random peer ID and we look it up via the process
+defined in [peer routing](#peer-routing). Peers encountered throughout the
+search are inserted in the routing table, as per usual business.
+
+This is repeated as many times per run as configuration parameter `QueryCount`
+(default: 1). In addition, to improve awareness of nodes close to oneself,
+implementations should include a lookup for their own peer ID.
+
+Every repetition is subject to a `QueryTimeout` (default: 10 seconds), which
+upon firing, aborts the run.


Should include our usual, here's one simple way of doing this. Especially since neither go nor rust libp2p's implementations actually do this.

👍 added via 3e6f8f5.

mxinden · 2021-06-28T08:52:16Z

@tomaka @alanshaw you are both still tracked as "requesting changes". I have addressed your comments and suggestions above. Could you either remove your request for changes or give this pull request another review?

BigLep · 2021-06-28T13:41:48Z

@mxinden : given this has had lots of eyeballs and iterations, I would suggest we give a cutoff date for comments (e.g., EOD 2021-06-29). Otherwise we merge. Further improvements can always be made after if necessary.

No attention span

mxinden · 2021-06-29T19:19:23Z

🎉 After more than two years, this is ready to be merged. Thanks to everyone involved! 🎉

WIP Initial Kademlia DHT spec.

5378e61

ghost assigned raulk Nov 7, 2018

ghost added the in progress label Nov 7, 2018

raulk requested a review from jhiesey November 7, 2018 18:49

raulk assigned jhiesey Nov 7, 2018

raulk changed the title ~~WIP Initial Kademlia DHT spec~~ WIP Kademlia DHT spec Nov 7, 2018

format kad-dht spec.

ee9734b

jhiesey reviewed Nov 8, 2018

View reviewed changes

vasco-santos reviewed Nov 13, 2018

View reviewed changes

kad-dht/README.md Outdated Show resolved Hide resolved

tomaka suggested changes Nov 14, 2018

View reviewed changes

kad-dht/README.md Outdated Show resolved Hide resolved

kad-dht/README.md Outdated Show resolved Hide resolved

kad-dht/README.md Outdated Show resolved Hide resolved

tomaka previously requested changes Nov 14, 2018

View reviewed changes

kad-dht/README.md Outdated Show resolved Hide resolved

kad-dht/README.md Outdated Show resolved Hide resolved

kad-dht/README.md Outdated Show resolved Hide resolved

kad-dht/README.md Outdated Show resolved Hide resolved

ghost mentioned this pull request Nov 19, 2018

Specs 2.0 & libp2p book #110

Open

16 tasks

raulk mentioned this pull request Nov 28, 2018

Discovery options for phase 0 ethresearch/p2p#7

Open

13 tasks

anacrolix reviewed Dec 21, 2018

View reviewed changes

kad-dht/README.md Show resolved Hide resolved

Update Kademlia DHT spec

8b89dc2

anacrolix reviewed Jan 10, 2019

View reviewed changes

kad-dht/README.md Outdated Show resolved Hide resolved

Stebalien mentioned this pull request Feb 14, 2019

Is the PING message type used? libp2p/go-libp2p-kad-dht#31

Closed

kad-dht/README.md: Fix typo

d742e2e

petar reviewed Jun 2, 2021

View reviewed changes

mxinden reviewed Jun 3, 2021

View reviewed changes

mxinden added 3 commits June 3, 2021 15:18

kad-dht/README: Remove requirement on kbucket data structure

9355a8f

kad-dht/README: Restructure and reword DHT operations section

072360f

kad-dht/README: Seed with k instead of alpha peers

c4d4b53

jacobheun mentioned this pull request Jun 7, 2021

Update the DHT to the latest libp2p spec libp2p/js-libp2p-kad-dht#183

Closed

kad-dht/README: Require algorithms to make progress towards target key

b074091

aschmahmann requested changes Jun 24, 2021

View reviewed changes

mxinden added 6 commits June 25, 2021 11:17

kad-dht/README: Remove /pk special namespace

a065aac

kad-dht/README: Replicate record to closest peers without it

20b3b73

kad-dht/README: Demote validate purity to should

3e13846

kad-dht/README: Do not require storing provider addresses

6ec65b5

kad-dht/README: Remove periodic record pruning section

1dcb218

kad-dht/README: Include bootstrap lookup for oneself

c755a41

mxinden reviewed Jun 25, 2021

View reviewed changes

mxinden added 4 commits June 25, 2021 16:32

kad-dht/README: Make k value recommended instead of default

e9c18bd

kad-dht/README: Always return k closest peers with FIND_NODE

dab4549

kad-dht/README: Extend on reasoning for quorums

324f915

kad-dht/README: Stress republishing to close nodes once more

dbd17a2

aschmahmann approved these changes Jun 25, 2021

View reviewed changes

kad-dht/README: Add disclaimer for bootstrap process

3e6f8f5

mxinden merged commit 5879eb8 into master Jun 29, 2021

mxinden mentioned this pull request Jul 1, 2021

*: Update Kademlia and Circuit Relay v2 links #345

Merged


		### Kademlia routing table

		The data structure backing this system is a k-bucket routing table, closely

		- Getting providers for a given key from the nodes closest to that key via
		`GET_PROVIDERS`.

		- Peer routing

		- Finding the closest nodes to a given key via `FIND_NODE`.

		For performance reasons, a node may prune expired advertisements only
		periodically, e.g. every hour.

		node to be found. `closerPeers` is set in the response; for an exact match
		exactly one `Peer` is returned; otherwise `k` closest `Peer`s are returned.

Add minimal Kademlia DHT spec #108

Add minimal Kademlia DHT spec #108

Conversation

raulk commented Nov 7, 2018 • edited by mxinden Loading

jhiesey left a comment

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

tomaka commented Nov 14, 2018 • edited Loading

tomaka commented Nov 14, 2018

tomaka commented Nov 14, 2018 • edited Loading

raulk commented Nov 14, 2018

tomaka commented Nov 15, 2018 • edited Loading

raulk commented Nov 17, 2018

tomaka commented Nov 28, 2018

raulk commented Dec 17, 2018

jhiesey commented Dec 21, 2018

jhiesey commented Dec 21, 2018

jhiesey commented Jan 10, 2019

jhiesey commented Jan 11, 2019

Mikerah commented Feb 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

mxinden Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

mxinden Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

mxinden commented Jun 8, 2021

petar commented Jun 8, 2021

aschmahmann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aschmahmann Jun 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raulk commented Nov 7, 2018 •

edited by mxinden

Loading

tomaka commented Nov 14, 2018 •

edited

Loading

tomaka commented Nov 14, 2018 •

edited

Loading

tomaka commented Nov 15, 2018 •

edited

Loading

mxinden Jun 3, 2021 •

edited

Loading

mxinden Jun 3, 2021 •

edited

Loading

mxinden Jun 3, 2021 •

edited

Loading

mxinden Jun 3, 2021 •

edited

Loading

mxinden Jun 3, 2021 •

edited

Loading

mxinden Jun 3, 2021 •

edited

Loading

mxinden Jun 3, 2021 •

edited

Loading

mxinden Jun 3, 2021 •

edited

Loading

aschmahmann Jun 25, 2021 •

edited

Loading