Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Circuit Relay Specification: The first iteration #15

Closed
wants to merge 10 commits into from
Closed

The Circuit Relay Specification: The first iteration #15

Changes from 4 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
63897d6
first pass
daviddias Mar 28, 2017
4c5a51c
update addresses examples
daviddias Mar 28, 2017
b74a39e
add error table
daviddias Mar 28, 2017
5ad252d
update wire format section with why notes
daviddias Mar 28, 2017
d65d541
Feat/update relay spec (#16)
dryajov Mar 30, 2017
5105b41
reviewed Overview, Dramatization and Addressing
daviddias May 8, 2017
9ca982b
apply lars CR
daviddias May 9, 2017
1759516
review Wire protocol section from the relay spec
daviddias May 9, 2017
76dfb72
Merge pull request #18 from libp2p/review/relay/wire-protocol
daviddias Jul 8, 2017
a758d06
Merge branch 'master' into feat/update-relay-spec
daviddias Jul 8, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
263 changes: 171 additions & 92 deletions relay/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,66 +2,47 @@

> Circuit Switching in libp2p

Implementations
- [js-libp2p-circuit](https://github.com/libp2p/js-libp2p-circuit) -- status: started
- [go-libp2p-circuit](https://github.com/libp2p/go-libp2p-circuit) -- status: ready-to-start
## Implementations

- [js-libp2p-circuit](https://github.com/libp2p/js-libp2p-circuit)
- [go-libp2p-circuit](https://github.com/libp2p/go-libp2p-circuit)

## Table of Contents

Table of Contents
- [Overview](#overview)
- [Dramatization](#dramatization)
- [Addressing](#addressing)
- [Wire protocol](#wire-protocol)
- [Interfaces](#interfaces)
- [Removing existing relay protocol](#removing-existing-relay-protocol)

- [Implementation Details](#implementation-details)

## Overview

The circuit relay is a means of establishing connectivity between
libp2p nodes (such as IPFS) that wouldn't otherwise be able to connect to each other.
The circuit relay is a means of establishing connectivity between libp2p nodes (such as IPFS) that wouldn't otherwise be able to connect to each other.

This helps in situations where nodes are behind NAT or reverse proxies,
or simply don't support the same transports (e.g. go-ipfs vs. browser-ipfs).
libp2p already has modules for NAT ([go-libp2p-nat](https://github.com/libp2p/go-libp2p-nat)),
but these don't always do the job, just because NAT traversal is complicated.
That's why it's useful to have a simple relay protocol.
This helps in situations where nodes are behind NAT or reverse proxies, or simply don't support the same transports (e.g. go-ipfs vs. browser-ipfs). libp2p already has modules for NAT ([go-libp2p-nat](https://github.com/libp2p/go-libp2p-nat)), but these don't always do the job, just because NAT traversal is complicated. That's why it's useful to have a simple relay protocol.

Unlike a transparent **tunnel**, where a libp2p peer would just proxy a
communication stream to a destination (the destination being unaware of the
original source), a circuit-relay makes the destination aware of the original
source and the circuit followed to establish communication between the two.
This provides the destination side with full knowledge of the circuit which,
if needed, could be rebuilt in the opposite direction.
Unlike a transparent **tunnel**, where a libp2p peer would just proxy a communication stream to a destination (the destination being unaware of the original source), a circuit-relay makes the destination aware of the original source and the circuit followed to establish communication between the two. This provides the destination side with full knowledge of the circuit which, if needed, could be rebuilt in the opposite direction.

Apart from that, this relayed connection behaves just like a regular
connection would, but over an existing swarm stream with another peer
(instead of e.g. TCP.): One node asks a relay node to connect to another node
on its behalf. The relay node shortcircuits its streams to the two nodes,
and they are then connected through the relay.
Apart from that, this relayed connection behaves just like a regular connection would, but over an existing swarm stream with another peer (instead of e.g. TCP.): One node asks a relay node to connect to another node on its behalf. The relay node shortcircuits its streams to the two nodes, and they are then connected through the relay.

Relayed connections are end-to-end encrypted just like regular connections.

The circuit relay is both a tunneled transport and a mounted swarm protocol.
The transport is the means of ***establishing*** and ***accepting*** connections,
and the swarm protocol is the means to ***relaying*** connections.
The transport is the means of ***establishing*** and ***accepting*** connections, and the swarm protocol is the means to ***relaying*** connections.

```
+-------+ /ip4/.../tcp/.../ws/p2p/QmRelay +---------+ /ip4/.../tcp/.../p2p/QmTwo +-------+
| QmOne | <------------------------------------> | QmRelay | <-----------------------------------> | QmTwo |
+-------+ (/libp2p/relay/circuit multistream) +---------+ (/libp2p/relay/circuit multistream) +-------+
^ +-----+ ^
| | | |
| /p2p-circuit/QmTwo | | |
+--------------------------------------------+ +-------------------------------------------+
+-----+ /ip4/.../tcp/.../ws/p2p/QmRelay +-------+ /ip4/.../tcp/.../p2p/QmTwo +-----+
|QmOne| <------------------------------------>|QmRelay|<----------------------------------->|QmTwo|
+-----+ (/libp2p/relay/circuit multistream) +-------+ (/libp2p/relay/circuit multistream) +-----+
^ +-----+ ^
| /p2p-circuit/QmTwo | | |
+-----------------------------------------+ +-----------------------------------------+
```

Note: we're using the `/p2p` multiaddr protocol instead of `/ipfs` in this document.
`/ipfs` is currently the canonical way of addressing a libp2p or IPFS node,
but given the growing non-IPFS usage of libp2p, we'll migrate to using `/p2p`.

Note: at the moment we're not including a mechanism for discovering relay nodes.
For the time being, they should be configured statically.
Note: we're using the `/p2p` multiaddr protocol instead of `/ipfs` in this document. `/ipfs` is currently the canonical way of addressing a libp2p or IPFS node, but given the growing non-IPFS usage of libp2p, we'll migrate to using `/p2p`.

Note: at the moment we're not including a mechanism for discovering relay nodes. For the time being, they should be configured statically.

## Dramatization

Expand All @@ -84,91 +65,189 @@ Scene 2:
- QmOne tries to connect via relaying, because it shares this transport with QmTwo.
- A lively and prolonged dialogue ensues.


## Addressing

`/p2p-circuit` multiaddrs don't carry any meaning of their own.
They need to encapsulate a `/p2p` address, or
be encapsulated in a `/p2p` address, or both.
`/p2p-circuit` multiaddrs don't carry any meaning of their own. They need to encapsulate a `/p2p` address, or be encapsulated in a `/p2p` address, or both.

As with all other multiaddrs, encapsulation of different protocols determines which metaphorical tubes to connect to each other.

A `/p2p-circuit` circuit address, is formated following:

`/p2p-circuit[<relay peer multiaddr>]<destination peer multiaddr>`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe put slashes in here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, after @lgierth clarifications this should be <relay peer multiaddr>/p2p-circuit/<destination peer multiaddr>


This opens the room for multiple hop relay, where the first relay is encapsulated in the seconf relay multiaddr, such as:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: s/seconf/second/


As with all other multiaddrs, encapsulation of different protocols
determines which metaphorical tubes to connect to each other.
`/p2p-circuit/p2p-circuit/<first relay>/<first hop multiaddr>/<destination peer multiaddr>`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it makes a lot more sense to do something like - /p2p-circuit<first relay>/<first hop multiaddr>/p2p-circuit<second relay>/<second hop multiaddr>/<destination peer multiaddr>

Copy link
Member

@dryajov dryajov Mar 29, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to reformat this example to match @lgierth clarifications /p2p-circuit/p2p-circuit/<first relay>/<first hop multiaddr>/<destination peer multiaddr>

I think the above is intended to read as follows:

  • <first relay>/p2-pcircuit/<second relay>/p2p-circuit/ipfs/QmDst
    • use the first hop <first relay>/p2-pcircuit
      • either peer routed or using its explicit transport if specified
    • use the second hop <second relay>/p2p-circuit
      • either peer routed or using its explicit transport if specified
        * circuit /ipfs/QmDst over the < first relay > <-> < second relay > pipe


A few examples:

Using any relay available:

- `/p2p-circuit/p2p/QmTwo`
- Dial QmTwo, through any available relay node.
- The relay node will use peer routing to find an address for QmTwo.
- `/p2p/QmRelay/p2p-circuit/p2p/QmTwo`
- Dial QmTwo, through QmRelay.
- Use peer routing to find an address for QmRelay.
- The relay node will also use peer routing, to find an address for QmTwo.
- Dial QmTwo, through any available relay node (or find one node that can relay).
- The relay node will use peer routing to find an address for QmTwo if it doesn't have a direct connection.
- `/p2p-circuit/ip4/../tcp/../p2p/QmTwo`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be :

/ip4/../tcp/../p2p-circuit/p2p/QmTwo

- Dial QmTwo, through any available relay node,
but force the relay node to use the encapsulated `/ip4` multiaddr for connecting to QmTwo.
- We'll probably not support forced addresses for now, just because it's complicated.
- `/ip4/../tcp/../p2p/QmRelay/p2p-circuit`
- Listen for connections relayed through QmRelay.

Specify a relay:

- `/p2p-circuit/p2p/QmRelay/p2p/QmTwo`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be:

/p2p/QmRelay/p2p-circuit/p2p/QmTwo

- Dial QmTwo, through QmRelay.
- Use peer routing to find an address for QmRelay.
- The relay node will also use peer routing, to find an address for QmTwo.
- `/p2p-circuit/ip4/../tcp/../p2p/QmRelay/p2p/QmTwo`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be:

/ip4/../tcp/../p2p/QmRelay/p2p-circuit/p2p/QmTwo

- Dial QmTwo, through QmRelay.
- Includes info for connecting to QmRelay.
- Also makes QmRelay available for relayed dialing, based on how listeners currently relate to dialers.
- `/p2p/QmRelay/p2p-circuit`
- Same as previous example, but use peer routing to find an address for QmRelay.
- The relay node will use peer routing to find an address for QmTwo.

Double relay:

- `/p2p-circuit/p2p/QmTwo/p2p-circuit/p2p/QmThree`
- Dial QmThree, through a relayed connection to QmTwo.
- The relay nodes will use peer routing to find an address for QmTwo and QmThree.
- We'll probably not support nested relayed connections for now, there are edge cases to think of.
- `/ip4/../tcp/../p2p/QmRelay/p2p-circuit/p2p/QmTwo`
- Dial QmTwo, through QmRelay.

--

?? I don't understand the usage of the following:

- `/ip4/../tcp/../p2p/QmRelay/p2p-circuit`
- Listen for connections relayed through QmRelay.
- Includes info for connecting to QmRelay.
- The relay node will use peer routing to find an address for QmTwo.
- Also makes QmRelay available for relayed dialing, based on how listeners currently relate to dialers.
- `/p2p/QmRelay/p2p-circuit`
- Same as previous example, but use peer routing to find an address for QmRelay.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm, these addresses exist to have a way to make the node always dial to the relay peer (punch out), correct?


?? I believe we don't need this one:
- `/p2p-circuit`
- Use relay discovery to find a suitable relay node. (Neither specified nor implemented.)
- Listen for relayed connections.
- Dial through the discovered relay node for any `/p2p-circuit` multiaddr.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, the act of 'enabling relayed connections' is a config value and not a multiaddr. Do you agree @lgierth ?

Copy link
Member

@dryajov dryajov Mar 29, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@diasdavid @lgierth @whyrusleeping

I think it depends on how you look at relay. To me, relay is just another transport, hence making it behave like any other transport makes sense - here, I'm specifically talking about the dialer/listener (/stop) parts of the relay. The circuit-relay itself, can/should be controlled by config options.

Example:

Swarm: {
  Addresses: [
    '/p2p-circuit/<multiaddr of the Relay>' // or just plain /p2p-circuit enables circuit for listening (/stop) and dialing
  ]
},
Relay: {
  Circuit: {
    Enabled: false, // enable/disable circuit-relaying
    list-peers: false // I think this will be moved to its own proto that implements `ls`
    Proactive: false // should be renamed `Active`? (Active/Passive) naming
  }
}

Edit:

Thinking about this a bit more, I think we should enable circuit transport by default, no need for an explicit p2p-circuit in the swarm addrs.


TODO: figure out forced addresses.
TODO: figure out nested relayed connections.
TODO: figure out forced addresses. -> what is forced addresses?
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgierth Can you elaborate on what "forced addresses" are?


## Wire format

The wire format (or codec) is named `/ipfs/relay/circuit` and is simple.
A variable-length header consisting of two length-prefixed multiaddrs
is followed by a bidirectional stream of arbitrary data,
and the eventual closing of the stream.
### Overview

#### Setup

Peers involved:
- A, B, R
- A wants to connect to B, but needs to relay through R

#### Assumptions

A has connection to R, R has connection to B

#### Process

- A opens new stream `sAR` to R using protocol RELAY
- A writes Bs multiaddr `/ipfs/QmB` on `sAR`
- R receives stream `sAR` and reads `/ipfs/QmB` from it.
- R opens a new stream `sRB` to B using protocol RELAY
- R writes `/ipfs/QmB` on `sRB`
- B receives stream `sRB` and reads `/ipfs/QmB` from it.
- B sees that the multiaddr it read is its own and chooses to handle this stream as an endpoint instead of attempting to relay further
- TODO: step for R to send back status code to A
Copy link
Member

@dryajov dryajov Mar 29, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on status codes:

  • R doesn't send any success codes to A, only failure codes
    • R would potentially also send err codes from B if the circuit is not yet established
  • A would get a success or failure code from B once the circuit is established

- R now pipes `sAR` and `sRB` together
- TODO: step for B to send back status code to A
- B passes stream to `NewConnHandler` to be handled like any other new incoming connection

### Under the microscope

Peer A wants to connect to peer B through peer R.

`maddrA` is peer A's multiaddr
`maddrR` is peer R's multiaddr
`maddrB` is peer B's multiaddr
`maxAddrLen` is arbitrarily 1024

#### Function definitions
##### Process for reading a multiaddr
We define `readLpMaddr` to be the following:

Read a Uvarint `V` from the stream. If the value is higher
than `maxAddrLen`, (write an error message back?) close the
stream and halt the relay attempt.

Then read `V` bytes from the stream and checks if its a valid multiaddr.
If it is not a valid multiaddr (write an error back?) close the stream and return.

#### Opening a relay
Peer A opens a new stream to R on the 'hop' protocol and writes:
```
<src><dst><data>

^ ^ ^
| | |
| | +-- bidirectional data stream
| | (usually /multistream-select in the case of /p2p multiaddrs)
| |
| +------- multiaddr of the listening node
|
+------------ multiaddr of the dialing node
<uvarint len(maddrA)><madddrA><uvarint len(maddrB)><madddrB>
```

After getting a stream to the relay node from its libp2p swarm,
the dialing transport writes the header to the stream.
The relaying node reads the header, gets a stream to the destination node,
then writes the header to the destination stream and shortcircuits the two streams.
It then waits for a response in the form of:
```
<uvarint error code><uvarint msglength><message>
```

Once it receives that, it checks if the status code is `OK`. If it is, it passes the new connection to its `NewConnHandler`.
Otherwise, it returns the error message to the caller.

### 'hop' protocol handler
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By hop, you mean /libp2p/circuit/relay ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, i mean /libp2p/circuit/relay/1.0.0/hop


Peer R receives a new stream on the 'hop' protocol.
It then calls `readLpMaddr` twice. The first value is `<src>` and the second is `<dst>`.
Peer R checks to make sure that `<src>` matches the remote peer of the stream its reading
from. If it does not match, it (writes an error back?) closes the stream and halts the relay attempt.

Peer R checks if `<dst>` refers to itself, if it does, it (writes an error back?) closes the stream and halts the relay attempt.
Peer R then checks if it has an open connection to the peer specified by `<dst>`.
If it does not, and the relay is not an "active" relay it (writes an error back) closes the stream, and halts the relay attempt.
If R does not have a connection to `<dst>`, and it *is* an "active" relay, it attempts to connect to `<dst>`.
If this connection succeeds it continues, otherwise it (writes back an error) closes the stream, and halts the relay attempt.
R now opens a new stream to B with the 'stop' relay protocol ID, and writes:
```
<uvarint len(maddrA)><madddrA><uvarint len(maddrB)><madddrB>
```

After this, R simply pipes the stream from A and the stream it opened to B together. R's job is complete.

### 'stop' protocol handler

Peer B receives a new stream on the 'stop' protocol. It then calls `readLpMaddr` twice on this stream.
The first value is `<src>` and the second value is `<dst>`. Any error from those calls should be written back accordingly.

B now verifies that `<dst>` matches its peer ID. It then also checks that `<src>` is valid. It uses src as the
'remote addr' of the new 'incoming relay connection' it will create.

Peer B now writes back a message of the form:
```
<uvarint 'OK'><uvarint len(msg)><string "OK">
```

And passes the relayed connection into its `NewConnHandler`.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused by this stop handler. What case is it enabling exactly?

Copy link
Member

@dryajov dryajov Mar 28, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thats the end point in the relay flow, hop is the relay listener, stop listens for relayed connections coming from hop. This is needed because you can be both a relay and listener on the same node - hence hop and stop sub protos.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood.


### Error table

Each relayed connection corresponds to two multistreams,
one between QmOne and QmRelay, the other between QmRelay and QmTwo.
This is a table of error codes and sample messages that may occur during a relay setup. Codes in the 200 range are returned by the relay node. Codes in the 300 range are returned by the destination node.

Implementation details:
- The relay node has the `Swarm.EnableRelaying` config option enabled
- The relay node allows only one relayed connection between any two nodes.
- The relay node validates the `src` header field.
- The listening node validates the `dst` header field.

| Code | Message | Meaning |
| ----- |:-----------------:|:----------:|
| 100 | OK | Relay was setup correctly |
| 220 | "src address too long" | |
| 221 | "dst address too long" | |
| 250 | "failed to parse src addr: no such protocol ipfs" | The `<src>` multiaddr in the header was invalid |
| 251 | "failed to parse dst addr: no such protocol ipfs" | The `<dst>` multiaddr in the header was invalid |
| 260 | "passive relay has no connection to dst" | |
| 261 | "active relay could not connect to dst: connection refused" | relay could not form new connection to target peer |
| 262 | "could not open new stream to dst: BAD ERROR" | relay has connection to dst, but failed to open a new stream |
| 270 | "<dst> does not support relay" | |
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really want to use 2XX+3XX for errors? Can we follow the pattern laid by HTTP:

- 1×× Informational
- 2×× Success
- 3×× Redirection
- 4×× Client Error
- 5×× Server Error

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍HTTP codes are well known, which makes them more intuitive and expected.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesnt map exactly though, i'm fine moving the 100 OK to 200 OK.

I guess we could map all the 200 errors to 400 'client' errors, and map all the 300 errors to 500 'server' errors

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@whyrusleeping exactly what I was hoping for ❤️

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@diasdavid @whyrusleeping

After, re-reading the table... They don't really map to HTTP at all... 200s are relay codes and 300 are dst errors. So, I think 1xx, 2xx, 3xx make sense here.

| 320 | "src address too long" | |
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From IRC:

14:13 <@daviddias> I don't understand why the 'send the src' addr keeps appearing as a thing for relay
16:05 <+dryajov> diasdavid re srs addr: I dont think we need it anymore, you are right. We can always do getPeerInfo on the circuited conn and should be able to get the src addrs. It would only make sense, if the full relay addr is sent so that the relay can be rebuilt from the dst to the src, if that ever makes sense?
16:23 <@daviddias> " It would only make sense, if the full relay addr is sent so that the relay can be rebuilt from the dst to the src, if that ever makes sense?" even in that case, we can always verify who dialed us with the getPeerInfo call
16:26 <+dryajov> True, that would be a swarm addr and should come back with the getPeerInfo. We don't need it.

| 321 | "dst address too long" | |
| 350 | "failed to parse src addr: no such protocol ifps" | The `<src>` multiaddr in the header was invalid |
| 351 | "failed to parse dst addr: no such protocol ifps" | The `<dst>` multiaddr in the header was invalid |

## Interfaces

As explained above, the relay is both a transport (`tpt.Transport`)
and a mounted stream protocol (`p2pnet.StreamHandler`).
In addition it provides a means of specifying relay nodes to listen/dial through.
As explained above, the relay is both a transport (`tpt.Transport`) and a mounted stream protocol (`p2pnet.StreamHandler`). In addition it provides a means of specifying relay nodes to listen/dial through.

TODO: the usage of p2pnet.StreamHandler is a little bit off, but it gets the point across.

Expand All @@ -192,8 +271,9 @@ type CircuitRelay interface {
fund NewCircuitRelay(h p2phost.Host)
```

## Implementation details

### Removing existing relay protocol
### Removing existing relay protocol in Go

Note that there is an existing swarm protocol colloqiually called relay.
It lives in the go-libp2p package and is named `/ipfs/relay/line/0.1.0`.
Expand All @@ -208,5 +288,4 @@ It lives in the go-libp2p package and is named `/ipfs/relay/line/0.1.0`.
- Capable of *accepting* connections, and *relaying* connections.
- Not capable of *connecting* via relaying.

Since the existing protocol is incomplete, insecure, and certainly not used,
we can safely remove it.
Since the existing protocol is incomplete, insecure, and certainly not used, we can safely remove it.