Skip to content

Commit

Permalink
link to registry; reword the spec
Browse files Browse the repository at this point in the history
  • Loading branch information
@sukunrt
sukunrt committed Nov 19, 2024
1 parent 3060cd0 commit 196f49e
Show file tree
Hide file tree
Showing 3 changed files with 68 additions and 77 deletions.
35 changes: 18 additions & 17 deletions connections/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,22 +22,23 @@ and spec status.
## Table of Contents

- [Connection Establishment in libp2p](#connection-establishment-in-libp2p)
- [Table of Contents](#table-of-contents)
- [Overview](#overview)
- [Definitions](#definitions)
- [Protocol Negotiation](#protocol-negotiation)
- [multistream-select](#multistream-select)
- [Upgrading Connections](#upgrading-connections)
- [Opening New Streams Over a Connection](#opening-new-streams-over-a-connection)
- [Practical Considerations](#practical-considerations)
- [Interoperability](#interoperability)
- [State Management](#state-management)
- [Peer Metadata Storage](#peer-metadata-storage)
- [Connection Limits](#connection-limits)
- [Supported protocols](#supported-protocols)
- [Connection Lifecycle Events](#connection-lifecycle-events)
- [Hole punching](#hole-punching)
- [Future Work](#future-work)
- [Table of Contents](#table-of-contents)
- [Overview](#overview)
- [Definitions](#definitions)
- [Protocol Negotiation](#protocol-negotiation)
- [multistream-select](#multistream-select)
- [Upgrading Connections](#upgrading-connections)
- [Inlining Muxer Negotiation](#inlining-muxer-negotiation)
- [Opening New Streams Over a Connection](#opening-new-streams-over-a-connection)
- [Practical Considerations](#practical-considerations)
- [Interoperability](#interoperability)
- [State Management](#state-management)
- [Peer Metadata Storage](#peer-metadata-storage)
- [Connection Limits](#connection-limits)
- [Supported protocols](#supported-protocols)
- [Connection Lifecycle Events](#connection-lifecycle-events)
- [Hole punching](#hole-punching)
- [Future Work](#future-work)

## Overview

Expand Down Expand Up @@ -181,7 +182,7 @@ If the peers agree on a protocol, multistream-select's job is done, and future
traffic over the channel will adhere to the rules of the agreed-upon protocol.

If a peer receives a `"na"` response to a proposed protocol id, they can either
try again with a different protocol id or close the channel.
try again with a different protocol id or close the channel with error code `PROTOCOL_NEGOTIATION_FAILED` as defined in [libp2p error codes](./../error-codes/README.md) spec


## Upgrading Connections
Expand Down
72 changes: 32 additions & 40 deletions error-codes/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,80 +17,69 @@ When closing a connection or resetting a stream, it's useful to provide the peer
with a code that explains the reason for the closure. This enables the peer to
better respond to the abrupt closures. For instance, it can implement a backoff
strategy to retry _only_ when it receives a `RATE_LIMITED` error code. An error
code doesn't always indicate an error condition. For example, a connection may
be closed because a connection to the same peer over a better transport is
available.
code doesn't always indicate an error condition. For example, a node can terminate an idle connection, or close a connection because a connection to the same peer over a better transport is available. In both these cases, it can signal an appropriate error code to the other end.

## Semantics
Error codes are unsigned 32-bit integers. The range 0 to 10000 is reserved for
libp2p errors. Application specific errors can be defined by protocols from
integers outside of this range. Implementations supporting error codes MUST
provide the error code provided by the other end to the application.

Error codes provide a best effort guarantee that the error will be propagated to
the application layer. This provides backwards compatibility with older nodes,
allows smaller implementations, and using transports that don't provide a
mechanism to provide an error code. For example, Yamux has no equivalent of
QUIC's [STOP_SENDING](https://www.rfc-editor.org/rfc/rfc9000.html#section-3.5-4)
frame that would tell the peer that the node has stopped reading. So there's no
way of signaling an error while closing the read end of the stream on a yamux
connection.
integers outside of this range. Error Codes can be signaled on Closing a connection or on resetting a Stream.

From an application perspective, error codes provide a best effort guarantee. On resetting a libp2p stream or closing a connection with an error code, the error code may or may not be delivered to the application on the remote end. The specifics depend on the transport used. For example, WebTransport doesn't support error codes at all, while WebRTC doesn't support Connection Close error codes, but supports Stream Reset error codes.

### Connection Close and Stream Reset Error Codes
Error codes are defined separately for Connection Close and Stream Reset. Stream
Reset errors are from the range 0 to 5000 and Connection Close errors are from
the range 5001 to 10000. Having separate errors for Connection Close and stream
reset requires some overlap between the error code definitions but provides more
information to the receiver. Receiving a `Bad Request: Connection Closed` error
on reading from a stream also tells the receiver that no more streams can be
started on the same connection. Implementations MUST provide the Connection
Close error code on streams that are reset as a result of remote closing the
connection.

For stream resets, when the underlying transport supports it, implementations
SHOULD allow sending an error code on both closing the read side of the stream,
and resetting the write side of the stream.

## Libp2p Reserved Error Codes
Error codes are defined separately for Connection Close and Stream Reset. The namespace doesn't overlap as it is clear from the context whether the stream was reset by the other end, or it was reset as a result of a connection close.
Implementations MUST provide the Connection Close error code on streams that are reset as a result of remote closing the connection.

Libp2p streams are reset unilaterally, calling `Reset` on a stream resets both the read and write end of a stream. For transports, like QUIC, which support cancelling the read and write ends of the stream separately, implementations MAY provide the ability to signal error codes separately on resetting either end.

## Error Codes Registry
Libp2p connections are shared by multiple applications. The same connection used in the dht may be used for gossip sub, or for any other application. Any of these applications can close the underlying connection on an error, resetting streams used by the other applications. To correctly distinguish which application closed the connection, Connection Close error codes are allocated to applications from a central registry.

For simplicity, we manage both Connection Close and Stream Reset error codes from a central registry. The libp2p error codes registry is at: https://github.com/libp2p/error-codes/

### Libp2p Reserved Error Codes
Error code 0 signals that no error code was provided. Implementations MUST handle closing a connection with error code 0 as they handle closing a connection with no error code, and resetting a stream with error code 0 as they handle resetting a stream with no error.

Error codes from 1 to 100 are reserved for transport errors. These are used by the transports to terminate connections on transport errors.

Error codes from 100 - 10000 are reserved for libp2p. This includes multistream error codes, as it is necessary for libp2p connection establishment over TCP, but not kad-dht or gossip-sub error codes.

see [Libp2p error codes](./libp2p-error-codes.md) for the libp2p reserved error
codes.

## Wire Encoding
Different transports will encode the 32-bit error code differently.
## Transport Specification and Wire Encoding
Different transports will encode the 32-bit error code differently on the wire. They also provide different semantics: Webtransport doesn't define error codes, WebRTC doesn't support Connection Close error codes, Yamux error codes on Connection Close cannot be reliably sent over the wire.

### QUIC
QUIC provides the ability to send an error on closing the read end of the
stream, resetting the write end of the stream and on closing the connection.

For stream resets, the error code MUST be sent on the `RESET_STREAM` or the
`STOP_SENDING` frame using the `Application Protocol Error Code` field as per
For stream resets, the error code MUST be sent on `RESET_STREAM` and `STOP_SENDING` frames using the `Application Protocol Error Code` field as per
the frame definitions in the
[RFC](https://www.rfc-editor.org/rfc/rfc9000.html#name-reset_stream-frames).

For Connection Close, the error code MUST be sent on the CONNECTION_CLOSE frame
For Connection Close, the error code MUST be sent on `CONNECTION_CLOSE` frame
using the Error Code field as defined in the
[RFC](https://www.rfc-editor.org/rfc/rfc9000.html#section-19.19-6.2.1).

### Yamux
Yamux has no `STOP_SENDING` frame, so there's no way to signal an error on
closing the read side of the stream.
Yamux streams are reset unilaterally. Receiving a stream frame with `RST` flag set resets both the read and write end of the stream. So, there's no way to separately signal error code on closing the read end of the stream, or resetting the write end of the stream.

For Connection Close, the 32-bit Length field is interpreted as the error code.

For Stream Resets, the error code is sent in the `Window Update` frame, with the
32-bit Length field interpreted as the error code. See [yamux spec
extension](https://github.com/libp2p/specs/pull/622).

Note: On TCP connections with `SO_LINGER` set to 0, the Connection Close error
code may not be delivered.
Connection Close error code delivery to the other end depends on the OS TCP implementation and the TCP options used for the socket. In particular, when `SO_LINGER` TCP option is set to 0 and the implementation closes the connection immediately after writing the error code containing frame, the error code may not be delivered.

### WebRTC
There is no way to provide any information on closing a peer connection in
WebRTC. Providing error codes on Connection Close will be taken up in the
future.

For Stream Resets, the error code can be sent in the `errorCode` field of the
WebRTC message with `flag` set to `RESET_STREAM` .
WebRTC message with `flag` set to `RESET_STREAM`.

### WebTransport
Error codes for WebTransport will be introduced when browsers upgrade to draft-9
Expand All @@ -102,3 +91,6 @@ as the latest WebTransport draft,
[draft-9](https://www.ietf.org/archive/id/draft-ietf-webtrans-http3-02.html#section-4.3-2)
allows for a 4-byte error code to be sent on stream resets, we will introduce
error codes over WebTransport later.

### Multistream Select
Multistream-Select is used to negotiate Security protocol for TCP connections before a stream muxer has been selected. There's only one error code defined for such cases, `PROTOCOL_NEGOTIATION_FAILED`. To encode this error, send the string `101` prefixed with the length and close the TCP connection.
38 changes: 18 additions & 20 deletions error-codes/libp2p-error-codes.md
View file
Original file line number Diff line number Diff line change
@@ -1,32 +1,30 @@
# Libp2p error codes

## TODO!
make this a CSV

## Connection Error Codes

| Name | Code | Description |
| --- | --- | --- |
| NO_ERROR | 0 | No reason provided for disconnection. This is equivalent to closing a connection or resetting a stream without any error code. |
| Reserved For Transport | 0x1 - 0x1000 | Reserved for transport level error codes. |
| GATED | 0x1001 | The connection was gated. Most likely the IP / node is blacklisted. |
| RESOURCE_LIMIT_EXCEEDED | 0x1002 | Rejected because we ran into a resource limit. Implementations MAY retry with a backoff |
| RATE_LIMITED | 0x1003 | Rejected because the connection was rate limited. Implementations MAY retry with a backoff |
| PROTOCOL_VIOLATION | 0x1004 | Peer violated the protocol |
| SUPPLANTED | 0x1005 | Connection closed because a connection over a better tranpsort was available |
| GARBAGE_COLLECTED | 0x1006 | Connection was garbage collected |
| SHUTDOWN | 0x1007 | The node is going down |
| PROTOCOL_NEGOTIATION_FAILED | 0x1008 | Rejected because we couldn't negotiate a protocol |
| Reserved For Transport | 1 - 100 | Reserved for transport level error codes. |
| PROTOCOL_NEGOTIATION_FAILED | 101 | Rejected because we couldn't negotiate a protocol. Used by multistream select for security negotiation |
| RESOURCE_LIMIT_EXCEEDED | 102 | Rejected because we ran into a resource limit. Implementations MAY retry with a backoff |
| RATE_LIMITED | 103 | Rejected because the connection was rate limited. Implementations MAY retry with a backoff |
| PROTOCOL_VIOLATION | 104 | Peer violated the protocol |
| SUPPLANTED | 105 | Connection closed because a connection over a better tranpsort was available |
| GARBAGE_COLLECTED | 106 | Connection was garbage collected |
| SHUTDOWN | 107 | The node is shutting down |
| GATED | 108 | The connection was gated. Most likely the IP / node is blacklisted. |


## Stream Error Codes

| Name | Code | Description |
| --- | --- | --- |
| NO_ERROR | 0 | No reason provided for disconnection. This is equivalent to resetting a stream without any error code. |
| Reserved For Transport | 0x1 - 0x1000 | Reserved for transport level error codes. |
| PROTOCOL_NEGOTIATION_FAILED | 0x1001 | Rejected because we couldn't negotiate a protocol |
| RESOURCE_LIMIT_EXCEEDED | 0x1002 | Connection rejected because we ran into a resource limit. Implementations MAY retry with a backoff |
| RATE_LIMITED | 0x1003 | Rejected because the connection was rate limited. Implementations MAY retry with a backoff |
| BAD_REQUEST | 0x1004 | Rejected because the request was invalid |
| PROTOCOL_VIOLATION | 0x1005 | Rejected because the stream protocol was violated. MAY be used interchangably with `BAD_REQUEST` |
| Reserved For Transport | 1 - 100 | Reserved for transport level error codes. |
| PROTOCOL_NEGOTIATION_FAILED | 101 | Rejected because we couldn't negotiate a protocol. Used by multistream select|
| RESOURCE_LIMIT_EXCEEDED | 102 | Connection rejected because we ran into a resource limit. Implementations MAY retry with a backoff |
| RATE_LIMITED | 103 | Rejected because the connection was rate limited. Implementations MAY retry with a backoff |
| PROTOCOL_VIOLATION | 104 | Rejected because the stream protocol was violated. MAY be used interchangably with `BAD_REQUEST` |
| SUPPLANTED | 105 | Resetted because a better transport is available for the stream |
| GARBAGE_COLLECTED | 106 | Connection was garbage collected |
| SHUTDOWN | 107 | The node is shutting down |
| GATED | 108 | The stream was gated. Most likely the IP / node is blacklisted. |

0 comments on commit 196f49e

Please sign in to comment.