docs: update limits doc (#1456)

Incorporate feedback from #1421

Co-authored-by: Steve Loeppky <biglep@protocol.ai>

Author: achingbrain

doc/LIMITS.md

@@ -2,100 +2,133 @@
 
 In order to prevent excessive resource consumption by a libp2p node it's important to understand limits are applied and how to tune them to the needs of your application.
 
+This is important for [DoS](https://en.wikipedia.org/wiki/Denial-of-service_attack) attack mitgation - there is a more holistic discussion and general advice on that topic at [the main libp2p docs website](https://docs.libp2p.io/reference/dos-mitigation/).
+
 ## Table of contents <!-- omit in toc -->
 
 - [Connection limits](#connection-limits)
+- [Closing connections](#closing-connections)
 - [Inbound connection threshold](#inbound-connection-threshold)
 - [Data transfer and Event Loop limits](#data-transfer-and-event-loop-limits)
 - [Stream limits](#stream-limits)
   - [Mplex](#mplex)
   - [Yamux](#yamux)
   - [Protocol limits](#protocol-limits)
-- [Closing connections](#closing-connections)
 - [Transport specific limits](#transport-specific-limits)
   - [TCP](#tcp)
 - [Allow/deny lists](#allowdeny-lists)
+- [How much memory will be used for buffering?](#how-much-memory-will-be-used-for-buffering)
 
 ## Connection limits
 
-It's possible to limit the amount of incoming and outgoing connections a node is able to make.  When this limit is reached and an attempt to open a new connection is made, existing connections may be closed to make room for the new connection.
+It's possible to limit the total amount of connections a node is able to make (combining incoming and outgoing). When this limit is reached and an attempt to open a new connection is made, existing connections may be closed to make room for the new connection (see [Closing connections][#closing-connections]).
+
+* Note: there currently isn't a way to specify different limits for incoming vs. outgoing. Connection limits are applied across both incoming and outgoing connections combined. There is a backlog item for this [here](https://github.com/libp2p/js-libp2p/issues/1508).
 
 We can also limit the number of connections in a "pending" state. These connections have been opened by a remote peer but peer IDs have yet to be exchanged and/or connection encryption and multiplexing negotiated. Once this limit is hit further connections will be closed unless the remote peer has an address in the [allow list](#allowdeny-lists).
 
-```js
+All fields are optional. The default values are defined in [src/connection-manager/index.ts](https://github.com/libp2p/js-libp2p/blob/master/src/connection-manager/index.ts) - please see that file for the current values.
+
+```ts
 const node = await createLibp2pNode({
   connectionManager: {
     /**
      * The total number of connections allowed to be open at one time
      */
-    maxConnections: 200,
+    maxConnections: number
 
     /**
      * If the number of open connections goes below this number, the node
-     * will try to connect to nearby peers from the peer store
+     * will try to connect to randomly selected peers from the peer store
      */
-    minConnections: 20,
+    minConnections: number
 
     /**
      * How many connections can be open but not yet upgraded
      */
-    maxIncomingPendingConnections: 10
+    maxIncomingPendingConnections: number
   }
 })
 ```
 
+## Closing connections
+
+When choosing connections to close the connection manager sorts the list of connections by the value derived from the tags given to each peer. The values of all tags are summed and connections with lower valued peers are eligible for closing first.
+
+```js
+// tag a peer
+await libp2p.peerStore.tagPeer(peerId, 'my-tag', {
+  value: 50, // 0-100 is the typical value range
+  ttl: 1000 // optional field, this tag will be deleted after this many ms
+})
+```
+
 ## Inbound connection threshold
 
-To prevent individual peers from opening multiple connections to a node, an `inboundConnectionThreshold` is configurable. This is the number of connections per second an individual remote host can open to a node, once this threshold is crossed all further connections opened by that host will be rejected.
+To prevent individual peers from opening multiple connections to a node, an `inboundConnectionThreshold` is configurable. This is the number of connections per second an individual peer can open to a node, once this threshold is crossed all further connections opened by that peer will be rejected until the threshold resets in the next second.
 
+All fields are optional. The default values are defined in [src/connection-manager/index.ts](https://github.com/libp2p/js-libp2p/blob/master/src/connection-manager/index.ts) - please see that file for the current values.
 
-```js
+```ts
 const node = await createLibp2pNode({
   connectionManager: {
     /**
      * A remote peer may attempt to open up to this many connections per second,
      * any more than that will be automatically rejected
      */
-    inboundConnectionThreshold: 5
+    inboundConnectionThreshold: number
   }
 })
 ```
 
 ## Data transfer and Event Loop limits
 
-If metrics are enabled the node will track the amount of data being sent to and from the network. If the amount sent is over the threshold connections will be trimmed to free up resources.  The default amount is `Ininity` so this must be explicitly enabled.
+If metrics are enabled the node will track the amount of data being sent to and from the network. If the rate of data sent is over the threshold connections will be trimmed to free up resources. The default rate is `Ininity` so this must be explicitly enabled.
 
 Connections may also be trimmed if [event loop](https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick) latency exceeds the configured limit.
 
-```js
+All fields are optional. The default values are defined in [src/connection-manager/index.ts](https://github.com/libp2p/js-libp2p/blob/master/src/connection-manager/index.ts) - please see that file for the current values.
+
+```ts
 const node = await createLibp2pNode({
   metrics: {
     enabled: true
   },
   connectionManager: {
     /**
      * If the node transfers more than this amount of data in bytes/second
-     * low value connections may be closed
+     * connections to untagged peers or those not in the allow list may be
+     * closed.
+     *
+     * It is bytes per second.
      */
-    maxData: 1024 * 1024,
+    maxData: number
 
     /**
      * If the node sends more than this amount of data in bytes/second
-     * low value connections may be closed
+     * connections to untagged peers or those not in the allow list may be
+     * closed.
+     *
+     * It is bytes per second.
      */
-    maxSentData: 1024 * 1024
+    maxSentData: number
 
     /**
      * If the node receives more than this amount of data in bytes/second
-     * low value connections may be closed
+     * connections to untagged peers or those not in the allow list may be
+     * closed.
+     *
+     * It is bytes per second.
      */
-    maxReceivedData: 1024 * 1024,
+    maxReceivedData: number
 
     /**
-     * If the event loop takes longer than this many ms to run,  low value
-     * connections may be closed
+     * If the event loop takes longer than this many ms to run, connections
+     * to untagged peers or those not in the allow list may be closed.
+     *
+     * It is milliseconds.
      */
-    maxEventLoopDelay: 1000
+    maxEventLoopDelay: number
   }
 })
 ```
@@ -108,85 +141,88 @@ These settings are done on a per-muxer basis, please see the README of the relev
 
 ### Mplex
 
-[@libp2p/mplex](https://github.com/libp2p/js-libp2p-mplex) supports the following:
+[@libp2p/mplex](https://github.com/libp2p/js-libp2p-mplex) supports the following.
 
-```js
+All fields are optional. The default values are defined in [@libp2p/mplex/src/mplex.ts](https://github.com/libp2p/js-libp2p-mplex/blob/master/src/mplex.ts) - please see that file for the current values.
+
+```ts
 const node = await createLibp2pNode({
   muxers: [
-    new Mplex({
+    mplex({
       /**
        * The total number of inbound protocol streams that can be opened on a given connection
        */
-      maxInboundStreams: 1024,
+      maxInboundStreams: number
 
       /**
        * The total number of outbound protocol streams that can be opened on a given connection
        */
-      maxOutboundStreams: 1024,
+      maxOutboundStreams: number
+
+      /**
+       * How much incoming data in bytes to buffer while attempting to parse messages - peers sending many small messages in batches may cause this buffer to grow
+       */
+      maxUnprocessedMessageQueueSize: number
 
       /**
-       * How much incoming data to buffer before resetting the stream
+       * How much message data in bytes to buffer after parsing - slow stream consumers may cause this buffer to grow
        */
-      maxStreamBufferSize: 4 * 1024 * 1024,
+      maxStreamBufferSize: number
 
       /**
        * Mplex does not support backpressure so to protect ourselves, if `maxInboundStreams` is
        * hit and the remote opens more than this many streams per second, close the connection
        */
-      disconnectThreshold: 5
+      disconnectThreshold: number
     })
   ]
 })
 ```
 
 ### Yamux
 
-[@chainsafe/libp2p-yamux](https://github.com/Chainsafe/js-libp2p-yamux) supports the following:
+[@chainsafe/libp2p-yamux](https://github.com/Chainsafe/js-libp2p-yamux) supports the following.
 
-```js
+All fields are optional. The default values are defined in [@chainsafe/libp2p-yamux/src/config.ts](https://github.com/ChainSafe/js-libp2p-yamux/blob/master/src/config.ts) - please see that file for the current values.
+
+```ts
 const node = await createLibp2pNode({
   muxers: [
-    new Yamux({
+    yamux({
       /**
        * The total number of inbound protocol streams that can be opened on a given connection
+       *
+       * This field is optional, the default value is shown
        */
-      maxInboundStreams: 1024,
+      maxInboundStreams: number
 
       /**
        * The total number of outbound protocol streams that can be opened on a given connection
+       *
+       * This field is optional, the default value is shown
        */
-      maxOutboundStreams: 1024
+      maxOutboundStreams: number
     })
   ]
 })
 ```
 
 ### Protocol limits
 
-When registering listeners for custom protocols, the maximum number of simultaneously open inbound and outbound streams per-connection can be specified. If not specified these will default to 32 inbound streams and 64 outbound streams.
+When registering listeners for custom protocols, the maximum number of simultaneously open inbound and outbound streams per-connection can be specified. If not specified these will default to [32 inbound streams and 64 outbound streams](https://github.com/libp2p/js-libp2p/blob/master/src/registrar.ts#L14-L15).
 
 If more than this number of streams for the given protocol are opened on a single connection, subsequent new streams for that protocol will be immediately reset.
 
-Since incoming stream data is buffered until it is comsumed, you should attempt to specify the minimum amount of streams required to keep memory usage to a minimum.
+Since incoming stream data is buffered until it is consumed, you should attempt to specify the minimum amount of streams required to keep memory usage to a minimum.
 
-```js
+All fields are optional. The default values are defined in [src/registrar.ts](https://github.com/libp2p/js-libp2p/blob/master/src/registrar.ts) - please see that file for the current values.
+
+```ts
 libp2p.handle('/my-protocol/1.0.0', (streamData) => {
   // ..handle stream
 }, {
-  maxInboundStreams: 10, // defaults to 32
-  maxOutboundStreams: 10, // defaults to 64
-})
-```
-
-## Closing connections
-
-When choosing connections to close the connection manager sorts the list of connections by the value derived from the tags given to each peer. The values of all tags are summed and connections with lower valued peers are elibible for closing first.
-
-```js
-// tag a peer
-await libp2p.peerStore.tagPeer(peerId, 'my-tag', {
-  value: 50, // 0-100 is the typical value range
-  ttl: 1000 // optional field, this tag will be deleted after this many ms
+  maxInboundStreams: number
+  maxOutboundStreams: number
 })
 ```
 
@@ -198,28 +234,30 @@ A non-exhaustive list follows:
 
 ### TCP
 
-The [@libp2p/tcp](https://github.com/libp2p/js-libp2p-tcp) transport allows additional limits to be configured
+The [@libp2p/tcp](https://github.com/libp2p/js-libp2p-tcp) transport allows additional limits to be configured.
 
-```js
+All fields are optional. The full list of options is defined in [@libp2p/tcp/src/index.ts](https://github.com/libp2p/js-libp2p-tcp/blob/master/src/index.ts) - please see that file for more details.
+
+```ts
 const node = await createLibp2pNode({
   transports: [
-    new TCP({
+    tcp({
       /**
-       * Inbound connections with no activity in this timeframe (ms) will be closed
+       * Inbound connections with no activity in this time frame (ms) will be closed
        */
-      inboundSocketInactivityTimeout: 30000,
+      inboundSocketInactivityTimeout: number
 
       /**
-       * Outbound connections with no activity in this timeframe (ms) will be closed
+       * Outbound connections with no activity in this time frame (ms) will be closed
        */
-      outboundSocketInactivityTimeout: 60000,
+      outboundSocketInactivityTimeout: number
 
       /**
        * Once this many connections are open on this listener any further connections
        * will be rejected - this will have no effect if it is larger than the value
        * configured for the ConnectionManager maxConnections parameter
        */
-      maxConnections: 200
+      maxConnections: number
     })
   ]
 })
@@ -257,3 +295,24 @@ const node = await createLibp2pNode({
   }
 })
 ```
+
+## How much memory will be used for buffering?
+
+There is no a single config value to control the amount of memory js-libp2p uses.
+
+Important details for ascertaining this are:
+
+* Each connection has a multiplexer
+* Each multiplexer has a buffer for raw incoming data (`muxer.maxUnprocessedMessageQueueSize`)
+* The incoming data is parsed into messages for each stream and queued (`muxer.maxStreamBufferSize`)
+* Each multiplexer has a stream limit for number of streams (`muxer.maxInboundStreams`, `muxer.maxOutboundStreams`).
+
+As a result, the max amount of memory buffered by libp2p is approximately:
+
+```
+connectionManager.maxConnections *
+  (muxer.maxUnprocessedMessageQueueSize
+   + (muxer.maxInboundStreams * muxer.maxStreamBufferSize)
+   + (muxer.maxOutboundStreams * muxer.maxStreamBufferSize)
+  )
+```