Skip to content

Commit

Permalink
Merge pull request #5773 from hashicorp/docs/rate-limiting-from-ip-ad…
Browse files Browse the repository at this point in the history 
…dresses-1.16

updated docs for rate limiting for IP addresses - 1.16
  • Loading branch information
@trujillo-adam @im2nguyen
trujillo-adam authored and im2nguyen committed Jun 12, 2023
1 parent b1d3ec0 commit 755bd95
Show file tree
Hide file tree
Showing 7 changed files with 448 additions and 93 deletions.
48 changes: 34 additions & 14 deletions website/content/docs/agent/limits/index.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,27 +7,47 @@ description: Rate limiting is a set of Consul server agent configurations that y

# Traffic rate limiting overview

This topic provides an overview of the rates limits you can configure for Consul servers.

This topic provides overview information about the traffic rates limits you can configure for Consul datacenters.

## Introduction
You can configure global RPC rate limits to mitigate the risks to Consul servers when clients send excessive read or write requests to Consul resources. A _read request_ is defined as any request that does not modify Consul internal state. A _write request_ is defined as any request that modifies Consul internal state. Rate limits for read and write requests are configured separately.

## Rate limit modes
Configuring rate limits on RPC and gRPC traffic mitigates the risks to Consul servers when client agents or services send excessive read or write requests to Consul resources. A _read_ request is defined as any request that does not modify Consul internal state. A _write_ request is defined as any request that modifies Consul internal state. Configure read and write request limits independently.

## Workflow

You can set global limits on the rate of read and write requests that affect individual servers in the datacenter. You can set limits for all source IP addresses, which enables you to specify a budget for read and write requests to prevent any single source IP from overwhelming the Consul server and negatively affecting the network. The following steps describe the general process for setting global read and write rate limits:

1. Set arbitrary limits to begin understanding the upper boundary of RPC and gRPC loads in your network. Refer to [Initialize rate limit settings](/consul/docs/agent/limits/usage/init-rate-limits) for additional information.

You can set one of the following modes to determine how Consul servers react when exceeding request limits.

- **Enforcing mode**: The rate limiter denies requests to a server once they exceed the configured rate. In this mode, Consul generates metrics and logs to help you understand your network's load and configure limits accordingly.
- **Permissive mode**: The rate limiter allows requests to a server once they exceed the configured rate. In this mode, Consul generates metrics and logs to help you understand your Consul load and configure limits accordingly. Use this mode to help you debug specific issues as you configure limits.
- **Disabled mode**: Disables the rate limiter. This mode allows all requests Consul does not generate logs or metrics. This is the default mode.
1. Monitor the metrics and logs and readjust the initial configurations as necessary. Refer to [Monitor rate limit data](/consul/docs/agent/limits/usage/monitor-rate-limit-data)

Refer to [`rate_limits`](/consul/docs/agent/config/config-files#request_limits) for additional configuration information.
1. Define your final operational limits based on your observations. If you are defining global rate limits, refer to [Set global traffic rate limits](/consul/docs/agent/limits/usage/set-global-rate-limits) for additional information. For information about setting limits based on source IP, refer to [Limit traffic rates for a source IP](/consul/docs/agent/limits/usage/set-source-ip-rate-limits).

## Request denials
### Order of operations

When an HTTP request is denied for rate limiting reason, Consul returns one of the following errors:
You can define request rate limits in the agent configuration and in the control plane request limit configuration entry. The configuration entry also supports rate limit configurations for Consul resources. Consul perfroms the following order of operations when determing request rate limits:

- **429 Resource Exhausted**: Indicates that a server is not able to perform the request but that another server could potentially fulfill it. This error is most common on stale reads because any server may fulfill stale read requests. To resolve this type of error, we recommend immediately retrying the request to another server. If the request came from a Consul client agent, the agent automatically retries the request up to the limit set in the [`rpc_hold_timeout`](/consul/docs/agent/config/config-files#rpc_hold_timeout) configuration .
1. Parse request.
1. Does the request reach a global server limit?
- No: Proceed to the next stage.
- Yes: Return an error that the requested resource has been exhausted.
1. Does the request reach a limit associated with its source IP address?
- No: Proceed to the next stage.
- Yes: Return an error that the requested resource has been exhausted.
1. Resolve the Consul Enterprise metadata.
1. Does the request reach a limit associated with the source partition?
- No: Proceed to the next stage.
- Yes: Return an error that the requested resource has been exhausted.
1. Does the request reach a limit associated with the source namespace?
- No: Proceed to the next stage.
- Yes: Return an error that the requested resource has been exhausted.
1. Resolve the ACL identity associated with the request.
1. Does the request reach a limit associated with its identity?
- No: Proceed to the next stage.
- Yes: Return an error that the requested resource has been exhausted.
1. Handle the request.

- **503 Service Unavailable**: Indicates that server is unable to perform the request and that no other server can fulfill the request, either. This usually occurs on consistent reads or for writes. In this case we recommend retrying according to an exponential backoff schedule. If the request came from a Consul client agent, the agent automatically retries the request according to the [`rpc_hold_timeout`](/consul/docs/agent/config/config-files#rpc_hold_timeout) configuration.
## Kubernetes

Refer to [Rate limit reached on the server](/consul/docs/troubleshoot/common-errors#rate-limit-reached-on-the-server) for additional information.
To define global rate limits, configure the `request_limits` settings in the Consul Helm chart. Refer to the [Helm chart reference]() for additional information. Refer to the [control plane request limit configuration entry reference](/consul/docs/connect/config-entries/control-plane-request-limit) for information about applying a CRD for limiting traffic rates from source IPs.
7 changes: 3 additions & 4 deletions ...nt/docs/agent/limits/init-rate-limits.mdx → ...s/agent/limits/usage/init-rate-limits.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,16 @@ Because each network has different needs and application, you need to find out w
- Number of servers and the projected load
- Existing metrics expressing requests per second

1. Set the `mode` to `permissive`. In the following example, the configuration allows up to 1000 reads and 500 writes per second for each Consul agent:
1. Set the [`limits.request_limits.mode`](/consul/docs/agent/config/config-files#mode-1) parameter in the agent configuration to `permissive`. In the following example, the configuration allows up to 1000 reads and 500 writes per second for each Consul agent:

```hcl
request_limits {
mode = "permissive"
read_rate = 1000.0
write_rate = 500.0
}
```

1. Observe the logs and metrics for your application's typical cycle, such as a 24 hour period. Refer to [`log_file`](/consul/docs/agent/config/config-files#log_file) for information about where to retrieve logs. Call the [`/agent/metrics`](/consul/api-docs/agent#view-metrics) HTTP API endpoint and check the data for the following metrics:
```
1. Observe the logs and metrics for your application's typical cycle, such as a 24 hour period. Refer to [Monitor traffic rate limit data](/consul/docs/agent/limits/usage/monitor-rate-limit) for additional information. Call the [`/agent/metrics`](/consul/api-docs/agent#view-metrics) HTTP API endpoint and check the data for the following metrics:

- `rpc.rate_limit.exceeded` with value `global/read` for label `limit_type`
- `rpc.rate_limit.exceeded` with value `global/write` for label `limit_type`
Expand Down
66 changes: 66 additions & 0 deletions website/content/docs/agent/limits/usage/limit-request-rates-from-ips.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
layout: docs
page_title: Limit traffic rates for a source IP address
description: Learn how to set read and request rate limits on RPC and gRPC traffic from all source IP addresses to a Consul resource.
---

# Limit traffic rates from source IP addresses

This topic describes how to configure RPC and gRPC traffic rate limits for source IP addresses. This enables you to specify a budget for read and write requests to prevent any single source IP from overwhelming the Consul server and negatively affecting the network. For information about setting global traffic rate limits, refer to [Set a global limit on traffic rates](/consul/docs/agent/limits/usage/set-glogal-traffic-rate-limits). For an overview of Consul's server rate limiting capabilities, refer to [Limit traffic rates overview](/consul/docs/agent/limits/overview).

## Overview

You can set limits on the rate of read and write requests from source IP addresses to specific resources, which mitigates the risks to Consul servers when consul clients send excessive requests to a specific resource type. Before configuring traffic rate limits, you should complete the initialization process to understand normal traffic loads in your network. Refer to [Initialize rate limit settings](/consul/docs/agent/limits/init-rate-limits) for additional information.

Complete the following steps to configure traffic rate limits from a source IP address:

1. Define rate limits in a control plan request limit configuration entry. You can set limits for different types of resources calls.

1. Apply the configuration entry to enact the limits.

You should also monitor read and write rate activity and make any necessary adjustments. Refer to [Monitor rate limit data](/consul/docs/agent/limits/usage/monitor-rate-limits) for additional information.

## Define rate limits

Create a control plane request limit configuration entry in the `default` partition. The configuration entry applies to all client requests targeting any partition. Refer to the [control plane request limit configuration entry](/consul/docs/connect/config-entries/control-plan-request-limit) reference documentation for details about the available configuration parameters.

Specify the following parameters:

- `kind`: This must be set to `control-plane-request-limit`.
- `name`: Specify the name of the service that you want to limit read and write operations to.
- `read_rate`: Specify overall number of read operations per second allowed from the service.
- `write_rate`: Specify overall number of write operations per second allowed from the service.

You can also configure limits on calls to the key-value store, ACL system, and Consul catalog.

## Apply the configuration entry

If your network is deployed to virtual machines, use the `consul config write` command and specify the control plane request limit configuration entry to apply the configuration. For Kubernetes-orchestrated networks, use the `kubectl apply` command.

<Tabs>
<Tab heading="HCL" group="hcl">

```shell-session
$ consul config write control-plane-request-limit.hcl
```

</Tab>
<Tab heading="JSON" group="json">

```shell-session
$ consul config write control-plane-request-limit.json
```

</Tab>
<Tab heading="Kubernetes" group="kubernetes">

```shell-session
$ kubectl apply control-plane-request-limit.yaml
```

</Tab>
</Tabs>

## Disable request rate limits

Set the [limits.request_limits.mode](/consul/docs/agent/config/config-files#mode-1) in the agent configuration to `disabled` to allow services to exceed the specified read and write requests limits. The `disabled` mode applies to all request rate limits, even limits specifed in the [control plane request limits configuration entry](/consul/docs/connect/config-entries/control-plane-request-limits). Note that any other mode specified in the agent configuration only applies to global traffic rate limits.
77 changes: 77 additions & 0 deletions website/content/docs/agent/limits/usage/monitor-rate-limits.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
layout: docs
page_title: Monitor traffic rate limit data
description: Learn about the metrics and logs you can use to monitor server rate limiting activity, include rate limits for read operations and writer operations
---

# Monitor traffic rate limit data

This topic describes Consul functionality that enables you to monitor read and write request operations taking place in your network. Use the functionality to help you understand normal workloads and set safe limits on the number of requests Consul client agents and services can make to Consul servers.

## Access rate limit logs

Consul prints a log line for each rate limit request. The log provides the necessary information for identifying the source of the request and the configured limit. The log provides the information necessary for identifying the source of the request and the configured limit. Consul prints the log `DEBUG` log level and can drop the log to avoid affecting the server health. Dropping a log line increments the `rpc.rate_limit.log_dropped` metric.

The following example log shows that RPC request from `127.0.0.1:53562` to `KVS.Apply` exceeded the limit:

```text
2023-02-17T10:01:15.565-0500 [DEBUG] agent.server.rpc-rate-limit: RPC
exceeded allowed rate limit: rpc=KVS.Apply source_addr=127.0.0.1:53562
limit_type=global/write limit_enforced=false
```

Refer to [`log_file`](/consul/docs/agent/config/config-files#log_file) for information about where to retrieve log files.

## Review rate limit metrics

Consul captures the following metrics associated with rate limits:

- Type of limit
- Operation
- Rate limit mode

Call the `/agent/metrics` API endpoint to view the metrics associated with rate limits. Refer to [View Metrics](/consul/api-docs/agent#view-metrics) for API usage information. In the following example, Consul dropped a call to the consul service because it exceeded the limit by one call:

```shell-session
$ curl http://127.0.0.1:8500/v1/agent/metrics
{
. . .
"Counters": [
{
"Name": "consul.rpc.rate_limit.exceeded",
"Count": 1,
"Sum": 1,
"Min": 1,
"Max": 1,
"Mean": 1,
"Stddev": 0,
"Labels": {
"service": "consul"
}
},
{
"Name": "consul.rpc.rate_limit.log_dropped",
"Count": 1,
"Sum": 1,
"Min": 1,
"Max": 1,
"Mean": 1,
"Stddev": 0,
"Labels": {}
}
],
. . .
}
```

Refer to [Telemetry](/consul/docs/telemetry) for additional information.

## Request denials

When an HTTP request is denied for rate limiting reason, Consul returns one of the following errors:

- **429 Resource Exhausted**: Indicates that a server is not able to perform the request but that another server could potentially fulfill it. This error is most common on stale reads because any server may fulfill stale read requests. To resolve this type of error, we recommend immediately retrying the request to another server. If the request came from a Consul client agent, the agent automatically retries the request up to the limit set in the [`rpc_hold_timeout`](/consul/docs/agent/config/config-files#rpc_hold_timeout) configuration .

- **503 Service Unavailable**: Indicates that server is unable to perform the request and that no other server can fulfill the request, either. This usually occurs on consistent reads or for writes. In this case we recommend retrying according to an exponential backoff schedule. If the request came from a Consul client agent, the agent automatically retries the request according to the [`rpc_hold_timeout`](/consul/docs/agent/config/config-files#rpc_hold_timeout) configuration.

Refer to [Rate limit reached on the server](/consul/docs/troubleshoot/common-errors#rate-limit-reached-on-the-server) for additional information.
60 changes: 4 additions & 56 deletions ...limits/set-global-traffic-rate-limits.mdx → .../usage/set-global-traffic-rate-limits.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,62 +53,10 @@ limits = {

</CodeTabs>

## Access rate limit logs
## Monitor request rate traffic

Consul prints a log line for each rate limit request. The log includes information to identify the source of the request and the server's configured limit. Consul prints to `DEBUG` log level, and can be configured to drop log lines to avoid affecting the server health. Dropping a log line increments the `rpc.rate_limit.log_dropped` metric.
You should continue to mmonitor request traffic to ensure that request rates remain within the threshold you defined. Refer to [Monitor traffic rate limit data](/consul/docs/agent/limits/usage/monitor-rate-limits) for instructions about checking metrics and log entries, as well as troubleshooting informaiton.

The following example log shows that RPC request from `127.0.0.1:53562` to `KVS.Apply` exceeded the rate limit:
## Disable request rate limits

<CodeBlockConfig hideClipboard>

```shell-session
2023-02-17T10:01:15.565-0500 [DEBUG] agent.server.rpc-rate-limit: RPC
exceeded allowed rate limit: rpc=KVS.Apply source_addr=127.0.0.1:53562
limit_type=global/write limit_enforced=false
```
</CodeBlockConfig>

## Review rate limit metrics

Consul captures the following metrics associated with rate limits:

- Type of limit
- Operation
- Rate limit mode

Call the `agent/metrics` API endpoint to view the metrics associated with rate limits. Refer to [View Metrics](/consul/api-docs/agent#view-metrics) for API usage information.

In the following example, Consul dropped a call to the `consul` service because it exceeded the limit by one call:

```shell-session
$ curl http://127.0.0.1:8500/v1/agent/metrics
{
. . .
"Counters": [
{
"Name": "consul.rpc.rate_limit.exceeded",
"Count": 1,
"Sum": 1,
"Min": 1,
"Max": 1,
"Mean": 1,
"Stddev": 0,
"Labels": {
"service": "consul"
}
},
{
"Name": "consul.rpc.rate_limit.log_dropped",
"Count": 1,
"Sum": 1,
"Min": 1,
"Max": 1,
"Mean": 1,
"Stddev": 0,
"Labels": {}
}
],
. . .
```

Refer to [Telemetry](/consul/docs/agent/telemetry) for additional information.
Set the [limits.request_limits.mode](/consul/docs/agent/config/config-files#mode-1) to `disabled` to allow services to exceed the specified read and write requests limits, even limits specifed in the [control plane request limits configuration entry](/consul/docs/connect/config-entries/control-plane-request-limits). Note that any other mode specified in the agent configuration only applies to global traffic rate limits.
Loading

0 comments on commit 755bd95

Please sign in to comment.