Integrate redis rate limiter with saas connectors

CHANGELOG.md

@@ -27,6 +27,7 @@ The types of changes are:
 * Individually select and reprocess DSRs that have errored [#1203](https://github.com/ethyca/fides/pull/1489)
 * Bulk select and reprocess DSRs that have errored [#1205](https://github.com/ethyca/fides/pull/1489)
 * Config Wizard: AWS scan results populate in system review forms. [#1454](https://github.com/ethyca/fides/pull/1454)
+* Integrate rate limiter with Saas Connectors. [#1433](https://github.com/ethyca/fides/pull/1433)
 
 ### Changed
 * Updated mypy to version 0.981 and Python to version 3.10.7 [#1448](https://github.com/ethyca/fides/pull/1448)

docs/fides/docs/saas_connectors/saas_config.md

@@ -39,10 +39,15 @@ saas_config:
         username: <username>
         password: <api_key>
 
+  rate_limit_config:
+    limits:
+    - rate: 10
+      period: second
+
   test_request:
     method: GET
     path: /3.0/lists
-    
+
   endpoints:
   - name: messages
     requests:
@@ -62,6 +67,8 @@ saas_config:
               field: from_email
               value:
                 identity: email
+        rate_limit_config:
+          enabled: false
   - name: conversations
     requests:
       read:
@@ -125,6 +132,7 @@ The above configuration also contains the following complex fields
 
 - `connector_params`
 - `client_config`
+- `rate_limit_config`
 - `test_request`
 - `endpoints`
 - `data_protection_request`
@@ -154,7 +162,7 @@ external_references:
 
 #### Client config
 
-The `client_config` describes the necessary information to be able to create a base HTTP client. The values for host, username, and password are not defined here, only referenced in the form of a `connector_param` which Fides uses to insert the actual value from the stored secrets.
+The `client_config` field describes the necessary information to be able to create a base HTTP client. The values for host, username, and password are not defined here, only referenced in the form of a `connector_param` which Fides uses to insert the actual value from the stored secrets.
 
 ```yaml
 client_config:
@@ -180,6 +188,20 @@ authentication:
 
 Fides also supports [OAuth2 authentication](saas_oauth2.md).
 
+#### Rate limits
+
+The `rate_limit_config` field describes rate limits which can be used to limit how fast requests can be made to an endpoint. They can also be configured at the [endpoint request level](#endpoints). 
+
+```yaml
+rate_limit_config:
+  enabled: true
+  limits:
+  - rate: <rate>
+    period: <period>
+```
+
+Rate limiter supports `second`, `minute`, `hour` and `day`periods. See [Saas Rate Limiting](./saas_rate_limiting.md) for additional details.
+
 #### Test request
 
 Once the base client is defined, use a `test_request` to verify the hostname and credentials. This is in the form of an idempotent request (usually a `read` request). The testing approach is the same for any [Connection](../getting-started/database_connectors.md#test-your-connection)
@@ -245,6 +267,7 @@ The `requests` configuration further contains the following fields:
 | `pagination` | An optional strategy used to get the next set of results from APIs with resources spanning multiple pages. Details can be found under [SaaS Pagination](saas_pagination.md).
 |`grouped_inputs` | An optional list of reference fields whose inputs are dependent upon one another.  For example, an endpoint may need both an `organization_id` and a `project_id` from another endpoint.  These aren't independent values, as a `project_id` belongs to an `organization_id`.  You would specify this as `["organization_id", "project_id"]`.
 | `client_config` | Specify optional embedded Client Configs if an individual request needs a different protocol, host, or authentication strategy from the base Client Config.
+| `rate_limit_config` | An optional rate limit configuration. Can be used to set rate limits or disable limiting for a specific endpoint. See [Saas Rate Limiting](./saas_rate_limiting.md) for additional details.
 
 ## Param_values in more detail
 

docs/fides/docs/saas_connectors/saas_rate_limiting.md

@@ -0,0 +1,79 @@
+# SaaS Rate limiting
+
+Rate limits can be used to limit how fast requests can be made to saas endpoints. Rate limiting works accross Fides instances that share a Redis cluster. 
+
+A rate limit configuration can be set the within the client config for saas connectors. It can also be set within endpoint request config to override the config for specific endpoints. 
+
+## Configuration details
+- `enabled` (_bool_): *Optional.* Determines if the rate limiter is enabled. Default is true.
+- `limits` (_list([RateLimit](#rate-limit-configuration))_): *Optional.* A list of RateLimit objects which can define multiple rate limits for endpoint requests.
+
+### Rate Limit 
+- `rate` (_int_): Number of calls which are allow for the specified period
+- `period` (_str_): The time period for which to limit calls. Allows values are `second`, `minute`, `hour`, `day`.
+- `custom_key` (_str_): *Optional.* An optional key which can be used to group rate usage accross endpoints or connectors. 
+
+## Implementation details
+
+Fides implements rate limiting as a fixed window algorithm. Epoch seconds are divided into discrete buckets based on the configured period. 
+
+## Examples
+
+### Limiting Connector
+
+In this example we limit our connector to 10 TPS for all endpoints
+```yaml
+saas_config:
+  fides_key: my_connector
+
+rate_limit_config:
+    limits:
+    - rate: 10
+      period: second
+```
+
+### Disable Limit For Specific Endpoint
+
+In this example we limit our connector request rate but disable the limiter for `my_non_limited_entity` read requests.
+
+```yaml
+saas_config:
+  fides_key: my_connector
+
+  rate_limit_config:
+    limits:
+    - rate: 10
+      period: second
+
+  endpoints:
+  - name: my_non_limited_entity
+    requests:
+      read:
+        rate_limit_config:
+          enabled: false
+
+```
+
+### Shared Rate Usage Accross Connectors
+
+In this example `my_connector_1` and `my_connector_2` both contribute requests to the same time window as they share the custom key `shared_custom_key`.
+
+
+```yaml
+saas_config:
+  fides_key: my_connector_1
+
+rate_limit_config:
+    limits:
+    - rate: 10
+      period: second
+      custom_key: shared_custom_key
+
+  fides_key: my_connector_2
+
+rate_limit_config:
+    limits:
+    - rate: 10
+      period: second
+      custom_key: shared_custom_key
+```

docs/fides/mkdocs.yml

@@ -53,6 +53,7 @@ nav:
           - SaaS OAuth2 Configuration: saas_connectors/saas_oauth2.md
           - SaaS Post-Processors: saas_connectors/saas_postprocessors.md
           - SaaS Pagination: saas_connectors/saas_pagination.md
+          - SaaS Rate Limiting: saas_connectors/rate_limiting.md
           - Connection Examples:
               - Adobe Campaign: saas_connectors/example_configs/adobe.md
               - Hubspot: saas_connectors/example_configs/hubspot.md

src/fides/api/ops/schemas/limiter/rate_limit_config.py

@@ -0,0 +1,52 @@
+from enum import Enum
+from typing import Dict, List, Optional
+
+from pydantic import BaseModel, root_validator, validator
+
+
+class RateLimitPeriod(str, Enum):
+    """
+    Defines the periods supported by rate limit config
+    """
+
+    second = "second"
+    minute = "minute"
+    hour = "hour"
+    day = "day"
+
+
+class RateLimit(BaseModel):
+    """
+    A config object which allows configuring rate limits for connectors
+    """
+
+    rate: int
+    period: RateLimitPeriod
+    custom_key: Optional[str]
+
+    @validator("rate")
+    def rate_more_than_zero(cls, v: int) -> int:
+        assert v > 0, "rate must be more than zero"
+        return v
+
+
+class RateLimitConfig(BaseModel):
+    """
+    A config object which allows configuring rate limits for connectors
+    """
+
+    limits: Optional[List[RateLimit]]
+    enabled: Optional[bool] = True
+
+    @root_validator
+    def validate_all(cls, values: Dict) -> Dict:
+        limits: Optional[List[RateLimit]] = values["limits"]
+        enabled: Optional[bool] = values["enabled"]
+
+        if enabled:
+            assert (
+                limits and len(limits) > 0
+            ), "limits must be set if rate limiter is enabled"
+        if not enabled:
+            assert not limits, "limits cannot be set if enabled is false"
+        return values

src/fides/api/ops/schemas/saas/saas_config.py

@@ -13,6 +13,7 @@
 )
 from fides.api.ops.schemas.base_class import BaseSchema
 from fides.api.ops.schemas.dataset import FidesCollectionKey, FidesopsDatasetReference
+from fides.api.ops.schemas.limiter.rate_limit_config import RateLimitConfig
 from fides.api.ops.schemas.saas.shared_schemas import HTTPMethod
 from fides.api.ops.schemas.shared_schemas import FidesOpsKey
 
@@ -102,6 +103,7 @@ class SaaSRequest(BaseModel):
     pagination: Optional[Strategy]
     grouped_inputs: Optional[List[str]] = []
     ignore_errors: Optional[bool] = False
+    rate_limit_config: Optional[RateLimitConfig]
 
     class Config:
         """Populate models with the raw value of enum fields, rather than the enum itself"""
@@ -308,6 +310,7 @@ class SaaSConfig(SaaSConfigBase):
     endpoints: List[Endpoint]
     test_request: SaaSRequest
     data_protection_request: Optional[SaaSRequest] = None  # GDPR Delete
+    rate_limit_config: Optional[RateLimitConfig]
 
     @property
     def top_level_endpoint_dict(self) -> Dict[str, Endpoint]: