Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(idempotency): add cdk table example #2434

Merged
merged 8 commits into from
Jun 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 27 additions & 38 deletions docs/utilities/idempotency.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,31 +77,16 @@ If you're not [changing the default configuration for the DynamoDB persistence l
???+ tip "Tip: You can share a single state table for all functions"
You can reuse the same DynamoDB table to store idempotency state. We add `module_name` and [qualified name for classes and functions](https://peps.python.org/pep-3155/){target="_blank"} in addition to the idempotency key as a hash key.

```yaml hl_lines="5-13 21-23" title="AWS Serverless Application Model (SAM) example"
Resources:
IdempotencyTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
TimeToLiveSpecification:
AttributeName: expiration
Enabled: true
BillingMode: PAY_PER_REQUEST

HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
Runtime: python3.9
...
Policies:
- DynamoDBCrudPolicy:
TableName: !Ref IdempotencyTable
```
=== "sam.yaml"

```yaml hl_lines="6-14 24-31" title="AWS Serverless Application Model (SAM) example"
--8<-- "examples/idempotency/sam.yaml"
```
=== "cdk.py"

```python hl_lines="10 13 16 19-21" title="AWS Cloud Development Kit (CDK) Construct example"
--8<-- "examples/idempotency/cdk.py"
```
rubenfonseca marked this conversation as resolved.
Show resolved Hide resolved

???+ warning "Warning: Large responses with DynamoDB persistence layer"
When using this utility with DynamoDB, your function's responses must be [smaller than 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items){target="_blank"}.
Expand Down Expand Up @@ -148,7 +133,7 @@ You can quickly start by initializing the `DynamoDBPersistenceLayer` class and u

=== "Example event"

```json
```json
rubenfonseca marked this conversation as resolved.
Show resolved Hide resolved
{
"username": "xyz",
"product_id": "123456789"
Expand Down Expand Up @@ -334,10 +319,12 @@ In this example, we have a Lambda handler that creates a payment for a user subs

Imagine the function executes successfully, but the client never receives the response due to a connection issue. It is safe to retry in this instance, as the idempotent decorator will return a previously saved response.

**What we want here** is to instruct Idempotency to use `user` and `product_id` fields from our incoming payload as our idempotency key. If we were to treat the entire request as our idempotency key, a simple HTTP header change would cause our customer to be charged twice.
**What we want here** is to instruct Idempotency to use `user` and `product_id` fields from our incoming payload as our idempotency key.
If we were to treat the entire request as our idempotency key, a simple HTTP header change would cause our customer to be charged twice.

???+ tip "Deserializing JSON strings in payloads for increased accuracy."
The payload extracted by the `event_key_jmespath` is treated as a string by default. This means there could be differences in whitespace even when the JSON payload itself is identical.
The payload extracted by the `event_key_jmespath` is treated as a string by default.
This means there could be differences in whitespace even when the JSON payload itself is identical.

To alter this behaviour, we can use the [JMESPath built-in function](jmespath_functions.md#powertools_json-function){target="_blank"} `powertools_json()` to treat the payload as a JSON object (dict) rather than a string.

Expand Down Expand Up @@ -410,15 +397,17 @@ Imagine the function executes successfully, but the client never receives the re
???+ note
This is automatically done when you decorate your Lambda handler with [@idempotent decorator](#idempotent-decorator).

To prevent against extended failed retries when a [Lambda function times out](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-verify-invocation-timeouts/){target="_blank"}, Powertools for AWS Lambda (Python) calculates and includes the remaining invocation available time as part of the idempotency record.
To prevent against extended failed retries when a [Lambda function times out](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-verify-invocation-timeouts/){target="_blank"},
Powertools for AWS Lambda (Python) calculates and includes the remaining invocation available time as part of the idempotency record.

???+ example
If a second invocation happens **after** this timestamp, and the record is marked as `INPROGRESS`, we will execute the invocation again as if it was in the `EXPIRED` state (e.g, `expire_seconds` field elapsed).

This means that if an invocation expired during execution, it will be quickly executed again on the next retry.

???+ important
If you are only using the [@idempotent_function decorator](#idempotent_function-decorator) to guard isolated parts of your code, you must use `register_lambda_context` available in the [idempotency config object](#customizing-the-default-behavior) to benefit from this protection.
If you are only using the [@idempotent_function decorator](#idempotent_function-decorator) to guard isolated parts of your code,
you must use `register_lambda_context` available in the [idempotency config object](#customizing-the-default-behavior) to benefit from this protection.

Here is an example on how you register the Lambda context in your handler:

Expand Down Expand Up @@ -698,14 +687,14 @@ When using DynamoDB as a persistence layer, you can alter the attribute names by

Idempotent decorator can be further configured with **`IdempotencyConfig`** as seen in the previous example. These are the available options for further configuration

| Parameter | Default | Description |
| ------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](/utilities/jmespath_functions){target="_blank"} |
| **payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload |
| **raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request |
| **expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired |
| **use_local_cache** | `False` | Whether to locally cache idempotency results |
| **local_cache_max_items** | 256 | Max number of items to store in local cache |
| Parameter | Default | Description |
| ------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](/utilities/jmespath_functions){target="_blank"} |
| **payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload |
| **raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request |
| **expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired |
| **use_local_cache** | `False` | Whether to locally cache idempotency results |
| **local_cache_max_items** | 256 | Max number of items to store in local cache |
| **hash_function** | `md5` | Function to use for calculating hashes, as provided by [hashlib](https://docs.python.org/3/library/hashlib.html){target="_blank"} in the standard library. |

### Handling concurrent executions with the same payload
Expand Down
21 changes: 21 additions & 0 deletions examples/idempotency/cdk.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
from aws_cdk import RemovalPolicy
from aws_cdk import aws_dynamodb as dynamodb
from aws_cdk import aws_iam as iam
from constructs import Construct


class IdempotencyConstruct(Construct):
def __init__(self, scope: Construct, name: str, lambda_role: iam.Role) -> None:
super().__init__(scope, name)
self.idempotency_table = dynamodb.Table(
self,
"IdempotencyTable",
partition_key=dynamodb.Attribute(name="id", type=dynamodb.AttributeType.STRING),
billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
removal_policy=RemovalPolicy.DESTROY,
time_to_live_attribute="expiration",
point_in_time_recovery=True,
)
self.idempotency_table.grant(
lambda_role, "dynamodb:PutItem", "dynamodb:GetItem", "dynamodb:UpdateItem", "dynamodb:DeleteItem"
)
31 changes: 31 additions & 0 deletions examples/idempotency/sam.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
Transform: AWS::Serverless-2016-10-31
Resources:
IdempotencyTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
TimeToLiveSpecification:
AttributeName: expiration
Enabled: true
BillingMode: PAY_PER_REQUEST

HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
Runtime: python3.9
Handler: app.py
Policies:
- Statement:
- Sid: AllowDynamodbReadWrite
Effect: Allow
Action:
- dynamodb:PutItem
- dynamodb:GetItem
- dynamodb:UpdateItem
- dynamodb:DeleteItem
Resource: !GetAtt IdempotencyTable.Arn