Skip to content

Commit 153567e

Browse files
authored
docs(batch): add example on how to integrate with sentry.io (#308)
1 parent 271f560 commit 153567e

File tree

1 file changed

+104
-52
lines changed

1 file changed

+104
-52
lines changed

Diff for: docs/utilities/batch.md

+104-52
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,13 @@ description: Utility
55

66
The SQS batch processing utility provides a way to handle partial failures when processing batches of messages from SQS.
77

8-
**Key Features**
8+
## Key Features
99

1010
* Prevent successfully processed messages being returned to SQS
1111
* Simple interface for individually processing messages from a batch
1212
* Build your own batch processor using the base classes
1313

14-
**Background**
14+
## Background
1515

1616
When using SQS as a Lambda event source mapping, Lambda functions are triggered with a batch of messages from SQS.
1717

@@ -25,77 +25,111 @@ are returned to the queue.
2525

2626
More details on how Lambda works with SQS can be found in the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html)
2727

28+
## Getting started
2829

29-
**IAM Permissions**
30+
### IAM Permissions
3031

31-
This utility requires additional permissions to work as expected. Lambda functions using this utility require the `sqs:DeleteMessageBatch` permission.
32+
Before your use this utility, your AWS Lambda function must have `sqs:DeleteMessageBatch` permission to delete successful messages directly from the queue.
3233

33-
## Processing messages from SQS
34+
> Example using AWS Serverless Application Model (SAM)
3435
35-
You can use either **[sqs_batch_processor](#sqs_batch_processor-decorator)** decorator, or **[PartialSQSProcessor](#partialsqsprocessor-context-manager)** as a context manager.
36+
=== "template.yml"
37+
```yaml hl_lines="2-3 12-15"
38+
Resources:
39+
MyQueue:
40+
Type: AWS::SQS::Queue
3641

37-
They have nearly the same behaviour when it comes to processing messages from the batch:
42+
HelloWorldFunction:
43+
Type: AWS::Serverless::Function
44+
Properties:
45+
Runtime: python3.8
46+
Environment:
47+
Variables:
48+
POWERTOOLS_SERVICE_NAME: example
49+
Policies:
50+
- SQSPollerPolicy:
51+
QueueName:
52+
!GetAtt MyQueue.QueueName
53+
```
3854

39-
* **Entire batch has been successfully processed**, where your Lambda handler returned successfully, we will let SQS delete the batch to optimize your cost
40-
* **Entire Batch has been partially processed successfully**, where exceptions were raised within your `record handler`, we will:
41-
- **1)** Delete successfully processed messages from the queue by directly calling `sqs:DeleteMessageBatch`
42-
- **2)** Raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
55+
### Processing messages from SQS
4356

44-
The only difference is that **PartialSQSProcessor** will give you access to processed messages if you need.
57+
You can use either **[sqs_batch_processor](#sqs_batch_processor-decorator)** decorator, or **[PartialSQSProcessor](#partialsqsprocessor-context-manager)** as a context manager if you'd like access to the processed results.
4558

46-
## Record Handler
59+
You need to create a function to handle each record from the batch - We call it `record_handler` from here on.
4760

48-
Both decorator and context managers require an explicit function to process the batch of messages - namely `record_handler` parameter.
61+
=== "Decorator"
4962

50-
This function is responsible for processing each individual message from the batch, and to raise an exception if unable to process any of the messages sent.
63+
```python hl_lines="3 6"
64+
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
5165

52-
**Any non-exception/successful return from your record handler function** will instruct both decorator and context manager to queue up each individual message for deletion.
66+
def record_handler(record):
67+
return do_something_with(record["body"])
5368

54-
### sqs_batch_processor decorator
69+
@sqs_batch_processor(record_handler=record_handler)
70+
def lambda_handler(event, context):
71+
return {"statusCode": 200}
72+
```
73+
=== "Context manager"
5574

56-
When using this decorator, you need provide a function via `record_handler` param that will process individual messages from the batch - It should raise an exception if it is unable to process the record.
75+
```python hl_lines="3 9 11-12"
76+
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
77+
78+
def record_handler(record):
79+
return_value = do_something_with(record["body"])
80+
return return_value
81+
82+
def lambda_handler(event, context):
83+
records = event["Records"]
84+
processor = PartialSQSProcessor()
85+
86+
with processor(records, record_handler) as proc:
87+
result = proc.process() # Returns a list of all results from record_handler
88+
89+
return result
90+
```
91+
92+
!!! tip
93+
**Any non-exception/successful return from your record handler function** will instruct both decorator and context manager to queue up each individual message for deletion.
94+
95+
If the entire batch succeeds, we let Lambda to proceed in deleting the records from the queue for cost reasons.
96+
97+
### Partial failure mechanics
5798

5899
All records in the batch will be passed to this handler for processing, even if exceptions are thrown - Here's the behaviour after completing the batch:
59100

60101
* **Any successfully processed messages**, we will delete them from the queue via `sqs:DeleteMessageBatch`
61102
* **Any unprocessed messages detected**, we will raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
62103

63104
!!! warning
64-
You will not have accessed to the <strong>processed messages</strong> within the Lambda Handler - all processing logic will and should be performed by the <code>record_handler</code> function.
105+
You will not have accessed to the **processed messages** within the Lambda Handler.
65106

66-
=== "app.py"
107+
All processing logic will and should be performed by the `record_handler` function.
67108

68-
```python
69-
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
109+
## Advanced
70110

71-
def record_handler(record):
72-
# This will be called for each individual message from a batch
73-
# It should raise an exception if the message was not processed successfully
74-
return_value = do_something_with(record["body"])
75-
return return_value
111+
### Choosing between decorator and context manager
76112

77-
@sqs_batch_processor(record_handler=record_handler)
78-
def lambda_handler(event, context):
79-
return {"statusCode": 200}
80-
```
113+
They have nearly the same behaviour when it comes to processing messages from the batch:
81114

82-
### PartialSQSProcessor context manager
115+
* **Entire batch has been successfully processed**, where your Lambda handler returned successfully, we will let SQS delete the batch to optimize your cost
116+
* **Entire Batch has been partially processed successfully**, where exceptions were raised within your `record handler`, we will:
117+
- **1)** Delete successfully processed messages from the queue by directly calling `sqs:DeleteMessageBatch`
118+
- **2)** Raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
119+
120+
The only difference is that **PartialSQSProcessor** will give you access to processed messages if you need.
83121

84-
If you require access to the result of processed messages, you can use this context manager.
122+
### Accessing processed messages
85123

86-
The result from calling `process()` on the context manager will be a list of all the return values from your `record_handler` function.
124+
Use `PartialSQSProcessor` context manager to access a list of all return values from your `record_handler` function.
87125

88126
=== "app.py"
89127

90128
```python
91129
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
92130

93131
def record_handler(record):
94-
# This will be called for each individual message from a batch
95-
# It should raise an exception if the message was not processed successfully
96-
return_value = do_something_with(record["body"])
97-
return return_value
98-
132+
return do_something_with(record["body"])
99133

100134
def lambda_handler(event, context):
101135
records = event["Records"]
@@ -108,7 +142,7 @@ The result from calling `process()` on the context manager will be a list of all
108142
return result
109143
```
110144

111-
## Passing custom boto3 config
145+
### Passing custom boto3 config
112146

113147
If you need to pass custom configuration such as region to the SDK, you can pass your own [botocore config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) to
114148
the `sqs_batch_processor` decorator:
@@ -159,30 +193,32 @@ the `sqs_batch_processor` decorator:
159193
```
160194

161195

162-
## Suppressing exceptions
196+
### Suppressing exceptions
163197

164198
If you want to disable the default behavior where `SQSBatchProcessingError` is raised if there are any errors, you can pass the `suppress_exception` boolean argument.
165199

166200
=== "Decorator"
167201

168-
```python hl_lines="2"
169-
...
202+
```python hl_lines="3"
203+
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
204+
170205
@sqs_batch_processor(record_handler=record_handler, config=config, suppress_exception=True)
171206
def lambda_handler(event, context):
172207
return {"statusCode": 200}
173208
```
174209

175210
=== "Context manager"
176211

177-
```python hl_lines="2"
178-
...
212+
```python hl_lines="3"
213+
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
214+
179215
processor = PartialSQSProcessor(config=config, suppress_exception=True)
180216

181217
with processor(records, record_handler):
182218
result = processor.process()
183219
```
184220

185-
## Create your own partial processor
221+
### Create your own partial processor
186222

187223
You can create your own partial batch processor by inheriting the `BasePartialProcessor` class, and implementing `_prepare()`, `_clean()` and `_process_record()`.
188224

@@ -192,11 +228,9 @@ You can create your own partial batch processor by inheriting the `BasePartialPr
192228

193229
You can then use this class as a context manager, or pass it to `batch_processor` to use as a decorator on your Lambda handler function.
194230

195-
**Example:**
196-
197231
=== "custom_processor.py"
198232

199-
```python
233+
```python hl_lines="3 9 24 30 37 57"
200234
from random import randint
201235

202236
from aws_lambda_powertools.utilities.batch import BasePartialProcessor, batch_processor
@@ -223,14 +257,12 @@ You can then use this class as a context manager, or pass it to `batch_processor
223257
def _prepare(self):
224258
# It's called once, *before* processing
225259
# Creates table resource and clean previous results
226-
# E.g.:
227260
self.ddb_table = boto3.resource("dynamodb").Table(self.table_name)
228261
self.success_messages.clear()
229262

230263
def _clean(self):
231264
# It's called once, *after* closing processing all records (closing the context manager)
232265
# Here we're sending, at once, all successful messages to a ddb table
233-
# E.g.:
234266
with ddb_table.batch_writer() as batch:
235267
for result in self.success_messages:
236268
batch.put_item(Item=result)
@@ -239,7 +271,6 @@ You can then use this class as a context manager, or pass it to `batch_processor
239271
# It handles how your record is processed
240272
# Here we're keeping the status of each run
241273
# where self.handler is the record_handler function passed as an argument
242-
# E.g.:
243274
try:
244275
result = self.handler(record) # record_handler passed to decorator/context manager
245276
return self.success_handler(record, result)
@@ -260,3 +291,24 @@ You can then use this class as a context manager, or pass it to `batch_processor
260291
def lambda_handler(event, context):
261292
return {"statusCode": 200}
262293
```
294+
295+
### Integrating exception handling with Sentry.io
296+
297+
When using Sentry.io for error monitoring, you can override `failure_handler` to include to capture each processing exception:
298+
299+
> Credits to [Charles-Axel Dein](https://github.com/awslabs/aws-lambda-powertools-python/issues/293#issuecomment-781961732)
300+
301+
=== "sentry_integration.py"
302+
303+
```python hl_lines="4 7-8"
304+
from typing import Tuple
305+
306+
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
307+
from sentry_sdk import capture_exception
308+
309+
class SQSProcessor(PartialSQSProcessor):
310+
def failure_handler(self, record: Event, exception: Tuple) -> Tuple: # type: ignore
311+
capture_exception() # send exception to Sentry
312+
logger.exception("got exception while processing SQS message")
313+
return super().failure_handler(record, exception) # type: ignore
314+
```

0 commit comments

Comments
 (0)