You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/utilities/batch.md
+104-52
Original file line number
Diff line number
Diff line change
@@ -5,13 +5,13 @@ description: Utility
5
5
6
6
The SQS batch processing utility provides a way to handle partial failures when processing batches of messages from SQS.
7
7
8
-
**Key Features**
8
+
## Key Features
9
9
10
10
* Prevent successfully processed messages being returned to SQS
11
11
* Simple interface for individually processing messages from a batch
12
12
* Build your own batch processor using the base classes
13
13
14
-
**Background**
14
+
## Background
15
15
16
16
When using SQS as a Lambda event source mapping, Lambda functions are triggered with a batch of messages from SQS.
17
17
@@ -25,77 +25,111 @@ are returned to the queue.
25
25
26
26
More details on how Lambda works with SQS can be found in the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html)
27
27
28
+
## Getting started
28
29
29
-
**IAM Permissions**
30
+
### IAM Permissions
30
31
31
-
This utility requires additional permissions to work as expected. Lambda functions using this utility require the `sqs:DeleteMessageBatch` permission.
32
+
Before your use this utility, your AWS Lambda function must have `sqs:DeleteMessageBatch` permission to delete successful messages directly from the queue.
32
33
33
-
## Processing messages from SQS
34
+
> Example using AWS Serverless Application Model (SAM)
34
35
35
-
You can use either **[sqs_batch_processor](#sqs_batch_processor-decorator)** decorator, or **[PartialSQSProcessor](#partialsqsprocessor-context-manager)** as a context manager.
36
+
=== "template.yml"
37
+
```yaml hl_lines="2-3 12-15"
38
+
Resources:
39
+
MyQueue:
40
+
Type: AWS::SQS::Queue
36
41
37
-
They have nearly the same behaviour when it comes to processing messages from the batch:
42
+
HelloWorldFunction:
43
+
Type: AWS::Serverless::Function
44
+
Properties:
45
+
Runtime: python3.8
46
+
Environment:
47
+
Variables:
48
+
POWERTOOLS_SERVICE_NAME: example
49
+
Policies:
50
+
- SQSPollerPolicy:
51
+
QueueName:
52
+
!GetAtt MyQueue.QueueName
53
+
```
38
54
39
-
***Entire batch has been successfully processed**, where your Lambda handler returned successfully, we will let SQS delete the batch to optimize your cost
40
-
***Entire Batch has been partially processed successfully**, where exceptions were raised within your `record handler`, we will:
41
-
-**1)** Delete successfully processed messages from the queue by directly calling `sqs:DeleteMessageBatch`
42
-
-**2)** Raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
55
+
### Processing messages from SQS
43
56
44
-
The only difference is that **PartialSQSProcessor**will give youaccess to processed messages if you need.
57
+
You can use either **[sqs_batch_processor](#sqs_batch_processor-decorator)** decorator, or **[PartialSQSProcessor](#partialsqsprocessor-context-manager)**as a context manager if you'd like access to the processed results.
45
58
46
-
## Record Handler
59
+
You need to create a function to handle each record from the batch - We call it `record_handler` from here on.
47
60
48
-
Both decorator and context managers require an explicit function to process the batch of messages - namely `record_handler` parameter.
61
+
=== "Decorator"
49
62
50
-
This function is responsible for processing each individual message from the batch, and to raise an exception if unable to process any of the messages sent.
63
+
```python hl_lines="3 6"
64
+
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
51
65
52
-
**Any non-exception/successful return from your record handler function** will instruct both decorator and context manager to queue up each individual message for deletion.
When using this decorator, you need provide a function via `record_handler` param that will process individual messages from the batch - It should raise an exception if it is unable to process the record.
75
+
```python hl_lines="3 9 11-12"
76
+
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
77
+
78
+
def record_handler(record):
79
+
return_value = do_something_with(record["body"])
80
+
return return_value
81
+
82
+
def lambda_handler(event, context):
83
+
records = event["Records"]
84
+
processor = PartialSQSProcessor()
85
+
86
+
with processor(records, record_handler) as proc:
87
+
result = proc.process() # Returns a list of all results from record_handler
88
+
89
+
return result
90
+
```
91
+
92
+
!!! tip
93
+
**Any non-exception/successful return from your record handler function** will instruct both decorator and context manager to queue up each individual message for deletion.
94
+
95
+
If the entire batch succeeds, we let Lambda to proceed in deleting the records from the queue for cost reasons.
96
+
97
+
### Partial failure mechanics
57
98
58
99
All records in the batch will be passed to this handler for processing, even if exceptions are thrown - Here's the behaviour after completing the batch:
59
100
60
101
***Any successfully processed messages**, we will delete them from the queue via `sqs:DeleteMessageBatch`
61
102
***Any unprocessed messages detected**, we will raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
62
103
63
104
!!! warning
64
-
You will not have accessed to the <strong>processed messages</strong> within the Lambda Handler - all processing logic will and should be performed by the <code>record_handler</code> function.
105
+
You will not have accessed to the **processed messages** within the Lambda Handler.
65
106
66
-
=== "app.py"
107
+
All processing logic will and should be performed by the `record_handler` function.
67
108
68
-
```python
69
-
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
109
+
## Advanced
70
110
71
-
def record_handler(record):
72
-
# This will be called for each individual message from a batch
73
-
# It should raise an exception if the message was not processed successfully
74
-
return_value = do_something_with(record["body"])
75
-
return return_value
111
+
### Choosing between decorator and context manager
They have nearly the same behaviour when it comes to processing messages from the batch:
81
114
82
-
### PartialSQSProcessor context manager
115
+
***Entire batch has been successfully processed**, where your Lambda handler returned successfully, we will let SQS delete the batch to optimize your cost
116
+
***Entire Batch has been partially processed successfully**, where exceptions were raised within your `record handler`, we will:
117
+
-**1)** Delete successfully processed messages from the queue by directly calling `sqs:DeleteMessageBatch`
118
+
-**2)** Raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
119
+
120
+
The only difference is that **PartialSQSProcessor** will give you access to processed messages if you need.
83
121
84
-
If you require access to the result of processed messages, you can use this context manager.
122
+
### Accessing processed messages
85
123
86
-
The result from calling `process()` on the context manager will be a list of all the return values from your `record_handler` function.
124
+
Use `PartialSQSProcessor`context manager to access a list of all return values from your `record_handler` function.
87
125
88
126
=== "app.py"
89
127
90
128
```python
91
129
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
92
130
93
131
def record_handler(record):
94
-
# This will be called for each individual message from a batch
95
-
# It should raise an exception if the message was not processed successfully
96
-
return_value = do_something_with(record["body"])
97
-
return return_value
98
-
132
+
return do_something_with(record["body"])
99
133
100
134
def lambda_handler(event, context):
101
135
records = event["Records"]
@@ -108,7 +142,7 @@ The result from calling `process()` on the context manager will be a list of all
108
142
return result
109
143
```
110
144
111
-
## Passing custom boto3 config
145
+
###Passing custom boto3 config
112
146
113
147
If you need to pass custom configuration such as region to the SDK, you can pass your own [botocore config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) to
114
148
the `sqs_batch_processor` decorator:
@@ -159,30 +193,32 @@ the `sqs_batch_processor` decorator:
159
193
```
160
194
161
195
162
-
## Suppressing exceptions
196
+
###Suppressing exceptions
163
197
164
198
If you want to disable the default behavior where `SQSBatchProcessingError` is raised if there are any errors, you can pass the `suppress_exception` boolean argument.
165
199
166
200
=== "Decorator"
167
201
168
-
```python hl_lines="2"
169
-
...
202
+
```python hl_lines="3"
203
+
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
You can create your own partial batch processor by inheriting the `BasePartialProcessor` class, and implementing `_prepare()`, `_clean()` and `_process_record()`.
188
224
@@ -192,11 +228,9 @@ You can create your own partial batch processor by inheriting the `BasePartialPr
192
228
193
229
You can then use this class as a context manager, or pass it to `batch_processor` to use as a decorator on your Lambda handler function.
194
230
195
-
**Example:**
196
-
197
231
=== "custom_processor.py"
198
232
199
-
```python
233
+
```python hl_lines="3 9 24 30 37 57"
200
234
from random import randint
201
235
202
236
from aws_lambda_powertools.utilities.batch import BasePartialProcessor, batch_processor
@@ -223,14 +257,12 @@ You can then use this class as a context manager, or pass it to `batch_processor
223
257
def _prepare(self):
224
258
# It's called once, *before* processing
225
259
# Creates table resource and clean previous results
0 commit comments