Feature request: support batching of AWS SDK calls in chunks

[PaulvdBijlNL]

Use case

The batch processing feature helps with handling failures in batches. Besides this, another reason to use batching is the possibility to batch (AWS SDK) calls. Examples of this are PutEventsCommand for EventBridge and BatchWriteItemCommand. The possibility to batch such calls will reduce the costs and increase performance.

With the current implementation, the only possibility to support batching of AWS SDK calls is by customizing the batch processor, as is shown https://docs.powertools.aws.dev/lambda/typescript/latest/utilities/batch/#create-your-own-partial-processor in the official documentation.

Solution/User Experience

Currently for an unexperienced user it might not be obvious that the current batching approach is not ideal. Events are processed in batches, but they are still processed one-by-one. This feature would make the experience of working with batching within an AWS Lambda function more smooth.

It would help the user to not have to write some boilerplate code by supporting:

• Allow to return the input for the AWS SDK call as return value from the recordHandler
• Combine all returned values of the recordHandler calls into params for the AWS SDK call
• Do the AWS SDK call based on what's configured when creating the batch processor
• Handle the result and mark all items that were not successful as 'failed' in the batch

Alternative solutions

As far as I can see there is no generic batching solution for AWS SDK calls, but each AWS SDK call that supports batching uses different input and output types. This means it will require custom code for each AWS SDK call. This seems like a lot of effort.

An alternative would be to support hiding some of the boilerplate code. Allowing to return a custom type from each recordHandler call and having a hook to process these and indicate which ones succeeded and failed. The current clean hook does not fully support this.

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @PaulvdBijlNL, thank you for opening this issue and for the detailed description.

As mentioned during our offline conversation a few days ago, while I think the name of the utility (Batch Processor) might be confusing, I also think the use case you describe might be a better for a dedicated utility - i.e. Chunk Processor or something like this.

My reasoning here is that the Batch Processor in its current form, focuses mainly on processing a batch of records in a consumer (SQS|Kinesis|DynamoDB Streams -> Lambda) as opposed to process items in batch acting as producer (Lambda ->).

With that said, I think a batcher utility that gets a set of items, creates chunks, and processes one chunk at the time as a unit could be interesting to consider.

Before deciding to move forward with an RFC, I'd like to hear the opinion of both other maintainers (@aws-powertools/lambda-python-core, @aws-powertools/lambda-typescript-core) as well as other customers.