aws_lambda_event_source_mapping: maximum_batching_window_in_seconds #10051

tiny-dancer · 2019-09-10T03:34:52Z

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Release note for CHANGELOG:

* resource/aws_lambda_event_source_mapping: Support `maximum_batching_window_in_seconds` configuration

From AWS Docs:

By default, Lambda invokes your function as soon as records are available in the stream. If the batch it reads from the stream only has one record in it, Lambda only sends one record to the function. To avoid invoking the function with a small number of records, you can tell the event source to buffer records for up to 5 minutes by configuring a batch window. Before invoking the function, Lambda continues to read records from the stream until it has gathered a full batch, or until the batch window expires.

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSLambdaEventSourceMapping_BatchWindow'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -count 1 -parallel 20 -run=TestAccAWSLambdaEventSourceMapping_BatchWindow -timeout 120m
?       github.com/terraform-providers/terraform-provider-aws   [no test files]
=== RUN   TestAccAWSLambdaEventSourceMapping_BatchWindow
=== PAUSE TestAccAWSLambdaEventSourceMapping_BatchWindow
=== CONT  TestAccAWSLambdaEventSourceMapping_BatchWindow
--- PASS: TestAccAWSLambdaEventSourceMapping_BatchWindow (115.84s)
PASS

tiny-dancer · 2019-10-01T15:11:34Z

Minor update, appears AWS updated the API to only accept SQS event source type mapping that have no maximum batching window set. Working with AWS on an open ticket if this is expected as modifying the request to include or exclude the property seems unnecessary.

for those interested:

$ aws lambda update-event-source-mapping --uuid xxxx-xxxx-xxxx- --function-name arn:aws:lambda:us-east-1:xxxx:function:xxx --enabled --batch-size 10 --maximum-batching-window-in-seconds 0

An error occurred (InvalidParameterValueException) when calling the UpdateEventSourceMapping operation: Unsupported MaxiumumBatchingWindowInSecond parameter for given event source mapping type.

$ aws lambda update-event-source-mapping --uuid xxxx-xxxx-xxxx- --function-name arn:aws:lambda:us-east-1:xxxx:function:xxx --enabled --batch-size 10 --maximum-batching-window-in-seconds 1

An error occurred (InvalidParameterValueException) when calling the UpdateEventSourceMapping operation: Unsupported MaxiumumBatchingWindowInSecond parameter for given event source mapping type.

$ aws lambda update-event-source-mapping --uuid xxxx-xxxx-xxxx- --function-name arn:aws:lambda:us-east-1:xxxx:function:xxx --enabled --batch-size 10
{
    "UUID": "f65fdf3f-435c-4250-8bb0-d0ff778ed973",
    "BatchSize": 10,
    "EventSourceArn": "arn:aws:sqs:us-east-1:346166872260:mgrose-lp-cl-loq-queue",
    "FunctionArn": "arn:aws:lambda:us-east-1:346166872260:function:mgrose-lp-cl-s3-producer",
    "LastModified": 1569942296.962,
    "State": "Enabling",
    "StateTransitionReason": "USER_INITIATED"
}

petewilcock · 2019-10-01T15:17:16Z

@tiny-dancer Are you sure? I too encountered this error however it was simply the case that I needed the latest aws-cli installed. This parameter was added very recently, so old versions don't recognise it. See the linked feature request for my example.

tiny-dancer · 2019-10-01T15:32:21Z

@petewilcock good call out with CLI version, will double check.

tiny-dancer · 2019-10-01T15:56:09Z

@petewilcock confirmed request successfully working on a kinesis event source mapping in same cli version. It appears closely aligned with the public announcement they added up front validation to their event source mapping API's, prior to the public announcement it was a little more liberal.

$ aws lambda update-event-source-mapping --uuid xxxx --function-name xxx --enabled --batch-size 10 --maximum-batching-window-in-seconds 0

{
    "UUID": "xxx",
    "BatchSize": 100,
    "MaximumBatchingWindowInSeconds": 299,
    "EventSourceArn": "xxx",
    "FunctionArn": "xxx",
    "LastModified": 1569944880.0,
    "LastProcessingResult": "PROBLEM: Function call failed",
    "State": "Updating",
    "StateTransitionReason": "User action"
}

Also encountered this in the golang SDK via terraform, intercepted API Request:
https://lambda.us-east-1.amazonaws.com/2015-03-31/event-source-mappings/xxxxxxxx

Request:

{
	"BatchSize": 10,
	"Enabled": true,
	"FunctionName": "arn:aws:lambda:us-east-1:xxx:function:xxx",
	"MaximumBatchingWindowInSeconds": 0
}

Response:

{
	"Type": "User",
	"message": "Unsupported MaxiumumBatchingWindowInSecond parameter for given event source mapping type."
}

The same terraform binary successfully updating a Kinesis event source mapping:

tiny-dancer · 2019-10-02T17:27:11Z

The parameter MaximumBatchingWindowInSeconds for the API call UpdateEventSourceMapping is only applicable to Kinesis Stream Event Sources, SQS event sources do not support "MaxiumumBatchingWindowInSecond". Hence, when you try to update this parameter for a SQS event source as you are doing, you are getting this error.

Confirmed via AWS.

v2.33.0

pbhasker · 2019-11-04T18:03:23Z

Is there an ETA for getting this change reviewed and merged?

bflad

Hi @tiny-dancer 👋 Thanks so much for this. Overall its looking really good. Two feedback items and this should be good to go. Please reach out if you have any questions or do not have time to implement them. 👍

aws/resource_aws_lambda_event_source_mapping.go

bflad · 2019-11-19T22:53:11Z

aws/resource_aws_lambda_event_source_mapping_test.go

+					testAccCheckAwsLambdaEventSourceMappingExists(resourceName, &conf),
+					resource.TestCheckResourceAttr(resourceName, "maximum_batching_window_in_seconds", strconv.Itoa(int(batchWindow))),
+				),
+			},


It would be great if there was an additional test step to verify updating maximum_batching_window_in_seconds 👍

Co-Authored-By: Brian Flad <bflad417@gmail.com>

tiny-dancer · 2019-11-19T23:11:31Z

aws/resource_aws_lambda_event_source_mapping.go

+	// AWS API will fail if this parameter is set (even as default value) for sqs event source.  Ideally this should be implemented in GO SDK or AWS API itself.
+	eventSourceArn, err := arn.Parse(d.Get("event_source_arn").(string))
+
+	if err == nil && eventSourceArn.Service != "sqs" {


@bflad your proposed change swallows the err here if it were to exist, was that intentional?

vs returning the err or printing a warning of sorts

Sorry if this was confusing, I should have elaborated on swallowing the error -- if anything, potentially returning a warning log would feel appropriate here since the ARN should have gone through multiple forms of validation at this point (either from Terraform or the API itself). Returning an error could leave operators in a potentially unrecoverable state if for some reason the event source ARN was not an ARN and they made it this far. Hope this makes sense. 😅

Makes sense - thanks for the explanation!

tiny-dancer · 2019-11-19T23:41:21Z

@bflad thanks for the review! addressed feedback and left the err swallowed as proposed in review.

let me know if there's anything else!

bflad

Looks great, thanks, @tiny-dancer 🚀

Output from acceptance testing:

--- PASS: TestAccAWSLambdaEventSourceMapping_sqs_withFunctionName (59.92s)
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_disappears (70.86s)
--- PASS: TestAccAWSLambdaEventSourceMapping_StartingPositionTimestamp (72.83s)
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_removeBatchSize (84.38s)
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_basic (84.60s)
--- PASS: TestAccAWSLambdaEventSourceMapping_BatchWindow (85.49s)
--- PASS: TestAccAWSLambdaEventSourceMapping_sqsDisappears (109.06s)
--- PASS: TestAccAWSLambdaEventSourceMapping_sqs_basic (122.73s)
--- PASS: TestAccAWSLambdaEventSourceMapping_changesInEnabledAreDetected (132.31s)

ghost · 2019-11-21T15:02:48Z

This has been released in version 2.39.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

ghost · 2020-03-29T17:17:39Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

tiny-dancer added 2 commits September 9, 2019 21:58

u[dates

009e1d8

updates

fe19139

tiny-dancer requested a review from a team September 10, 2019 03:34

updates

f1d02d4

tiny-dancer changed the title ~~LambdaEventSourceMapping batch window~~ LambdaEventSourceMapping batch window (MaximumBatchingWindowInSeconds) Sep 10, 2019

tiny-dancer and others added 2 commits September 10, 2019 12:29

Update lambda_event_source_mapping.html.markdown

0217ff7

updates

752013e

ghost added size/M Managed by automation to categorize the size of a PR. and removed size/S Managed by automation to categorize the size of a PR. labels Sep 10, 2019

update name

17f0a09

tiny-dancer changed the title ~~LambdaEventSourceMapping batch window (MaximumBatchingWindowInSeconds)~~ aws_lambda_event_source_mapping: maximum_batching_window_in_seconds Sep 10, 2019

docs

575ac32

eliran-lightricks mentioned this pull request Sep 24, 2019

Support for maximum batching window parameter for lambda event source mapping #10080

Closed

conditionally add maximum batch window

f4d5c9f

ghost added size/S Managed by automation to categorize the size of a PR. and removed size/M Managed by automation to categorize the size of a PR. labels Nov 1, 2019

tiny-dancer added 2 commits November 1, 2019 11:38

Merge tag 'v2.33.0' into feature/eventsourcemapping-batch-window

4a8171a

v2.33.0

better handling of sqs check

f56c030

bflad added the enhancement Requests to existing resources that expand the functionality or scope. label Nov 19, 2019

Merge branch 'master' into feature/eventsourcemapping-batch-window

cde4f69

bflad suggested changes Nov 19, 2019

View reviewed changes

bflad self-assigned this Nov 19, 2019

Update aws/resource_aws_lambda_event_source_mapping.go

39067e4

Co-Authored-By: Brian Flad <bflad417@gmail.com>

ghost added size/M Managed by automation to categorize the size of a PR. and removed size/S Managed by automation to categorize the size of a PR. labels Nov 19, 2019

tiny-dancer commented Nov 19, 2019

View reviewed changes

updates

5a6478e

bflad added this to the v2.39.0 milestone Nov 20, 2019

bflad self-requested a review November 20, 2019 13:44

bflad approved these changes Nov 20, 2019

View reviewed changes

bflad merged commit 2f7e161 into hashicorp:master Nov 20, 2019

bflad added a commit that referenced this pull request Nov 20, 2019

Update CHANGELOG for #10051

8cad6b5

tiny-dancer deleted the feature/eventsourcemapping-batch-window branch November 21, 2019 16:48

ghost locked and limited conversation to collaborators Mar 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws_lambda_event_source_mapping: maximum_batching_window_in_seconds #10051

aws_lambda_event_source_mapping: maximum_batching_window_in_seconds #10051

tiny-dancer commented Sep 10, 2019 •

edited

Loading

tiny-dancer commented Oct 1, 2019 •

edited

Loading

petewilcock commented Oct 1, 2019

tiny-dancer commented Oct 1, 2019

tiny-dancer commented Oct 1, 2019 •

edited

Loading

tiny-dancer commented Oct 2, 2019

pbhasker commented Nov 4, 2019

bflad left a comment

bflad Nov 19, 2019

tiny-dancer Nov 19, 2019

tiny-dancer Nov 19, 2019

bflad Nov 20, 2019

tiny-dancer Nov 20, 2019

tiny-dancer commented Nov 19, 2019

bflad left a comment

ghost commented Nov 21, 2019

ghost commented Mar 29, 2020

aws_lambda_event_source_mapping: maximum_batching_window_in_seconds #10051

aws_lambda_event_source_mapping: maximum_batching_window_in_seconds #10051

Conversation

tiny-dancer commented Sep 10, 2019 • edited Loading

Community Note

tiny-dancer commented Oct 1, 2019 • edited Loading

petewilcock commented Oct 1, 2019

tiny-dancer commented Oct 1, 2019

tiny-dancer commented Oct 1, 2019 • edited Loading

tiny-dancer commented Oct 2, 2019

pbhasker commented Nov 4, 2019

bflad left a comment

Choose a reason for hiding this comment

bflad Nov 19, 2019

Choose a reason for hiding this comment

tiny-dancer Nov 19, 2019

Choose a reason for hiding this comment

tiny-dancer Nov 19, 2019

Choose a reason for hiding this comment

bflad Nov 20, 2019

Choose a reason for hiding this comment

tiny-dancer Nov 20, 2019

Choose a reason for hiding this comment

tiny-dancer commented Nov 19, 2019

bflad left a comment

Choose a reason for hiding this comment

ghost commented Nov 21, 2019

ghost commented Mar 29, 2020

tiny-dancer commented Sep 10, 2019 •

edited

Loading

tiny-dancer commented Oct 1, 2019 •

edited

Loading

tiny-dancer commented Oct 1, 2019 •

edited

Loading