(lambda-event-sources): on_failure parameter is not supported for SelfManagedKafkaEventSource

[sawalanipawan]

Describe the bug

As per (https://docs.aws.amazon.com/cdk/api/v1/python/aws_cdk.aws_lambda_event_sources/SelfManagedKafkaEventSource.html
https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_lambda_event_sources/SelfManagedKafkaEventSource.html) we can specify on_failure parameter in the SelfManagedKafkaEventSource.

But the CloudFormation stack deployment fails with below error:

"Resource handler returned message: "Invalid request provided: Unsupported 'destinationConfig' parameter for given event source mapping type. (Service: Lambda, Status Code: 400)"

Upon checking the CFN documentation (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html#cfn-lambda-eventsourcemapping-destinationconfig) for AWS::Lambda::EventSourceMapping resource, adding destinationConfig is only supported for 'Stream' event sources, which are DynamoDB and Kinesis.

I spun up a test Kafka cluster and then using CDK I created the lambda function that has self-managed Apache Kafka event source.

        mylambda = lambda_.Function(self, "MyLambda",
            code=lambda_.Code.from_asset("lambda"),
            handler="index.handler",
            runtime=lambda_.Runtime.NODEJS_12_X,
            role=role
        )

        mylambda.add_event_source(SelfManagedKafkaEventSource(
            bootstrap_servers=bootstrap_servers,
            topic=topic,
            batch_size=100,  # default
            starting_position=lambda_.StartingPosition.LATEST,
            vpc=vpc,
            vpc_subnets=ec2.SubnetSelection(
                subnet_type=ec2.SubnetType.PRIVATE
            ),
            security_group=security_group
            on_failure=SqsDlq(dead_letter_queue)
        ))

The cdk synth command ran successfully.
After running the cdk deploy command, the corresponding CFN stack failed with below error:

Resource handler returned message: "Invalid request provided: Unsupported 'destinationConfig' parameter for given event source mapping type. (Service: Lambda, Status Code: 400)

This implies that on_failure parameter is not supported for SelfManagedKafkaEventSource.

Also, on AWS console, if you try adding the Apache Kafka event source, you will not find the On-failure destination option.

But you can see the On-failure destination option for DynamoDB and Kinesis event source.

Expected Behavior

For AWS::Lambda::EventSourceMapping resource, adding DestinationConfig is only supported for 'Stream' event sources, which are DynamoDB and Kinesis.

So, this option should be removed from SelfManagedKafkaEventSource.

Current Behavior

The cdk synth command ran successfully.
After running the cdk deploy command, the corresponding CFN stack failed with below error:

Resource handler returned message: "Invalid request provided: Unsupported 'destinationConfig' parameter for given event source mapping type. (Service: Lambda, Status Code: 400)

Reproduction Steps

        mylambda = lambda_.Function(self, "MyLambda",
            code=lambda_.Code.from_asset("lambda"),
            handler="index.handler",
            runtime=lambda_.Runtime.NODEJS_12_X,
            role=role
        )

        mylambda.add_event_source(SelfManagedKafkaEventSource(
            bootstrap_servers=bootstrap_servers,
            topic=topic,
            batch_size=100,  # default
            starting_position=lambda_.StartingPosition.LATEST,
            vpc=vpc,
            vpc_subnets=ec2.SubnetSelection(
                subnet_type=ec2.SubnetType.PRIVATE
            ),
            security_group=security_group
            on_failure=SqsDlq(dead_letter_queue)
        ))

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

1.152.0 (build 9487b39)

Framework Version

No response

Node.js Version

v16.13.0

OS

Mac

Language

Python

Language Version

No response

Other information

No response

[kaizencc]

Apologies about this, @sawalanipawan! This is a poor user experience we should rectify

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.