InvalidPayloadLength using Kinesis Enhanced Fan-Out consumer

[mjp9zn]

Describe the bug

InvalidPayloadLength('Payload length of 19271862 exceeded the maximum of 16777216')

Code location

We're processing a Kinesis stream using an Enhanced Fan-Out consumer, and see the above error raised by Botocore. This error is being raised because the payload sent by Kinesis is greater than a hard-coded _MAX_PAYLOAD_LENGTH at 16Mb. There is no recovery from this, as the stream will continue to attempt to send the same event that exceeds the payload max length on retry.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I would expect that the _MAX_PAYLOAD_LENGTH value be configured to match the limits of Kinesis when processing as an Enhanced Fan-Out consumer. If Kinesis doesn't have an inherent limit, then this value should be easily configurable or recoverable by clients.

Current Behavior

InvalidPayloadLength('Payload length of 19271862 exceeded the maximum of 16777216')

Reproduction Steps

This isn't very useful as reproduction depends on external state in Kinesis

import boto3

client = boto3.client("kinesis", region_name="us-east-1")
subscription_params = dict(
    ConsumerARN=...,
    ShardId=...,
    StartingPosition=...,
)
subscription = client.subscribe_to_shard(**subscription_params)
event_stream = subscription["EventStream"]
for event in event_stream:
    print(event)

Possible Solution

No response

Additional Information/Context

Worked around the issue with the following snippet, which is far from ideal for production code:

import botocore.eventstream

botocore.eventstream._MAX_PAYLOAD_LENGTH = 20 * 1024**2  # 20MB

SDK version used

1.35.77

Environment details (OS name and version, etc.)

AWS ECS container running image based off python:3.12.7-slim on AWS EC2 Linux/X86_64 (al2023-ami-ecs-hvm-2023.0.20240610-kernel-6.1-x86_64)

[khushail]

Hi @mjp9zn , thanks for reporting this. The internal specification for the protocol suggests there is a hard upper limit of 16MB for payload size. I have reached out to internal team for getting more context on this limit and get their insights. Also would file a ticket with Kinesis team to have this checked.

Will post updates soon.

Thanks.

[khushail]

Ticket created with Kinesis team - P195678195

EDIT: I have closed the ticket with Kinesis team. Looks like this issue is existent with all SDKs. Team is working on a fix.