Lambda with SnapStart enabled and Publish=True will block deployment of other lambda functions in the same program

[automagic]

What happened?

When updating multiple lambda function in the same program, enabling snapstart will cause each function to block until the previous publish is complete, which takes about 3min (sometimes longer) in AWS. In a stack with 20 lambdas, this can cause the deployment to take 1+ hrs.

Example

Example function config causing this issue:

lambda_handler = aws.lambda_.Function(
    f"{local_name}_handler",
    name=f"{local_name}_handler",
    role=iam_role_for_lambda.arn,
    package_type="Zip",
    runtime="java17",
    layers=["arn:aws:lambda:us-east-1:580247275435:layer:LambdaInsightsExtension:38"],
    code=FileArchive("..lambda.jar"),
    handler="handler",
    snap_start=aws.lambda_.FunctionSnapStartArgs(apply_on="PublishedVersions"),
    timeout=60,
    memory_size=768,
    vpc_config=aws.lambda_.FunctionVpcConfigArgs(
        subnet_ids=vpc.private_subnet_ids,
        security_group_ids=[security_group_for_lambdas.id]
    ),
    tracing_config=aws.lambda_.FunctionTracingConfigArgs(mode="PassThrough"),
    publish=True,
)

Output of pulumi about

3.111.0

Additional context

Example trace of multiple lambda blocking for 3min

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[justinvp]

This looks specific to AWS, so I'm going to transfer the issue to the AWS provider repo and someone will take a look.

[corymhall]

@automagic it looks like this will take some time to investigate the root cause. Do you know what the urgency of this is? Do we need to find a solution this week or next?

[mikhailshilkov]

This is a duplicate of #2206

The sequential update behavior doesn't really depend on SnapStart or Publish=True but those properties magnify the issue due to each Code Update operation taking minutes instead of seconds.

I'll close this issue as a duplicate but will add a comment to the original one.