aws-sdk: Increase in TimeoutErrors in Amplify hotswap builds

[ShadowCat567]

Describe the bug

Starting from 11/14, we have started seeing TimeoutError: Resource is not in the expected state due to waiter status: TIMEOUT, with a frequency we have never seen before. This error was added here in version 2.167.0.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

2.166.0

Expected Behavior

Very few to none TimeoutError: Resource is not in the expected state due to waiter status: TIMEOUT appearing to our customers.

Current Behavior

TimeoutError: Resource is not in the expected state due to waiter status: TIMEOUT has become one of the most common error messages our customers are receiving.

Reproduction Steps

1. Setup amplify (Gen2) https://docs.amplify.aws/react/start/manual-installation/
2. Setup a function https://docs.amplify.aws/react/build-a-backend/functions/set-up-function/
3. Make a change in lambda code

Possible Solution

Roll back this PR: #31702

Additional Information/Context

No response

CDK CLI Version

2.167.0

Framework Version

No response

Node.js Version

=Node18

OS

Linux/Mac/Windows

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Can't reproduce this using 2.168.0. The lambda func hot swapping looks good to me. Please provide more reproducible details as well as any screenshots if possible. Thank you.

[pahud]

Also, our latest version is 2.168.0 now. Can you verify with this version?

[ShadowCat567]

I have edited the description to provide some additional info about repro instructions, but this issue is very fickle and is hard to reproduce consistently. It does appear in the most recent cdk version.

[pahud]

I can't reproduce it but this could be the potential root cause:

The timeout occurs because AWS Lambda requires the function to be in an Active state and have LastUpdateStatus=Successful before it can be updated again. CDK waits for this state before proceeding, but has different wait times depending on the Lambda configuration:

aws-cdk/packages/aws-cdk/lib/api/hotswap/lambda-functions.ts

Lines 333 to 336 in 27babe6

	await lambda.waitUntilFunctionUpdated(delaySeconds, {
	FunctionName: functionName,
	});
	}

AWS SDK v3 has a different waiter implementation, timeout handling and delay strategy that handle requests differently from v2. This could affect how timeouts are processed.

I guess we should either increase the timeout somehow from here or add status checking like

const response = await lambda.getFunctionConfiguration({
  FunctionName: functionName
});
if (response.State === 'Active' && response.LastUpdateStatus === 'Successful') {
  // Proceed with hotswap
}

I'll bring this up to the core team for further investigation.

[otaviomacedo]

Just to be clear, this message was introduced in version 2.167.0. Previously, the same error would result in a generic error message. So, the fact that we are now seeing it in the wild, by itself, doesn't imply that there is anything wrong there. Indeed, it would be surprising if this message didn't start appearing.

To establish that there is an issue, we need to find a case in which this error message is shown in version 2.167.0 or later, and no error happens at all in earlier versions. Given that it's hard to reproduce it consistently, we would need to run each version a few times and compare the error rates.

[sobolk]

@otaviomacedo

The issue surfaced immediately in our e2e tests after we merged CDK version bump here aws-amplify/amplify-backend#2269 .

The failure https://github.com/aws-amplify/amplify-backend/actions/runs/12036537741/job/33558355227 .

It happened in all three jobs of same kind at first try. I can't find/recall examples of these tests failing due to timeout before.

[sobolk]

It has consistently failed in 5/6 runs.
From E2E test logs, It seems this is affected by:

1. Growing number of resources being hot-swapped
2. Hotswapps are executed couple of times in a row (they may trigger throttling and new code might be less tolerant to that then before.).

[sobolk]

I was able to establish local repro that explains why our tests are failing.

1. Assume IAM principal that has only AmplifyBackendDeployFullAccess policy attached.
2. Execute https://github.com/aws-amplify/amplify-backend/blob/main/packages/integration-tests/src/test-e2e/sandbox/data_storage_auth_with_triggers.sandbox.test.ts .

It seems that new waiter implementation is now requiring new IAM permissions to function.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.