(CLI): hotswapping should wait for Lambda's updateFunctionCode operation to complete
#18386
Comments
skinny85
added
feature-request
skinny85
added
effort/medium
skinny85
changed the title
updateFunctionCode operation to completeupdateFunctionCode operation to complete
|
After looking into this, there are some nuances to account for. Here is what I've found. There are a couple pieces of information that we get from Lambda after updating the function code.
|
|
Awesome work @corymhall researching this, super thorough! I agree with all of your recommendations 🙂. Given that, do you have enough information to start working on implementing this? |
|
@skinny85 yep I've already started working on it :) |
|
@corymhall one last question. I understand your comment gives a great summary of the current state. Do you know how will that description be affected by the changes outlined in the blog post linked in the issue description, that are scheduled to go out on February 1? |
|
@skinny85 my comment describes the state with those changes in place. In the current state the |
…mplete (#18536) There are [upcoming changes](https://aws.amazon.com/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/) that will rollout Lambda states to all Lambda Functions. Prior to this update (current functionality) when you made an `updateFunctionCode` request the function was immediately available for both invocation and future updates. Once this change is rolled out this will no longer be the case. With Lambda states, when you make an update to a Lambda Function, it will not be available for future updates until the `LastUpdateStatus` returns `Successful`. This PR introduces a custom waiter that will wait for the update to complete before proceeding. The waiter will wait until the `State=Active` and the `LastUpdateStatus=Successful`. The `State` controls whether or not the function can be invoked, and the `LastUpdateStatus` controls whether the function can be updated. Based on this, I am not considering a deployment complete until both are successful. To see a more in depth analysis of the different values see #18386. In my testing I found that the time it took for a function to go from `LastUpdateStatus=InProgress` to `LastUpdateStatus=Successful` was: - ~1 second for a zip Function not in a VPC - ~25 seconds for a container Function or a Function in a VPC - ~2 minutes to deploy a VPC function (best proxy for StateReasonCode=Restoring) There are a couple of built in waiters that could have been used for this, namely [functionUpdated](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#functionUpdated-waiter). This waiter uses `getFunctionConfiguration` which has a quota of 15 requests/second. In addition the waiter polls every 5 seconds and this cannot be configured. Because hotswapping is sensitive to any latency that is introduced, I created a custom waiter that uses `getFunction`. `getFunction` has a quota of 100 requests/second and the custom waiter can be configured to poll every 1 second or every 5 seconds depending on what type of function is being updated. fixes #18386 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
…mplete (aws#18536) There are [upcoming changes](https://aws.amazon.com/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/) that will rollout Lambda states to all Lambda Functions. Prior to this update (current functionality) when you made an `updateFunctionCode` request the function was immediately available for both invocation and future updates. Once this change is rolled out this will no longer be the case. With Lambda states, when you make an update to a Lambda Function, it will not be available for future updates until the `LastUpdateStatus` returns `Successful`. This PR introduces a custom waiter that will wait for the update to complete before proceeding. The waiter will wait until the `State=Active` and the `LastUpdateStatus=Successful`. The `State` controls whether or not the function can be invoked, and the `LastUpdateStatus` controls whether the function can be updated. Based on this, I am not considering a deployment complete until both are successful. To see a more in depth analysis of the different values see aws#18386. In my testing I found that the time it took for a function to go from `LastUpdateStatus=InProgress` to `LastUpdateStatus=Successful` was: - ~1 second for a zip Function not in a VPC - ~25 seconds for a container Function or a Function in a VPC - ~2 minutes to deploy a VPC function (best proxy for StateReasonCode=Restoring) There are a couple of built in waiters that could have been used for this, namely [functionUpdated](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#functionUpdated-waiter). This waiter uses `getFunctionConfiguration` which has a quota of 15 requests/second. In addition the waiter polls every 5 seconds and this cannot be configured. Because hotswapping is sensitive to any latency that is introduced, I created a custom waiter that uses `getFunction`. `getFunction` has a quota of 100 requests/second and the custom waiter can be configured to poll every 1 second or every 5 seconds depending on what type of function is being updated. fixes aws#18386 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
…mplete (aws#18536) There are [upcoming changes](https://aws.amazon.com/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/) that will rollout Lambda states to all Lambda Functions. Prior to this update (current functionality) when you made an `updateFunctionCode` request the function was immediately available for both invocation and future updates. Once this change is rolled out this will no longer be the case. With Lambda states, when you make an update to a Lambda Function, it will not be available for future updates until the `LastUpdateStatus` returns `Successful`. This PR introduces a custom waiter that will wait for the update to complete before proceeding. The waiter will wait until the `State=Active` and the `LastUpdateStatus=Successful`. The `State` controls whether or not the function can be invoked, and the `LastUpdateStatus` controls whether the function can be updated. Based on this, I am not considering a deployment complete until both are successful. To see a more in depth analysis of the different values see aws#18386. In my testing I found that the time it took for a function to go from `LastUpdateStatus=InProgress` to `LastUpdateStatus=Successful` was: - ~1 second for a zip Function not in a VPC - ~25 seconds for a container Function or a Function in a VPC - ~2 minutes to deploy a VPC function (best proxy for StateReasonCode=Restoring) There are a couple of built in waiters that could have been used for this, namely [functionUpdated](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#functionUpdated-waiter). This waiter uses `getFunctionConfiguration` which has a quota of 15 requests/second. In addition the waiter polls every 5 seconds and this cannot be configured. Because hotswapping is sensitive to any latency that is introduced, I created a custom waiter that uses `getFunction`. `getFunction` has a quota of 100 requests/second and the custom waiter can be configured to poll every 1 second or every 5 seconds depending on what type of function is being updated. fixes aws#18386 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Description
Right now, when hotswapping Lambda code, we don't wait for the
updateFunctionCodeAPI call we make to complete.However, according to the documentation, this operation is actually eventually consistent today for Lambda functions that use Docker images, or are present in a VPC, and, even more importantly, this API will become eventually consistent for all Functions starting February 1, 2022.
We need to change our logic to wait for the operation to complete.
While implementing this, we should be cognizant of making sure we wait as effectively as possible, to not affect Lambda hotswap times too adversely. The simplest solution might be to use the standard Lambda waiter, that we use if we need to publish a new Version. Perhaps we can use a custom Waiter for this purpose, like we do for ECS service hotswapping.
Thanks a lot to @tmokmss who figured out all of this while working on Docker image Function hotswapping.
Acknowledge
The text was updated successfully, but these errors were encountered: