sam sync --watch race condition when syncing Stack with multiple Lambdas within same .NET project

[theimowski]

Description:

When trying to sam sync --watch a Stack with multiple .NET Lambdas that use same .NET project, there's a race condition when updating source code - likely related to fact that multiple processes are invoked for same project.

Steps to reproduce:

1. Clone reproduction repository https://github.com/theimowski/sam-sync-watch-net-multi-lambdas
2. Run sam sync --watch

Observed result:

> sam sync --stack-name TheimowskiTestSAMAccelerate --watch

                Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-***

                Default capabilities applied: ('CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND')
To override with customized capabilities, use --capabilities flag or set it in samconfig.toml

This feature is currently in beta. Visit the docs page to learn more about the AWS Beta terms https://aws.amazon.com/service-terms/.

The SAM CLI will use the AWS Lambda, Amazon API Gateway, and AWS StepFunctions APIs to upload your code without
performing a CloudFormation deployment. This will cause drift in your CloudFormation stack.
**The sync command should only be used against a development stack**.

Confirm that you are synchronizing a development stack and want to turn on beta features.

Enter Y to proceed with the command, or enter N to cancel:
 [y/N]: y

Experimental features are enabled for this session.
Visit the docs page to learn more about the AWS Beta terms https://aws.amazon.com/service-terms/.

Queued infra sync. Wating for in progress code syncs to complete...
Starting infra sync.
Cache is invalid, running build and copying resources to function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Building codeuri: $path\sam-app\src\HelloWorld runtime: dotnetcore3.1 metadata: {} architecture: x86_64 functions: ['HelloWorldFunction', 'AnotherFunction']
Running DotnetCliPackageBuilder:GlobalToolInstall

Tool 'amazon.lambda.tools' was reinstalled with the latest stable version (version '5.2.0').
Running DotnetCliPackageBuilder:RunPackageAction
Amazon Lambda Tools for .NET Core applications (5.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder '$path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish'
... dotnet publish --output "$path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish" --configuration "Release" --framework "netcoreapp3.1" --runtime linux-x64 /p:GenerateRuntimeConfigurationFiles=true --self-contained false
... publish: Microsoft (R) Build Engine version 16.8.3+39993bd9d for .NET
... publish: Copyright (C) Microsoft Corporation. All rights reserved.
... publish:   Determining projects to restore...
... publish:   All projects are up-to-date for restore.
... publish:   HelloWorld -> $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\linux-x64\HelloWorld.dll
... publish:   HelloWorld -> $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish\
Zipping publish folder $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish to $path\sam-app\.aws-sam\auto-dependency-layer\HelloWorldFunction\HelloWorld.zip
Creating directory $path\sam-app\.aws-sam\auto-dependency-layer\HelloWorldFunction
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Serialization.Json.dll
... zipping: HelloWorld.deps.json
... zipping: HelloWorld.dll
... zipping: HelloWorld.pdb
... zipping: HelloWorld.runtimeconfig.json
... zipping: Newtonsoft.Json.dll
Created publish archive ($path\sam-app\.aws-sam\auto-dependency-layer\HelloWorldFunction\HelloWorld.zip).
Lambda project successfully packaged: $path\sam-app\.aws-sam\auto-dependency-layer\HelloWorldFunction\HelloWorld.zip

Build Succeeded

Built Artifacts  : .aws-sam\auto-dependency-layer
Built Template   : .aws-sam\auto-dependency-layer\template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t .aws-sam\auto-dependency-layer\template.yaml
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
[*] Deploy: sam deploy --guided --template-file .aws-sam\auto-dependency-layer\template.yaml


Successfully packaged artifacts and wrote output template to file $path\AppData\Local\Temp\tmpgg6mieyb.
Execute the following command to deploy the packaged template
sam deploy --template-file $path\AppData\Local\Temp\tmpgg6mieyb --stack-name <YOUR STACK NAME>


        Deploying with following values
        ===============================
        Stack name                   : TheimowskiTestSAMAccelerate
        Region                       : eu-central-1
        Disable rollback             : False
        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-***
        Capabilities                 : ["CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND"]
        Parameter overrides          : {}
        Signing Profiles             : null

Initiating deployment
=====================

2022-01-17 11:26:33 - Waiting for stack create/update to complete

CloudFormation events from stack operations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                      ResourceType                                        LogicalResourceId                                   ResourceStatusReason
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UPDATE_IN_PROGRESS                                  AWS::CloudFormation::Stack                          TheimowskiTestSAMAccelerate                         Transformation succeeded
UPDATE_COMPLETE_CLEANUP_IN_PROGRESS                 AWS::CloudFormation::Stack                          TheimowskiTestSAMAccelerate                         -
UPDATE_COMPLETE                                     AWS::CloudFormation::Stack                          TheimowskiTestSAMAccelerate                         -
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 HelloWorldFunctionIamRole
Description         Implicit IAM Role created for Hello World function
Value               arn:aws:iam::***:role/TheimowskiTestSAMAccelerate-HelloWorldFunctionRole-***

Key                 AnotherFunctionIamRole
Description         Implicit IAM Role created for Another function
Value               arn:aws:iam::***:role/TheimowskiTestSAMAccelerate-AnotherFunctionRole-***

Key                 HelloWorldApi
Description         API Gateway endpoint URL for Prod stage for Hello World function
Value               https://***.execute-api.eu-central-1.amazonaws.com/Prod/hello/

Key                 AnotherFunction
Description         Another Lambda Function ARN
Value               arn:aws:lambda:eu-central-1:***:function:TheimowskiTestSAMAccelerate-AnotherFunction-***

Key                 HelloWorldFunction
Description         Hello World Lambda Function ARN
Value               arn:aws:lambda:eu-central-1:***:function:TheimowskiTestSAMAccelerate-HelloWorldFunction-***
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Stack update succeeded. Sync infra completed.

{'StackId': 'arn:aws:cloudformation:eu-central-1:***:stack/TheimowskiTestSAMAccelerate/515878e0-777e-11ec-8cc1-06e3a48196a2', 'ResponseMetadata': {'RequestId': 'f1e977a4-8c8d-41cc-87b6-7217f179afe4', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'f1e977a4-8c8d-41cc-87b6-7217f179afe4', 'content-type': 'text/xml', 'content-length': '400', 'date': 'Mon, 17 Jan 2022 10:26:33 GMT'}, 'RetryAttempts': 0}}
Infra sync completed.
Syncing Lambda Function HelloWorldFunction...
Syncing Lambda Function AnotherFunction...
Cache is invalid, running build and copying resources to function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Building codeuri: $path\sam-app\src\HelloWorld runtime: dotnetcore3.1 metadata: {} architecture: x86_64 functions: ['HelloWorldFunction']
Running DotnetCliPackageBuilder:GlobalToolInstall
Skipping to update Amazon.Lambda.Tools install/update, since it is updated recently
Running DotnetCliPackageBuilder:RunPackageAction
Cache is invalid, running build and copying resources to function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Building codeuri: $path\sam-app\src\HelloWorld runtime: dotnetcore3.1 metadata: {} architecture: x86_64 functions: ['AnotherFunction']
Running DotnetCliPackageBuilder:GlobalToolInstall
Skipping to update Amazon.Lambda.Tools install/update, since it is updated recently
Running DotnetCliPackageBuilder:RunPackageAction
Amazon Lambda Tools for .NET Core applications (5.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder '$path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish'
... dotnet publish --output "$path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish" --configuration "Release" --framework "netcoreapp3.1" --runtime linux-x64 /p:GenerateRuntimeConfigurationFiles=true --self-contained false
... publish: Microsoft (R) Build Engine version 16.8.3+39993bd9d for .NET
... publish: Copyright (C) Microsoft Corporation. All rights reserved.
... publish:   Determining projects to restore...
... publish:   All projects are up-to-date for restore.
... publish:   HelloWorld -> $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\linux-x64\HelloWorld.dll
... publish:   HelloWorld -> $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish\
Zipping publish folder $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish to $path\sam-app\.aws-sam\auto-dependency-layer\HelloWorldFunction\HelloWorld.zip
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Serialization.Json.dll
... zipping: HelloWorld.deps.json
... zipping: HelloWorld.dll
... zipping: HelloWorld.pdb
... zipping: HelloWorld.runtimeconfig.json
... zipping: Newtonsoft.Json.dll
Created publish archive ($path\sam-app\.aws-sam\auto-dependency-layer\HelloWorldFunction\HelloWorld.zip).
Lambda project successfully packaged: $path\sam-app\.aws-sam\auto-dependency-layer\HelloWorldFunction\HelloWorld.zip
Amazon Lambda Tools for .NET Core applications (5.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command
... invoking 'dotnet publish', working folder '$path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish'
... dotnet publish --output "$path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish" --configuration "Release" --framework "netcoreapp3.1" --runtime linux-x64 /p:GenerateRuntimeConfigurationFiles=true --self-contained false
... publish: Microsoft (R) Build Engine version 16.8.3+39993bd9d for .NET
... publish: Copyright (C) Microsoft Corporation. All rights reserved.
... publish:   Determining projects to restore...
... publish:   All projects are up-to-date for restore.
... publish:   HelloWorld -> $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\linux-x64\HelloWorld.dll
... publish:   HelloWorld -> $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish\
Zipping publish folder $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish to $path\sam-app\.aws-sam\auto-dependency-layer\AnotherFunction\HelloWorld.zip
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Serialization.Json.dll
... zipping: HelloWorld.deps.json
... zipping: HelloWorld.dll
... zipping: HelloWorld.pdb
... zipping: HelloWorld.runtimeconfig.json
... zipping: Newtonsoft.Json.dll
Created publish archive ($path\sam-app\.aws-sam\auto-dependency-layer\AnotherFunction\HelloWorld.zip).
Lambda project successfully packaged: $path\sam-app\.aws-sam\auto-dependency-layer\AnotherFunction\HelloWorld.zip
Code sync encountered an error.
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\sync\sync_flow_executor.py", line 335, in _sync_flow_execute_wrapper
    dependent_sync_flows = sync_flow.execute()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\sync\sync_flow.py", line 291, in execute
    self.gather_resources()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\sync\flows\zip_function_sync_flow.py", line 93, in gather_resources
    build_result = builder.build()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\app_builder.py", line 200, in build
    return ApplicationBuildResult(build_graph, build_strategy.build())
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\build_strategy.py", line 504, in build
    result.update(super().build())
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\build_strategy.py", line 73, in build
    result.update(self._build_functions(self._build_graph))
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\build_strategy.py", line 84, in _build_functions
    function_build_results.update(self.build_single_function_definition(build_definition))
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\build_strategy.py", line 521, in build_single_function_definition
    return self._cached_build_strategy.build_single_function_definition(build_definition)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\build_strategy.py", line 268, in build_single_function_definition
    shutil.rmtree(str(cache_function_dir))
  File "shutil.py", line 740, in rmtree
  File "shutil.py", line 618, in _rmtree_unsafe
  File "shutil.py", line 616, in _rmtree_unsafe
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '$path\\.aws-sam\\cache\\f4f2dfc9-eddf-4622-9a5e-6006626996fa\\Amazon.Lambda.APIGatewayEvents.dll'
Updated source_hash and manifest_hash field in build.toml for function with UUID f4f2dfc9-eddf-4622-9a5e-6006626996fa
Syncing Lambda Function AnotherFunction...
Cache is invalid, running build and copying resources to function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Building codeuri: $path\sam-app\src\HelloWorld runtime: dotnetcore3.1 metadata: {} architecture: x86_64 functions: ['AnotherFunction']
Running DotnetCliPackageBuilder:GlobalToolInstall
Skipping to update Amazon.Lambda.Tools install/update, since it is updated recently
Running DotnetCliPackageBuilder:RunPackageAction
Finished syncing Lambda Function HelloWorldFunction.
Syncing Lambda Function HelloWorldFunction...
Amazon Lambda Tools for .NET Core applications (5.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder '$path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish'
... dotnet publish --output "$path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish" --configuration "Release" --framework "netcoreapp3.1" --runtime linux-x64 /p:GenerateRuntimeConfigurationFiles=true --self-contained false
... publish: Microsoft (R) Build Engine version 16.8.3+39993bd9d for .NET
... publish: Copyright (C) Microsoft Corporation. All rights reserved.
... publish:   Determining projects to restore...
... publish:   All projects are up-to-date for restore.
... publish:   HelloWorld -> $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\linux-x64\HelloWorld.dll
... publish:   HelloWorld -> $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish\
Zipping publish folder $path\sam-app\src\HelloWorld\bin\Release\netcoreapp3.1\publish to $path\sam-app\.aws-sam\auto-dependency-layer\AnotherFunction\HelloWorld.zip
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Serialization.Json.dll
... zipping: HelloWorld.deps.json
... zipping: HelloWorld.dll
... zipping: HelloWorld.pdb
... zipping: HelloWorld.runtimeconfig.json
... zipping: Newtonsoft.Json.dll
Created publish archive ($path\sam-app\.aws-sam\auto-dependency-layer\AnotherFunction\HelloWorld.zip).
Lambda project successfully packaged: $path\sam-app\.aws-sam\auto-dependency-layer\AnotherFunction\HelloWorld.zip
Updated source_hash and manifest_hash field in build.toml for function with UUID f4f2dfc9-eddf-4622-9a5e-6006626996fa
Valid cache found, copying previously built resources from function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Finished syncing Lambda Function HelloWorldFunction.
Syncing Lambda Function HelloWorldFunction...
Valid cache found, copying previously built resources from function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Finished syncing Lambda Function HelloWorldFunction.
Syncing Lambda Function HelloWorldFunction...
Finished syncing Lambda Function AnotherFunction.
Syncing Lambda Function AnotherFunction...
Valid cache found, copying previously built resources from function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Valid cache found, copying previously built resources from function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Finished syncing Lambda Function HelloWorldFunction.
Finished syncing Lambda Function AnotherFunction.
Syncing Lambda Function HelloWorldFunction...
Valid cache found, copying previously built resources from function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Syncing Lambda Function AnotherFunction...
Valid cache found, copying previously built resources from function build definition of f4f2dfc9-eddf-4622-9a5e-6006626996fa
Finished syncing Lambda Function HelloWorldFunction.
Finished syncing Lambda Function AnotherFunction.

Expected result:

sam sync should not fail with PermissionError: [WinError 32] The process cannot access the file because it is being used by another process error

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10
2. sam --version: 1.37.0
3. AWS region: eu-central-1

[qingchm]

Hi there @theimowski thanks for your input! I was able to reproduce similar race conditions using your project, when I was copying the cache dir it says the directory already exists while the previous rmtree should have guaranteed a clean cache dir. We will take this to the team and investigate the root cause of your cached build directory being used by multiple processes! Again thanks for reporting this

[mndeveci]

This issue should be resolved with SAM CLI v1.50.0