sam sync will break Lambda functions written in Java

[roamingthings]

Description:

When using sam sync on a Lambda function written in Java the function fails with an exception:

Steps to reproduce:

Create an application:

sam init --name hello-world-lambda \
--runtime java11 \
--dependency-manager gradle \
--app-template hello-world

Build and deploy

sam build
sam deploy --guided

Set parameters

Setting default arguments for 'sam deploy'
=========================================
Stack Name [sam-app]: sam-sync-test
AWS Region [eu-central-1]:
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [y/N]: y
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]:
#Preserves the state of previously provisioned resources when an operation fails
Disable rollback [y/N]:
HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
Save arguments to configuration file [Y/n]:
SAM configuration file [samconfig.toml]:
SAM configuration environment [default]:

Change HelloWorldFunction/src/main/java/helloworld/App.java (e.g. change the output string)

sam sync

Observed result:

https://<restapiid>.execute-api.eu-central-1.amazonaws.com/Prod/hello/
HTTP Status 502
{"message": "Internal server error"}

sam logs --stack-name sam-sync-test
2021/10/30/[$LATEST]..... 2021-10-30T18:57:04.917000 Error loading method handleRequest on class helloworld.App: java.lang.NoClassDefFoundError
java.lang.NoClassDefFoundError: com/amazonaws/services/lambda/runtime/events/APIGatewayProxyRequestEvent
        at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
        at java.base/java.lang.Class.privateGetDeclaredMethods(Unknown Source)
        at java.base/java.lang.Class.privateGetPublicMethods(Unknown Source)
        at java.base/java.lang.Class.getMethods(Unknown Source)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.services.lambda.runtime.events.APIGatewayProxyRequestEvent. Current classpath: file:/var/task/

The same happens if --code and/or --watch is used with sam sync.

sam deploy fixes the deployment.

Using --no-dependency-layer with sam sync does works without this problem.

Expected result:

https://<restapiid>.execute-api.eu-central-1.amazonaws.com/Prod/hello/
{ "message":  ...

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS 11.6
2. sam --version: SAM CLI, version 1.34.1
3. AWS region: eu-central-1

Add --debug flag to command you are running

2021-10-30 21:06:56,665 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-10-30 21:06:56,678 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-10-30 21:06:56,678 | Sending Telemetry: {'metrics': [{'templateWarning': {'requestId': '05aafe25-9440-4138-a42f-208f4d1d97e9', 'installationId': '0c823d35-064b-490f-8ba7-cd31cfc7ea99', 'sessionId': 'f8897bd8-5cb2-4c7e-b40e-28c3ea820c93', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.12', 'samcliVersion': '1.34.1', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'warningName': 'CodeDeployWarning', 'warningCount': 0}}]}
2021-10-30 21:06:57,496 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
2021-10-30 21:06:57,496 | Sending Telemetry: {'metrics': [{'templateWarning': {'requestId': '99d88d2b-143f-471f-88dd-139c350d271c', 'installationId': '0c823d35-064b-490f-8ba7-cd31cfc7ea99', 'sessionId': 'f8897bd8-5cb2-4c7e-b40e-28c3ea820c93', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.12', 'samcliVersion': '1.34.1', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'warningName': 'CodeDeployConditionWarning', 'warningCount': 0}}]}
2021-10-30 21:06:58,343 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
2021-10-30 21:06:58,344 | Using config file: samconfig.toml, config environment: default
2021-10-30 21:06:58,344 | Expand command line arguments to:
2021-10-30 21:06:58,344 | --template_file=<redacted>/hello-world-lambda/template.yaml --stack_name=sam-sync-test --dependency_layer --capabilities=('CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND')

                Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-<redacted>
                A different default S3 bucket can be set in samconfig.toml
                Or by specifying --s3-bucket explicitly.

                Default capabilities applied: ('CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND')
To override with customized capabilities, use --capabitilies flag or set it in samconfig.toml
2021-10-30 21:06:58,959 | Using build directory as .aws-sam/auto-dependency-layer
2021-10-30 21:06:58,960 | Using build directory as .aws-sam/auto-dependency-layer

This feature is currently in beta. Visit the docs page to learn more about the AWS Beta terms https://aws.amazon.com/service-terms/.

The SAM CLI will use the AWS Lambda, Amazon API Gateway, and AWS StepFunctions APIs to upload your code without
performing a CloudFormation deployment. This will cause drift in your CloudFormation stack.
**The sync command should only be used against a development stack**.

Confirm that you are synchronizing a development stack and want to turn on beta features.

Enter Y to proceed with the command, or enter N to cancel:
 [y/N]: y
2021-10-30 21:07:07,757 | No Parameters detected in the template
2021-10-30 21:07:07,782 | 2 stacks found in the template
2021-10-30 21:07:07,783 | No Parameters detected in the template
2021-10-30 21:07:07,802 | 2 resources found in the stack
2021-10-30 21:07:07,803 | No Parameters detected in the template
2021-10-30 21:07:07,824 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2021-10-30 21:07:07,824 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to <redacted>/hello-world-lambda/template.yaml
2021-10-30 21:07:07,824 | No Parameters detected in the template
2021-10-30 21:07:07,847 | Executing the build using build context.
2021-10-30 21:07:07,847 | Instantiating build definitions
2021-10-30 21:07:07,850 | Same function build definition found, adding function (Previous: BuildDefinition(java11, <redacted>/hello-world-lambda/HelloWorldFunction, Zip, f06ee645957ed5bf1a505335119e54b125445f33025fe5de00515ae6369f2061, 41574e68-4f67-4b43-80f1-5a6da7de5c45, {}, {}, x86_64, []), Current: BuildDefinition(java11, <redacted>/hello-world-lambda/HelloWorldFunction, Zip, , 9aa28613-2f53-4da5-83a8-6e3c68b31773, {}, {}, x86_64, []), Function: Function(name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='java11', memory=512, timeout=20, handler='helloworld.App::handleRequest', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='<redacted>/hello-world-lambda/HelloWorldFunction', environment={'Variables': {'PARAM1': 'VALUE'}}, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata=None, inlinecode=None, codesign_config_arn=None, architectures=['x86_64'], stack_path=''))
2021-10-30 21:07:07,851 | Async execution started
2021-10-30 21:07:07,852 | Invoking function functools.partial(<bound method CachedOrIncrementalBuildStrategyWrapper.build_single_function_definition of <samcli.lib.build.build_strategy.CachedOrIncrementalBuildStrategyWrapper object at 0x10ab0d5e0>>, <samcli.lib.build.build_graph.FunctionBuildDefinition object at 0x10aeabcd0>)
2021-10-30 21:07:07,852 | Running incremental build for runtime java11 for build definition 41574e68-4f67-4b43-80f1-5a6da7de5c45
2021-10-30 21:07:07,852 | Waiting for async results
2021-10-30 21:07:07,853 | Valid cache found, copying previously built resources from function build definition of 41574e68-4f67-4b43-80f1-5a6da7de5c45
2021-10-30 21:07:07,853 | Copying artifacts from <redacted>/hello-world-lambda/.aws-sam/cache/41574e68-4f67-4b43-80f1-5a6da7de5c45 to <redacted>/hello-world-lambda/.aws-sam/auto-dependency-layer/HelloWorldFunction
2021-10-30 21:07:07,857 | Async execution completed
2021-10-30 21:07:07,858 | Auto creating dependency layer for each function resource into a nested stack
2021-10-30 21:07:07,859 | No Parameters detected in the template
2021-10-30 21:07:07,880 | 2 resources found in the stack sam-sync-test
2021-10-30 21:07:07,880 | No Parameters detected in the template
2021-10-30 21:07:07,900 | Found Serverless function with name='HelloWorldFunction' and CodeUri='.aws-sam/auto-dependency-layer/HelloWorldFunction'
2021-10-30 21:07:07,901 | --base-dir is not presented, adjusting uri .aws-sam/auto-dependency-layer/HelloWorldFunction relative to <redacted>/hello-world-lambda/template.yaml

Build Succeeded

Built Artifacts  : .aws-sam/auto-dependency-layer
Built Template   : .aws-sam/auto-dependency-layer/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t .aws-sam/auto-dependency-layer/template.yaml
[*] Deploy: sam deploy --guided --template-file .aws-sam/auto-dependency-layer/template.yaml

2021-10-30 21:07:07,910 | Executing the packaging using package context.

Successfully packaged artifacts and wrote output template to file /var/folders/w_/2x46sybd5xg3h41kl_2vrby5j8bmhv/T/tmpenu0u5f1.
Execute the following command to deploy the packaged template
sam deploy --template-file /var/folders/w_/2x46sybd5xg3h41kl_2vrby5j8bmhv/T/tmpenu0u5f1 --stack-name <YOUR STACK NAME>

2021-10-30 21:07:18,419 | Executing the deployment using deploy context.

        Deploying with following values
        ===============================
        Stack name                   : sam-sync-test
        Region                       : eu-central-1
        Disable rollback             : False
        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-1co7ug0c2axut
        Capabilities                 : ["CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND"]
        Parameter overrides          : {}
        Signing Profiles             : null

Initiating deployment
=====================
2021-10-30 21:07:18,441 | No Parameters detected in the template
2021-10-30 21:07:18,462 | Unable to resolve property Layers: [OrderedDict([('Fn::GetAtt', ['AwsSamAutoDependencyLayerNestedStack', 'Outputs.HelloWorldFunction19d43fc4DepLayer'])])]. Leaving as is.
2021-10-30 21:07:18,463 | 3 stacks found in the template
2021-10-30 21:07:18,463 | No Parameters detected in the template
2021-10-30 21:07:18,484 | Unable to resolve property Layers: [OrderedDict([('Fn::GetAtt', ['AwsSamAutoDependencyLayerNestedStack', 'Outputs.HelloWorldFunction19d43fc4DepLayer'])])]. Leaving as is.
2021-10-30 21:07:18,484 | 3 resources found in the stack
2021-10-30 21:07:18,484 | No Parameters detected in the template
2021-10-30 21:07:18,507 | Unable to resolve property Layers: [OrderedDict([('Fn::GetAtt', ['AwsSamAutoDependencyLayerNestedStack', 'Outputs.HelloWorldFunction19d43fc4DepLayer'])])]. Leaving as is.

2021-10-30 21:07:20 - Waiting for stack create/update to complete

CloudFormation events from stack operations
---------------------------------------------------------------------------------------------------------
ResourceStatus             ResourceType               LogicalResourceId          ResourceStatusReason
---------------------------------------------------------------------------------------------------------
UPDATE_IN_PROGRESS         AWS::CloudFormation::Sta   sam-sync-test              Transformation succeeded
                           ck
CREATE_IN_PROGRESS         AWS::CloudFormation::Sta   AwsSamAutoDependencyLaye   -
                           ck                         rNestedStack
CREATE_IN_PROGRESS         AWS::CloudFormation::Sta   AwsSamAutoDependencyLaye   Resource creation
                           ck                         rNestedStack               Initiated
CREATE_COMPLETE            AWS::CloudFormation::Sta   AwsSamAutoDependencyLaye   -
                           ck                         rNestedStack
UPDATE_IN_PROGRESS         AWS::Lambda::Function      HelloWorldFunction         -
UPDATE_COMPLETE            AWS::Lambda::Function      HelloWorldFunction         -
UPDATE_COMPLETE_CLEANUP_   AWS::CloudFormation::Sta   sam-sync-test              -
IN_PROGRESS                ck
UPDATE_COMPLETE            AWS::CloudFormation::Sta   sam-sync-test              -
                           ck
---------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
------------------------------------------------------------------------------------------------------------
Outputs
------------------------------------------------------------------------------------------------------------
Key                 HelloWorldFunctionIamRole
Description         Implicit IAM Role created for Hello World function
Value               arn:aws:iam::<redacted>:role/sam-sync-test-HelloWorldFunctionRole-LRZ58MNCBWKD

Key                 HelloWorldApi
Description         API Gateway endpoint URL for Prod stage for Hello World function
Value               https://yobqv2u1oi.execute-api.eu-central-1.amazonaws.com/Prod/hello/

Key                 HelloWorldFunction
Description         Hello World Lambda Function ARN
Value               arn:aws:lambda:eu-central-1:<redacted>:function:sam-sync-test-HelloWorldFunction-
udYOyRtEIcOH
------------------------------------------------------------------------------------------------------------
2021-10-30 21:08:06,260 |
Stack update succeeded. Sync infra completed.

2021-10-30 21:08:06,261 | {'StackId': 'arn:aws:cloudformation:eu-central-1:<redacted>:stack/sam-sync-test/80cbe210-39b2-11ec-8be5-02886adc29f6', 'ResponseMetadata': {'RequestId': '4255f8b7-3c2c-4039-9827-d56553f8646a', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '4255f8b7-3c2c-4039-9827-d56553f8646a', 'content-type': 'text/xml', 'content-length': '386', 'date': 'Sat, 30 Oct 2021 19:07:19 GMT'}, 'RetryAttempts': 0}}
2021-10-30 21:08:06,267 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '43922c3f-960c-427f-9a22-997d85517726', 'installationId': '0c823d35-064b-490f-8ba7-cd31cfc7ea99', 'sessionId': 'f8897bd8-5cb2-4c7e-b40e-28c3ea820c93', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.12', 'samcliVersion': '1.34.1', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam sync', 'duration': 69602, 'exitReason': 'success', 'exitCode': 0}}]}
2021-10-30 21:08:07,255 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

[CoshUS]

Thanks for the feedback. Does the issue still persist if you directly deploy to a new stack with sam watch skipping sam deploy?

[roamingthings]

@CoshUS Did you really mean sam watch oder doing sam sync --stack-name my-stack --watch directly after creating the project?

When I do a sam sync --stack-name mystack --watch I get the same error.

[CoshUS]

My bad, I meant sam sync --watch.
If I got it correctly, the same error still shows up even when running --watch on a stack that does not exist?

[roamingthings]

Correct.

I created a new application as in the original post (sam init...) and after that sam sync --stack-name mystack --watch. Same error.
The stack-name I've chosen never existed in the account.

[CoshUS]

Reproduced and root cause found. We will address this issue in a future release. Thanks for reporting it!

[Hovard]

I had same problem, i think its critical. Non of new deployment works. Do you know any workaround except manual code upload? When we expect new release?