`sam build` broken after upgrading SAM CLI when using ECR (backward compatibility issue)

[romanek-adam]

Description:

I have a SAM application with an AWS ECR repository configured as the storage for the app snapshots/releases.

So far I was using just sam build and sam deploy to deploy my SAM application. It used to work with SAM CLI 1.24.1.

Unfortunately after I upgraded to SAM CLI 1.27.2 it stopped working. Now sam build says:

The resource AWS::Serverless::Function 'HelloFunction' has specified ECR registry image for ImageUri. It will not be built and SAM CLI does not support invoking it locally.

Upgrading SAM CLI to the current latest version 1.30.0 doesn't help either.

This is a breaking change. Not sure if it was announced anywhere. Also it seems the problem was introduced as part of #2934.

Honestly I don't know what I should do now and how I should fix it. For now I can stick to the SAM CLI version that works for me but this is only a temporary solution.

Steps to reproduce:

Create a SAM app with a template including:

Resources:
  HelloFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      ImageUri: <account-number>.dkr.ecr.<region>.amazonaws.com/hello-app
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./app
      DockerTag: python3.8-v1

Use sam build and then guided deploy with SAM CLI 1.24.1. Then upgrade to SAM CLI 1.27.2 - sam build will stop working.

Observed result:

sam build doesn't build the image anymore and sam deploy doesn't push the image to ECR. As a result sam deploy fails.

Expected result:

sam build should build the image, as it used to. sam deploy should push the image to ECR and update the app without issues.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Ubuntu 20.04
2. sam --version: 1.24.1, 1.27.2, 1.30.0
3. AWS region: Ireland (eu-west-1)

[romanek-adam]

Tagging @alexisfacques

[alexisfacques]

Heya

Given the fact, the ImageUri parameter is irrelevant in the build/deploy process (the final ECR Repo / Image URI to which you publish the image must be specified via deploy config parameters), it was agreed off-side in #2934 to skip the image build when the ImageUri parameter is a valid ECR image.
This allows you to deploy pre-packaged PackageType: Image functions with SAM, the same way you may deploy PackageType: Zip images pointing to a valid S3 location.

Changing the ImageUri parameter to something else than the ECR repo image name will fix the problem (e.g. hello-app:latest). This will be the tag used by SAM to build Docker image in your local environment, prior to deploying it

[romanek-adam]

Hi,

Thanks for providing some background. Next time I'd suggest making such new behavior optional so that existing users could adapt to it over time. Or you could just bump the major version of the tool to indicate a breaking change and document it properly in the release notes.

Anyway I was just looking at the description of ImageUri parameter:

The URI of the Amazon Elastic Container Registry (Amazon ECR) repository for the Lambda function's container image. This property only applies if the PackageType property is set to Image, otherwise it is ignored.

If setting ImageUri to something other than an ECR repo URL is supported then it should be mentioned here.

Moreover, while I understand that the change you introduced allows someone to use a pre-packaged and pre-published Docker image, the docs should still make it clear on how to use sam build/sam deploy together with ECR to achieve the simpler use-case where all you want to do is to use SAM CLI to do the build/push steps for you.

I checked other parts of the documentation related to building images with SAM apps and I must admit I haven't found a clear description of how this should be done. For example see Building applications, Example 2: Container image - it shows you an exemplary SAM template and a Dockerfile, but it doesn't say how it should (could) be built and published.

[awood45]

Can you explain more about why sam deploy is failing? What message do you get on deployment?

[romanek-adam]

The error message is:

Resource handler returned message: "Source image <account-nr>.dkr.ecr.eu-west-1.amazonaws.com/hello-world is not valid. Provide a valid source image.  (Service: Lambda, Status Code: 400, Request ID: <request-id>, Extended Request ID: null)" (RequestToken: <request-token>, HandlerErrorCode: InvalidRequest)

Below is a full log:

$ sam deploy

	Deploying with following values
	===============================
	Stack name                   : hello-world
	Region                       : eu-west-1
	Confirm changeset            : True
	Deployment image repository  : 
                                       {
                                           "HelloFunction": "<account-number>.dkr.ecr.eu-west-1.amazonaws.com/hello-world"
                                       }
	Deployment s3 bucket         : <bucket-name>
	Capabilities                 : ["CAPABILITY_IAM"]
	Parameter overrides          : {}
	Signing Profiles             : {}

Initiating deployment
=====================
File with same data already exists at hello-world/01dab9ed7445d678ea5d297a4b4e801c.template, skipping upload

Waiting for changeset to be created..

CloudFormation stack changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                  LogicalResourceId                                          ResourceType                                               Replacement                                              
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
* Modify                                                   HelloFunctionHelloPermission                               AWS::Lambda::Permission                                    Conditional                                              
* Modify                                                   HelloFunction                                              AWS::Lambda::Function                                      False                                                    
* Modify                                                   ServerlessHttpApiApiGatewayDefaultStage                    AWS::ApiGatewayV2::Stage                                   Conditional                                              
* Modify                                                   ServerlessHttpApi                                          AWS::ApiGatewayV2::Api                                     Conditional                                              
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:eu-west-1:<account-nr>:changeSet/samcli-deploy1630913936/1e0f72bc-453d-44f4-9b6f-5f34946eb8fb


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2021-09-06 09:39:09 - Waiting for stack create/update to complete

CloudFormation events from changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                             ResourceType                                               LogicalResourceId                                          ResourceStatusReason                                     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UPDATE_IN_PROGRESS                                         AWS::Lambda::Function                                      HelloFunction                                        -                                                        
UPDATE_ROLLBACK_IN_PROGRESS                                AWS::CloudFormation::Stack                                 hello-stack                                          The following resource(s) failed to update:              
                                                                                                                                                                                 [HelloFunction].                                   
UPDATE_FAILED                                              AWS::Lambda::Function                                      HelloFunction                                        Resource handler returned message: "Source image         
                                                                                                                                                                                 <account-nr>.dkr.ecr.eu-west-1.amazonaws.com/hello-world is not valid. Provide a valid source image.  
                                                                                                                                                                                 (Service: Lambda, Status Code: 400, Request ID:          
                                                                                                                                                                                 b31c6317-5934-4cd2-9c0d-461a0883344a, Extended Request   
                                                                                                                                                                                 ID: null)" (RequestToken:                                
                                                                                                                                                                                 dbae00b4-952f-0818-5a44-6cb126006e97, HandlerErrorCode:  
                                                                                                                                                                                 InvalidRequest)                                          
UPDATE_IN_PROGRESS                                         AWS::Lambda::Function                                      HelloFunction                                        -                                                       
UPDATE_COMPLETE                                            AWS::Lambda::Function                                      HelloFunction                                        -                                                        
UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS               AWS::CloudFormation::Stack                                 hello-stack                                          -                                                        
UPDATE_ROLLBACK_COMPLETE                                   AWS::CloudFormation::Stack                                 hello-stack                                          -                                                        
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Error: Failed to create/update the stack: hello-stack, Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "UPDATE_ROLLBACK_COMPLETE" at least once

Tested with sam deploy using SAM CLI 1.27.2, of course using the original SAM template, with a full ECR repository URI in ImageUri.

[romanek-adam]

Are you able to reproduce the issue?

[aahung]

@romanek-adam Sorry about the confusion.

When you provide the Metadata below and set the package type to Image, the ImageUri is ignored during the sam build process in previous version of sam-cli. The built artifact (.aws-sam/build/template.yaml) will have ImageUri with values like "ImageUri: helloworldfunction:nodejs14.x-v1" and sam package or sam deploy will upload the built imagine and replace it with uploaded image uri. So the previously provided ImageUri has no effect

Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./app
      DockerTag: python3.8-v1

Related doc:

Note: If the PackageType property is set to Image, then either ImageUri is required, or you must build your application with necessary Metadata entries in the AWS SAM template file. https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-imageuri

After the change in #2934, we skipped building functions with a "valid" ImageUri, and this change broke your workflow, sorry about the inconvenience.

You can remove ImageUri since the value is not used. Then do sam build then sam deploy (or sam package) to test whether it fixes your issue or not.

We will work on some documentation change to avoid such confusion in the future.

[romanek-adam]

I can confirm that with ImageUri removed sam build now builds a Docker image and sam deploy pushes it to ECR and deploys it without issues.

Thank you all for your support!