SAM Creates stage named "Stage" by default?

[sdole]

I have a template like this:

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Resources:
  Api:
    Properties:
      StageName: dv01
      DefinitionUri: api_spec_SAM.json
    Type: AWS::Serverless::Api

After package and deploy with the cli, it creates an API with two stages named "Stage" and "dv01". In reality, I do not want to create either stages. The StageName property is "required". Not sure why. My intention is to create a separate set of templates to create the stages, deployments, lambda function versions etc.

The reason I need separation is because we have multiple code pipelines that are owned by different people (dev, build teams). I would like these templates to live in separate git repos.

Is there a way to create an API without any stages?

Else, the work around is to simply ignore this one default stage created via SAM. Thanks!

[ghost]

I am also seeing this issue, however I do require a stage set in the AWS::Serverless::Api resource but I am seeing a second stage created as "Stage". I validated in the "processed" cfn template the resource AWS::ApiGateway::Stage is referencing a AWS::ApiGateway::Deployment but the stage name in the deployment is "Stage" not "qic" as set in the stage resource.

SAM Template:

ARSAPIGateway:
    Type: AWS::Serverless::Api
    Properties:
      DefinitionUri: apigateway/swagger.yaml
      StageName: qic
      Variables:
        LambdaFunctionName: !Ref LambdaFunction17

Processed CFN Template:

"APIGatewayDeploymenta727bb8729": {
      "Type": "AWS::ApiGateway::Deployment",
      "Properties": {
        "RestApiId": {
          "Ref": "APIGateway"
        },
        "Description": "RestApi deployment id: a727bb872967fd6e7bd621271f597376226329ee",
        "StageName": "Stage"
      }
    },
"APIGatewayStage": {
      "Type": "AWS::ApiGateway::Stage",
      "Properties": {
        "Variables": {
          "LambdaFunctionName": {
            "Ref": "LambdaFunction17"
          }
        },
        "RestApiId": {
          "Ref": "APIGateway"
        },
        "DeploymentId": {
          "Ref": "APIGatewayDeploymenta727bb8729"
        },
        "StageName": "qic"
      }
    }

Looks like the same/similar issue as here #176

[hagoss01]

I am having a related problem.

I am defining my swagger template inline and in it, the stage name "prod". During the deployment of the stack, I get an error regarding the stage name when another resource (api key and usage plan) tries to reference it. It appears that it doesn't exist (yet the Stage stage exists). It appears that the the processed template includes the "Stage" stage but the creation of the prod stage happens in the wrong order and there is no way to set a "DependsOn"

[mikecbrant]

You see similar (undesired?) behavior when defining AWS::Serverless:Function with API events and not using AWS::Serverless::Api. You end up getting an API gateway resource defined both stage and prod stages, when in reality only a single stage was really desired in that the stacks are being deployed using CodePipeline, where I was hoping the ownership over "Stage". At each stage in this pipeline an instance of the stack defined by the SAM template is deployed. As such, it seems odd to have two API gateway stages (stage/prod) deployed for each on my three CodePipeline-defined stages. Should default behavior be to only create a single single on API gateway rather than two?

[davewoodward]

I'm curious if any of the maintainers of this repo are ever going to weigh-in on this. It makes zero sense to me that both a stage and prod stage are created by default. @sanathkr

[jfuss]

@davewoodward To my knowledge this was and is a bug in SAM. We attempted to remove this behavior but that caused #168 and the fixed was needed to rollback.

The short answer is yes we are aware this happens but right now do not have a good way to fix it that would not cause a regression to all existing stacks that use SAM. There are a couple issue spread throughout this repo on this. We will be working towards cleaning these up into one issue soon. These are the ones that seem related at first glance: #176 #67 #78

[mikecbrant]

@jfuss it would seem that perhaps addition of a property on the resource to determine desired deployment behavior would be feasible, so long as in absence of such a property, the current stage/prod deployments would be the default behavior to maintain backwards compatibility. Of course adding new properties, especially ones that would in essence disappear during the transformation step, might not be desirable either.

[jfuss]

@mikecbrant I personally don't like that approach because you are adding in some property that has no true meaning, other than to hide a bug. This seems like the wrong way to approach a problem like this. The other issue I foresee, is with new customers adopting SAM. Without context of the issue, it is easy to leave this new property out and get into the state we currently are in. Maybe if done correctly, we could do something like this but requires a much deeper discussion and investigation.

Yes, having this stage there is annoying but is it really that harmful? Are there more issues with this other than it is created and is not updatable? Understanding this will help us in trying to prioritize it and give us a bigger reason to dig into it further. When we tried to roll out a fix for this in the past, we caused a regression for customers (#168). Until we have a good solution, the best advice is to just ignore this 'Stage' stage. This is no way ideal by any means but the best I can offer, as of now.

[jarruda]

The version of the transform is in the name (AWS::Serverless-2016-10-31). The purpose of versioning is to allow you to introduce backwards-incompatible changes, right?

While I can see not wanting to increase the version for a seemingly inconsequential bug, it is nevertheless a bug until such time that it's documented as expected behavior.

[jfuss]

@jarruda I have brought this up with the team in the past. I forget how that conversation went but there was some reason we didn't want to do that right now, if I remember correctly.

Regardless, I think we need to invest more time into this issue before taking this approach. We want to make sure we don't have another option to solve this but may be our only option. I will try to drive a conversation with the team, to see if we can get something moving on this. Or at very least communicate to everyone clearly what is going on and where we are at on it.

[upscore]

I'm going through the same issue. It doesn't make sense to me to have this "Stage" stage being created in parallel with the stage I specified. Could you provide us with any update regarding this?

[mikecbrant]

For me the workaround has been to use SAM mainly as a code generation tool. I get the template after transformation has been applied and then perform necessary clean-up like this. Over time, I have found myself just working with the native objects. Thus, it seems SAM is a good tool to minimize barrier to entry (a good mission for a piece of software), but once you start bumping up against some of the feature limitations, it behooves you to get to understand the underlying resources being deployed.

…

On Fri, Feb 9, 2018 at 6:26 PM, upscore ***@***.***> wrote: I'm going through the same issue. It doesn't make sense to me to have this "Stage" stage being created in parallel with the stage I specified. Could you provide us with any update regarding this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#191 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBVex4Wx9IYFxFu4ZY4qocnwws74CChks5tTOIpgaJpZM4PJWdY> .

[xkguq007]

Any progress on this?
For now, I developed using only 'test' in StageName. However I need to test on production mode soon.

[sdole]

We don't use transform for API GW at all because of this issue. For Lambdas we do use Transforms, however, there too, I feel like we really do not need to. All our deployment is via CloudFormation and this transform is simply an unnecessary step in a codebuild for us.

I feel like this SAM thing might be intended for some other purpose in the future. If one is intimately familiar with cloudformation, SAM seems to be just a different (and non-working) way of deploying.

[jfuss]

@xkguq007 The team has been focusing on other areas, mainly filling the gaps we have in API Gateway support. So unfortunately, we have not been able to make process on this issue just yet. We have had hallway/ one off converstations about this issue and still feeling out how best to handle this without causing a back breaking change for all customers.

@sdole "SAM seems to be just a different (and non-working) way of deploying."

• That is pretty harsh. Are we prefect? No. But saying 'non-working'...
  If we are missing the mark that far, cut us an issue. Let's have an open conversation about what is missing, what areas need further improvement, etc. The team is very open to this kinds of things and we love honest feedback. :)

[sdole]

@jfuss, sorry about making you feel that way. My comment was in context of this specific issue. It does not work if it creates an unnecessary stage. I am a big fan of AWS, I am a standard partner of AWS and have many happy customers because of AWS. At one customer I have created a global web app based on aws lambda and api gateway alone. Doing that work led me to try and fail at SAM and open this GitHub issue. We have AWS codepipelines in which there's an AWS codebuild step just to transform from SAM to regular Cloudformation. I could have had the devs just as effectively use a non SAM template. That was my second point. Finally, in the context of services such as Serverless Application Repository, I think SAM may play a role in defining reusable shareable templates etc. That was my last point in the previous post.

I definitely do not intend to disparage the entirety of SAM. I apologize.

[sanathkr]

Let me explain why this bug is tricky to fix. When you deploy an API with SAM, it will create a AWS::ApiGateway::Stage resource to setup the actual stage, and an AWS::ApiGateway::Deployment resource to deploy the APIs to the stage.

Here is an example Deployment resource.

"ServerlessRestApiDeployment127e3fb911": {
      "Type": "AWS::ApiGateway::Deployment", 
      "Properties": {
        "RestApiId": {
          "Ref": "ServerlessRestApi"
        }, 
        "Description": "RestApi deployment id: 127e3fb91142ab1ddc5f5446adb094442581a90d", 

        "StageName": "Stage"  <------ NOTE THIS LINE
      }

Notice the "StageName" property on this resource. API Gateway creates a Deployment, and creates a new stage called "Stage". This is in addition to the regular ("Prod") stage you wanted SAM to create.

So the fix is easy. Remove this one line from Deployment resource, right? Not so easy after all!

ApiGateway::Deployment is an immutable resource. Removing this line will have one of two outcomes next time you deploy the stack:

1. If your stack has a change to API configuration, it will roll forward and remove the extra Stage. This is because when API configuration changes, SAM detects it using a hash of the Swagger JSON, and appends the hash to ApiGateway::Deployment resource, essentially creating a new resource and deleting the old one.

2. If your stack does not change the API configuration, stack deployment will fail because CloudFormation tries remove the StageName property from existing Deployment resource, which is immutable. So ApiGateway rejects this request, and CloudFormation subsequently fails the deployment.

Unfortunately SAM is implemented as a stateless system. It does not know the current state of any of your resources. It is simply a stateless converter from SAM to CloudFormation. So SAM cannot do cool things like "remove StageName if and only if API configuration is changing".

Possible Solutions

1. Version Bump: Create a new SAM Transform 2.0 version that is backwards incompatible with 1.0. The only difference would be removing the StageName property. Customers have to upgrade to 2.0 only when changing the API Configuration.

2. Internal Property: Provide a property like ___REMOVE_STAGE_NAME, which when set to True will remove this StageName for you. But then your template is stuck with this property forever. This is really ugly and not customer obsessed.

We don't like any of the above solutions. Both of them make you, template author, to do more work. This is the reason we haven't fixed this bug yet.

We are working with folks internally to try and find a different solution to this problem. If you prefer either of the above solutions, or have other wild ideas, do let us know!

[jfuss]

@sdole Your statements make more sense to me now. Thank you for clarifying it.

Having this 'Stage Stage' being deployed is not ideal at all, no one likes it... Hopefully @sanathkr explanation clarifies a little deeper on what we have been thinking about and where we kind of stand on this (today).

Your statement about "I could have had the devs just as effectively use a non SAM template.". We want the opposite to be true. In many cases, we actually do reduce this effort but there are still some sharp edges to SAM we are trying to smooth out. It makes me sad, but I do understand where you are coming from. I would love to hear if there is actually more things that cause this feeling of "just use vanilla CloudFormation". I would like to keep this issue scoped to the Stage Stage discussion but welcome a deeper discussion on the pains you have encounter (or anyone for that matter) with using SAM.

[sdole]

@jfuss. For one example, when using SAM transform, it appears that the only way to get the transformed output is to local hard disk. That imposes a limit of 51,200 bytes on the template size. Granted that that is a lot of bytes, but, in our enterprise scenario, there are many units to consider. API GW, Lambda Resource Permissions, Headers, Versions, Aliases. Sometimes developers like to release more than one function in one template. All this adds up fairly quickly. We have 14 different environments (I know that sounds a bit crazy). If transform uploads template to S3, the size limit is 460,800 bytes. Much better.

For two: Before SAM came (or around the same time), I had written a GitHub hosted boto script to do what SAM does in terms of uploading code to S3, getting the new zip's version number and replacing it inside the template so that new code is deployed (without the version number - or a different file name - Cloudformation cannot know that code has changed). We use that extensively. It takes stack name, template parameters, create arguments and all such "never-changing" information from a yaml file we call "project" file. My script uses boto to read the project file, read the template and it's parameters to do packaging followed by creating a change set. The reason to state this is that the developer experience with this script is simple - create a "project" file, create a template and it's parameters and run 1 or 2 commands each time you want to deploy. This is simpler than when I use SAM on the AWS CLI. I need to write a long command string and get it right each time. Even if I used the generated cli skeleton to manage a few things, I still need to write long commands. So, neither SAM nor CF really are easy to use here.

Third, We use exports and imports extensively to deploy APIs and Lambda versions from stage to stage. So, create something in one API Stage, Test it in another stage, Release to prod in another stage (14 such stages). At the time of our testing SAM did not support all export/import. Then there was this very bug.

So, all in all, we don't use SAM much. Also, I doubt that large companies such as my clients will see the value of a developer focused workflow. We have had tremendous success with completely automating all aspects of releasing a serverless app using codepipeline. Our developers do not have any write permission in AWS except for checking in code. We will probably never go back to deploying anything AWS serverless by hand.

I do feel like I will find new value in SAM later, as more people adopt it. SAM may work well somehow with the AWS Cloud9 IDE. It might also find a place in re-usable/shareable constructs such as serverless application repository. But, that is from my narrow point of view.

Thank you for the invitation to discuss. I do find value in discussing with peers and leaders such as yourself. I truly hope not to offend or dishearten anyone.

[kadrach]

I've simply added a delete-stage call to my CD setup removing the default "Stage" stage. @sanathkr am I understanding your comment correct in that this is something I should not be doing?

It seems to have worked without issues so far, but perhaps I'm asking for trouble..

[sanathkr]

@kadrach You are in dangerous territory if you modify resources created by CloudFormation, outside of the stack. But because Deployment is immutable, you should probably be fine.

[kadrach]

Thank you @sanathkr, but the additional stage doesn't actually show in CF. Using SAM to deploy a single stage "master" into CF, I end up with a CF stack that has a single RestAPI resource with a single stage ("master"), but the RestAPI actually has two stages - the default stage "Stage" also exists, without being listed as part of the CloudFormation stack resources. How does that come about?

[sanathkr]

That is a side effect of APIGW::Deployment resource. When creating the Deployment resource, API Gateway creates a stage and associates this deployment to the stage. Since CloudFormation does not actively create the stage, it does not show up in the CFN Stack.

[AvinashDalvi89]

To get around this bug, we added the fix for it behind the OpenApiVersion flag. To opt into this fix, you would need to specify OpenApiVersion flag like here. Regardless of the value (2.0, 3.0.1, etc), the stage stage fix will be applied if the flag is present.

Thanks. I tried it. This solutions works ONLY when one creates API from scratch. If an API was created before, and user adds OpenApiVersion: '2.0' to it, it doesn't remove "Stage" stage. It needs to be added from the beginning. Here is my code:

  MyApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: dev
      Cors:
        AllowMethods: "'GET,POST,DELETE,OPTIONS'"
        AllowHeaders: "'authorization'"
        AllowOrigin: "'*'"
      OpenApiVersion: '2.0'

Thanks for this.

[iongion]

Still facing the same issue - they all end in Stage or Prod, no matter my input parameter

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: SAM Template for annotator

Globals:
  Function:
    Timeout: 600
    MemorySize: 512
    Runtime: nodejs12.x
  Api:
    OpenApiVersion: '3.0.1'
    # See https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessapi
    BinaryMediaTypes:
      - "*~1*"

Parameters:
  StageName:
    Type: String
    Default: dev
    Description: (Required) Enter dev, test, stag, prod. Default is dev.
    AllowedValues:
      - dev
      - test
      - stag
      - prod

Resources:
  AnnotatorApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: !Ref StageName
      Auth:
        DefaultAuthorizer: JWTCustomAuthorizer
        Authorizers:
          JWTCustomAuthorizer:
            FunctionArn: !GetAtt AnnotatorApiAuthorizerFunction.Arn

[specious]

I just opened issue aws/aws-sam-cli#3247 because StageName: !Ref Stage is being treated as an empty stage name (albeit template validation passes because the 'required' StageName property is technically present).

@iongion, what you're describing where whichever value you choose for your StageName parameter seems to take no effect sounds like running into the same issue.

[aprilmintacpineda]

StageName: !Ref Stage

Which will be weird because doing StageName: !Ref Stage actually deploys two stages, the actual value of !Ref Stage (say dev), and another one called "Stage".

[aprilmintacpineda]

  API:
    Type: AWS::Serverless::Api
    Properties:
      Name: myapp-api
      StageName: !Ref Stage

It creates a stage for whatever I provided in !Ref Stage and then it creates another one called Stage.

and !Ref Stage was resolved to dev but as you can see, there's an extra Stage in there, who created that? where that came from? Is still a mystery left unsolve.

[specious]

My apologies, I misinterpreted what was happening because StageName: !Ref Stage creates an AWS::ApiGateway::Stage resource with logical id <gateway-resource-id>Stage rather than <gateway-resource-id><StageName>Stage, which is the case when StageName is hard-coded.

My stages are being deployed correctly, however it may be so because I've been using:

Properties:
  OpenApiVersion: 3.0.3

[simplyi]

hmm...
When I use OpenApiVersion: 3.0.3 in my SAM template, I can deploy to one stage only. If I try to deploy to a new stage, the previous stage gets deleted. Is there a way to tell SAM not to delete a previously created stage?
If I create the second stage manually, then sam refuses to deploy to that new stage because it already exists...

How do you guys configure sam so that it can keep two stages and deploy to a specified stage?

[ptejada]

@simplyi
See https://stackoverflow.com/questions/68826108/how-to-deploy-to-different-environments-with-aws-sam
You basically need to have different stacks. You can't control multiple API Gateways Stages from a sam template.

[TingyuShare]

add OpenApiVersion works for me, it will delete the stage and rebuild:
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-api.html#sam-api-openapiversion

[nikitasius]

perfectly works (no stage created), but i have a strange issue #2377

[cookiejest]

This seems like a fundamental flaw of the SAM approach that this has been left open for years without being properly solved. I will use terraform based approach because of this as it seems more stable.