Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unsupported intrinsic functions #2533

Open
hoffa opened this issue Oct 11, 2022 · 24 comments
Open

Unsupported intrinsic functions #2533

hoffa opened this issue Oct 11, 2022 · 24 comments
Labels
area/intrinsics Ref, If, Sub, GetAtt, ... meta

Comments

@hoffa
Copy link
Contributor

hoffa commented Oct 11, 2022

The problem

Some properties have limited support for intrinsic functions (e.g. Ref, Fn::GetAtt, Fn::If, etc.).

Using unsupported intrinsic functions in such properties can cause issues with deployment.

Why it happens

AWS SAM is a AWS CloudFormation macro; it receives a SAM template as input (as-is, along with the intrinsic functions), and returns a CloudFormation template which is then deployed by CloudFormation.

This means that SAM is unable to resolve some intrinsic functions. For example, an !If condition with a Ref to a stack parameter is resolvable (SAM has access to the stack parameters), but a Ref to a stack resource is not (as the transform happens before deployment). Furthermore, SAM supports limited intrinsic function resolution for only some properties.

It's not an issue for properties that are passed as-is to properties of the underlying CloudFormation resources, but it becomes an issue when SAM must know the value for its transform logic.

Workarounds

Add the AWS::LanguageExtensions transform

The AWS::LanguageExtensions transform will resolve intrinsic functions if the value is known when Transforms are run.

Replace:

Transform: AWS::Serverless-2016-10-31

With:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

The AWS SAM CLI currently doesn't process AWS::LanguageExtensions locally, so it won't work where local transforms are needed.

Note

There is a known issue where AWS::Serverless-2016-10-31 and AWS::LanguageExtensions can conflict; see aws-cloudformation/cfn-language-discussion#109.

Use a pass-through property, if available

Pass-through properties are passed directly to the underlying CloudFormation resources, hence intrinsic functions work. Check the "AWS CloudFormation compatibility" notice under properties to see whether the property is passed as-is to an underlying CloudFormation resource.

For example for the Schedule event type, use the State property instead of Enabled.

Use raw CloudFormation

If nothing else works, you can always switch to using the underlying CloudFormation resources directly. Since they are not processed by SAM, CloudFormation will be able to resolve the intrinsic functions.

You can get the transformed CloudFormation template of a stack <my-stack> using:

aws cloudformation get-template --query TemplateBody --change-set-name "$(aws cloudformation describe-stacks --query 'Stacks[0].ChangeSetId' --output text --stack-name <my-stack>)"

Which you can then use to replace the affected resources.

See also #3007 (comment) for other ways of transforming a SAM template into a CloudFormation template.

@hoffa hoffa added area/intrinsics Ref, If, Sub, GetAtt, ... stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. meta and removed stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. labels Oct 11, 2022
@hoffa hoffa pinned this issue Oct 17, 2022
This was referenced Oct 17, 2022
@hoffa
Copy link
Contributor Author

hoffa commented Feb 25, 2023

@tariromukute If I'm understanding correctly the problem you're encountering is !Ref to local paths in AWS::Serverless::LayerVersion not working. I was able to reproduce it, and confirmed it works with CodeUri in AWS::Lambda::Function. I've created aws/aws-sam-cli#4767 for the issue. There's not much else that can be done on the SAM transform side as the intrinsic resolution to the local path must happen before the SAM transform receives it. While not ideal, as workaround in the meantime you could use sed or similar to replace the text.

@tariromukute
Copy link

@tariromukute If I'm understanding correctly the problem you're encountering is !Ref to local paths in AWS::Serverless::LayerVersion not working. I was able to reproduce it, and confirmed it works with CodeUri in AWS::Lambda::Function. I've created aws/aws-sam-cli#4767 for the issue. There's not much else that can be done on the SAM transform side as the intrinsic resolution to the local path must happen before the SAM transform receives it. While not ideal, as workaround in the meantime you could use sed or similar to replace the text.

Awesome. Thanks @hoffa for reproducing the issue and for creating an issue for it. I will try the sed route on the CI pipeline.

@garretwilson
Copy link

garretwilson commented Jun 2, 2023

This breaks CodeUri when deploying AWS lambda from a prebuilt JAR file using !Sub (see #271). Let's say I'm deploying some lambda function I built with Maven, and I don't want to hard-code the JAR version in my SAM template.

Resources:
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: !Sub "target/foo-bar-functions-${Ver}.jar"
      Handler: com.example.FooBar::foo

SAM gives me an error:

'CodeUri' requires Bucket and Key properties to be specified.

If I try the workaround noted here of adding AWS::LanguageExtensions, that SAM will at least upload my JAR to the S3 bucket. But then when it fails with:

'CodeUri' is not a valid S3 Uri of the form 's3://bucket/key' with optional versionId query parameter.

The problem is that SAM is not updating the CodeUri value before it uploads the template to S3, as it would if the value were a literal string. Shouldn't SAM know the dereferenced value at the time of uploading the template? Hasn't the template already been processed? There must be a bug where SAM forgets it needs to update the CodeUri value before uploading the template if !Ref was used. I filed bug aws/aws-sam-cli#5249.

@smaly-amazon
Copy link

If the AWS::LanguageExtensions transform is the solution, is that transform available in all regions where the serverless transform is available?

@davidchatz
Copy link

I've been using the language extension to workaround this issue with using !If in AutoPublishAlias, but this does not work with the new sam list command, which outputs: Error: [InvalidResourceException('SnsTopicConsumer', "'AutoPublishAlias' must be a string or a Ref to a template parameter")] ('SnsTopicConsumer', "'AutoPublishAlias' must be a string or a Ref to a template parameter")

@GavinZZ
Copy link
Contributor

GavinZZ commented Oct 10, 2023

Hi @davidchatz, feel free to create an issue in aws-sam-cli repository on this.

@ozcsdog
Copy link

ozcsdog commented Oct 31, 2023

I am coming from #1616.
Is there any way to set string to whole Step Functions TimeoutSeconds? Just like TimeoutSecondsPath for Task.
Now the whole Step Functions TimeoutSeconds is only for int, and can't get string from path.

@anashakt
Copy link

anashakt commented Jun 2, 2024

I just now ran into same situation while trying to pass "TimeoutSeconds" as integer in definitionSubstitutions

@carlosleon-wl
Copy link

Hi, I'm having a very simple problem:

      Events:
        Status:
          Type: HttpApi
          Properties:
            ApiId: !Ref HttpApi
            Method: get
            Path: !Sub "/${Version}/status"

I cannot use !Sub to substitute Version parameter.
This is a long-standing problem according to what I've seen. Is there anything I can do apart from hardcoding the Version?

@moelasmar
Copy link
Contributor

@carlosleon-wl .. you can use the workaround mentioned in this issue.

Add the AWS::LanguageExtensions transform

The AWS::LanguageExtensions transform will resolve intrinsic functions if the value is known when Transforms are run.

Replace:

Transform: AWS::Serverless-2016-10-31

With:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/intrinsics Ref, If, Sub, GetAtt, ... meta
Projects
None yet
Development

No branches or pull requests