[aws-codepipeline] Parameter overrides cannot use Fn::GetArtifactAtt

[piotrkubisa]

I have been trying to create a deployment pipeline for Lambda function using CodePipeline. During development I noticed there is weird issue which does not allow me to use Fn::GetArtifactAtt in the parameterOverrides (1), because default JSON serializer forces to use Fn::Join which does not accept Fn::GetArtifactAtt.

(1): https://github.com/awslabs/aws-cdk/blob/4af7c0d97b17820a41ef9d611ed6768f62d64a6c/packages/%40aws-cdk/aws-cloudformation/lib/pipeline-actions.ts#L189

I see it was noticed in #566, but for 0.22.0 (build 644ebf5) version it does not work for me.

import iam = require('@aws-cdk/aws-iam');
import codepipeline = require('@aws-cdk/aws-codepipeline');
import cloudformation = require('@aws-cdk/aws-cloudformation');
import codepipelineapi = require('@aws-cdk/aws-codepipeline-api');

const addDeployPrepareAction = (
    id: string,
    buildArtifact:  codepipelineapi.Artifact,
    props: {
        stage: codepipeline.Stage,
        stackName: string,
        region: string,
        changeSetName: string,
        role: iam.Role
    }
) => {
    const { stackName, stage, region, changeSetName, role } = props

    new cloudformation.PipelineCreateReplaceChangeSetAction(stage, id, {
        stackName,
        stage,
        region,
        role,
        changeSetName,
        runOrder: 1,
        adminPermissions: false,
        capabilities: cloudformation.CloudFormationCapabilities.NamedIAM,
        parameterOverrides: {
            'LambdaCodeZip': buildArtifact.objectKey,
            'LambdaCodeBucket': buildArtifact.bucketName,
        },
    })
}

Expected (one of the possible solution):

ParameterOverrides: !Sub |
  {
    "LambdaCodeZip" : { "Fn::GetArtifactAtt" : [ "BuildOutput", "ObjectKey" ] },
    "LambdaCodeBucket": { "Fn::GetArtifactAtt" : [ "BuildOutput", "BucketName" ] }
  }

Actual:

ParameterOverrides:
  Fn::Join:
    - ""
    - - '{"LambdaCodeZip":"'
      - Fn::GetArtifactAtt:
          - BuildOutput
          - ObjectKey
      - '","LambdaCodeBucket":"'
      - Fn::GetArtifactAtt:
          - BuildOutput
          - BucketName
      - '"}'

Error:

aws cloudformation validate-template --template-body file://pipeline.json

An error occurred (ValidationError) when calling the ValidateTemplate operation: Template Error: Encountered unsupported function: Fn::GetArtifactAtt Supported functions are: [Fn::Base64, Fn::GetAtt, Fn::GetAZs, Fn::ImportValue, Fn::Join, Fn::Split, Fn::FindInMap, Fn::Select, Ref, Fn::Equals, Fn::If, Fn::Not, Condition, Fn::And, Fn::Or, Fn::Contains, Fn::EachMemberEquals, Fn::EachMemberIn, Fn::ValueOf, Fn::ValueOfAll, Fn::RefAll, Fn::Sub, Fn::Cidr]

[rix0rrr]

The problem here seems to be that {Fn::GetArtifactAtt} is modeled as an actual CloudFormation intrinsic, whereas it's supposed to be just opaque JSON in the ParameterOverrides property?

Expected (one of the possible solution):
ParameterOverrides: !Sub |

What does the !Sub still add here? Does it do anything?

[rix0rrr]

cc @skinny85

[piotrkubisa]

The problem here seems to be that {Fn::GetArtifactAtt} is modeled as an actual CloudFormation intrinsic, whereas it's supposed to be just opaque JSON in the ParameterOverrides property?

Yes, I think so. I guess I will encounter the same problem with Fn::GetParam in ParameterOverrides. Please read more about these two dedicated functions: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-parameter-override-functions.html

Expected (one of the possible solution):
ParameterOverrides: !Sub |

What does the !Sub still add here? Does it do anything?

• It takes care of creating a valid JSON object.
• I stripped few more parameters in that example which does not relate directly to the issue, here is an example how you might use Fn::Sub with some Parameter or using any other function (Fn::GetAtt, Fn::Ref...) inside the ParameterOverrides:

ParameterOverrides: !Sub |
  {
    "LambdaCodeZip" : { "Fn::GetArtifactAtt" : [ "BuildOutput", "ObjectKey" ] },
    "LambdaCodeBucket": { "Fn::GetArtifactAtt" : [ "BuildOutput", "BucketName" ] },
    "AssetsBucketName": "${AssetsBucket}",
    "AssetsBucketDomainName": "${AssetsBucket.DomainName}",
    "GitHubRepositoryName": "${GitHubRepositoryName}"
  }

For an example I provided in first post !Sub is not required to work, but you have to keep the | pipe character to inform you are starting a multi-line block.

YAML:

ParameterOverrides: |
  {
    "LambdaCodeZip" : { "Fn::GetArtifactAtt" : [ "BuildOutput", "ObjectKey" ] },
    "LambdaCodeBucket": { "Fn::GetArtifactAtt" : [ "BuildOutput", "BucketName" ] }
  }

JSON:

{
  "ParameterOverrides": "{\n  \"LambdaCodeZip\" : { \"Fn::GetArtifactAtt\" : [ \"BuildOutput\", \"ObjectKey\" ] },\n  \"LambdaCodeBucket\": { \"Fn::GetArtifactAtt\" : [ \"BuildOutput\", \"BucketName\" ] }\n}\n",
}

[piotrkubisa]

Edit: It does not work.

---

As a workaround we may mock it in the following way:

parameterOverrides: {
  'LambdaCodeZip': `{ "Fn::GetArtifactAtt" : [ "${buildArtifact.name}", "ObjectKey" ] }`,
  'LambdaCodeBucket': `{ "Fn::GetArtifactAtt" : [ "${buildArtifact.name}", "BucketName" ] }`,
  'GitHubRepositoryName': cdk.Fn.sub('${GitHubRepositoryName}'),
},

Output YAML:

ParameterOverrides:
  Fn::Join:
    - ""
    - - '{"LambdaCodeZip":"{ \"Fn::GetArtifactAtt\" : [
        \"BuildOutput\", \"ObjectKey\" ]
        }","LambdaCodeBucket":"{ \"Fn::GetArtifactAtt\" : [
        \"BuildOutput\", \"BucketName\" ]
        }","GitHubRepositoryName":"'
      - Fn::Sub: ${GitHubRepositoryName}
      - '"}'

Output JSON:

{
  "ParameterOverrides": {
    "Fn::Join": [
      "",
      [
        "{\"LambdaCodeZip\":\"{ \\\"Fn::GetArtifactAtt\\\" : [ \\\"BuildOutput\\\", \\\"ObjectKey\\\" ] }\",\"LambdaCodeBucket\":\"{ \\\"Fn::GetArtifactAtt\\\" : [ \\\"BuildOutput\\\", \\\"BucketName\\\" ] }\",\"GitHubRepositoryName\":\"",
        {
          "Fn::Sub": "${GitHubRepositoryName}"
        },
        "\"}"
      ]
    ]
  },
}

[skinny85]

@piotrkubisa thanks for reporting this one, I'm looking into it.

Quick question: did your workaround from this comment work? I does deploy through CFN, however the Pipeline with those Parameter Overrides set fails updating its Stack for me, because of escaping (CFN thinks { "Fn::GetArtifactAtt" : [ "BuildOutput", "BucketName" ] is literally the Bucket name).

[piotrkubisa]

Sadly, no it does not work 😢 You are right it will provide { "Fn::GetArtifactAtt" : [ "BuildOutput", "BucketName" ] } as a string value as a parameter.

[skinny85]

I figured out what the issue is. I submitted a PR with the fix here.

[piotrkubisa]

Thanks!

[skinny85]

NP, thanks for reporting the issue!

[rebolyte]

This seems to be still be happening when not used in the Cfn parameterOverrides property. Given this:

new PipelineProject(scope, `Project`, {
	// ...
	environmentVariables: {
		WEB_API_LOC: {
			type: BuildEnvironmentVariableType.PLAINTEXT,
			value: JSON.stringify(props.webApiLambdaCode.assign(webApiOutput.s3Location))
		}
	}
	// ...
})

You still get this output:

ProjectD3596EDB:
  Type: AWS::CodeBuild::Project
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Environment:
      ComputeType: BUILD_GENERAL1_SMALL
      EnvironmentVariables:
        - Name: WEB_API_LOC
          Type: PLAINTEXT
          Value:
            Fn::Join:
              - ""
              - - '{"ApiApiFuncLambdaSourceBucketNameParameterB6671E0E":"'
                - Fn::GetArtifactAtt:
                    - WebApiOut
                    - BucketName
                - '","ApiApiFuncLambdaSourceObjectKeyParameter1B2544EC":"'
                - Fn::GetArtifactAtt:
                    - WebApiOut
                    - ObjectKey
                - '"}'

[skinny85]

That could be a miss from our side, but I'm 99% sure this won't work anyway (the only place Fn::GetArtifactAtt can be used is inside a CloudFormation CodePipeline action, they are meaningless in a CodeBuild project, and will never get substituted by anything).

I believe what you're looking for is codepipeline.Artifacts.

[rebolyte]

@skinny85 Thanks for the response. My end goal is to work around this issue about passing more than a few parameters via parameterOverrides, and my thought was to write out the artifact attributes to a templateConfiguration file. It makes sense that Fn::GetArtifactAtt wouldn't be substituted if it came from there though. Can I reference artifact attributes another way?

[skinny85]

A couple of things about that:

1. The names you have there, LambdaSourceBucketNameParameter and LambdaSourceObjectKeyParameter, are just the defaults. You can create your own:

lambda.Code.fromCfnParameters({
  bucketNameParam: new CfnParameter(this, 'A'),
  objectKeyParam: new CfnParameter(this, 'B'),
});

2. Are you using the default Artifact names? Because you can always name them explicitly, thus saving a lot of characters:

const sourceOutput = new codepipeline.Artifact('S');

Hope this helps!

Adam

[rebolyte]

I think specifying my own CfnParameters will be good enough for this use case. Thanks! 🎉

[rebolyte]

For anyone finding this in the future, I finally got a response from AWS Support regarding the issue:

I've queried the CodePipeline team and searched though the current development workflows and couldn't find any current activity related to increasing the limit for parameters passed to a CloudFormation stack or any alternative method for this action, so we have issued a feature request based on your request for our development team.

I'm not able to provide an estimated time for this feature to be available, but you can follow the release on new features through the CloudFormation (1) and CodePipeline (2) official pages to check when the new feature will be available. Additionally when I receive any update from the feature request ticket I will send a new reply on your case, even if it was closed.

(1) CloudFormation Release History
URL: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html
(2) CodePipeline History
URL: https://docs.aws.amazon.com/codepipeline/latest/userguide/history.html

So for now, it looks like the CfnParameter workaround is the best option.

[matheushent]

I don't think this issue is solved. I'm facing it just right now.

This is how my code looks like:

from aws_cdk import (
    core,
    aws_s3 as s3,
    aws_ssm as ssm,
    aws_codepipeline as codepipeline,
    aws_codepipeline_actions as actions,
)

from typing import Dict


class PipelineTest(core.Stack):
    def __init__(self, app: core.App, id: str, props: Dict, **kwargs) -> None:
        super().__init__(app, id, **kwargs)

        source_output = codepipeline.Artifact(artifact_name='source')
        build_output = codepipeline.Artifact(artifact_name='build')

        core.CfnOutput(
            self, "Outpu1t",
            value=build_output.s3_location.bucket_name
        )

        core.CfnOutput(
            self, "Output2",
            value=build_output.s3_location.object_key
        )

This is the output when I run cdk deploy '*':

failed: Error [ValidationError]: Template Error: Encountered unsupported function: Fn::GetArtifactAtt Supported functions are: [Fn::Base64, Fn::GetAtt, Fn::GetAZs, Fn::ImportValue, Fn::Join, Fn::Split, Fn::FindInMap, Fn::Select, Ref, Fn::Equals, Fn::If, Fn::Not, Condition, Fn::And, Fn::Or, Fn::Contains, Fn::EachMemberEquals, Fn::EachMemberIn, Fn::ValueOf, Fn::ValueOfAll, Fn::RefAll, Fn::Sub, Fn::Cidr]
    at Request.extractError (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'ValidationError',
  time: 2021-04-09T11:13:12.367Z,
  requestId: 'b26864df-a3de-4c2b-82b6-a2ac9bcf164b',
  statusCode: 400,
  retryable: false,
  retryDelay: 577.931740145933
}
Template Error: Encountered unsupported function: Fn::GetArtifactAtt Supported functions are: [Fn::Base64, Fn::GetAtt, Fn::GetAZs, Fn::ImportValue, Fn::Join, Fn::Split, Fn::FindInMap, Fn::Select, Ref, Fn::Equals, Fn::If, Fn::Not, Condition, Fn::And, Fn::Or, Fn::Contains, Fn::EachMemberEquals, Fn::EachMemberIn, Fn::ValueOf, Fn::ValueOfAll, Fn::RefAll, Fn::Sub, Fn::Cidr]

This is how the outputs from template looks like:

"Outputs": {
    "Output": {
      "Value": {
        "Fn::GetArtifactAtt": [
          "build",
          "BucketName"
        ]
      }
    },
    "Output2": {
      "Value": {
        "Fn::GetArtifactAtt": [
          "build",
          "ObjectKey"
        ]
      }
    }
  }

CDK version: 1.87.1 (build 9eeaa93)
OS system: macOS Catalina 10.15.5

[skinny85]

@matheushent As the error message states, Fn::GetArtifactAtt is not a CloudFormation function. It's confusing, I know, because it looks like one. But it is not.

It's a special marker token that can only be used in the ParameterOverrides property of the CloudFormation CodePipeline actions.

[matheushent]

@matheushent As the error message states, Fn::GetArtifactAtt is not a CloudFormation function. It's confusing, I know, because it looks like one. But it is not.

It's a special marker token that can only be used in the ParameterOverrides property of the CloudFormation CodePipeline actions.

Ok but, will simply do not change it?

[skinny85]

@matheushent As the error message states, Fn::GetArtifactAtt is not a CloudFormation function. It's confusing, I know, because it looks like one. But it is not.
It's a special marker token that can only be used in the ParameterOverrides property of the CloudFormation CodePipeline actions.

Ok but, will simply do not change it?

Sorry, I don't understand your question. Can you rephrase?