(pipelines): Need a way to get Stack Outputs as file artifact if they are > 1000 chars

[fab-mindflow]

What is the problem?

It looks like CodeBuildStep.envFromCfnOutputs property is limited to 1-1000 characters. When injecting variables, it can quickly reaches this limit and the CodePipeline build fails with:

8:35:20 AM | CREATE_FAILED        | AWS::CodePipeline::Pipeline                  | ##############
ActionConfiguration Map value must satisfy constraint: [Member must have length less than or equal to 1000, Member must have length greater than or equal to 1] (Service: AWSCodePipeline; Status Code: 400; Error Code: ValidationException; Request ID: ############; Proxy: null)

Reproduction Steps

Include many environment variables from CloudFormation output in CodeBuildStep.envFromCfnOutputs.

What did you expect to happen?

CodeBuildStep successful.

What actually happened?

CodeBuildStep failure with too vague error message.

CDK CLI Version

1.127.0

Framework Version

No response

Node.js Version

14.x

OS

macOS 11.6

Language

Typescript

Language Version

No response

Other information

No response

[fab-mindflow]

Any known workaround?

This seems to be more or less related to the same limit as: #12747

[brennadotdev]

I've run in to this as well, but only in one specific AWS account. I use the same CDK code in other accounts and it works as expected. I'm not sure what the difference is yet since they all have the same number of variables in the build step and reference SSM params of the same names.

[fab-mindflow]

For now, I've used a workaround to dump half of my environment variables in a file within a preliminary CodeBuildStep and read it in the next one.

Unfortunately, the #17074 has broken this workaround from CDK 1.129.0. See my #17074 (comment)

[fab-mindflow]

@skinny85 @rix0rrr Any news on this?
We are totally stuck to upgrade our CDK pipeline to latest release (and take benefit of awesome cdk watch :-)

[rix0rrr]

This is unfortunate. I'm not sure it's fixable, this is a limitation imposed by CodePipeline.

I guess an alternative would be to make the outputs available as a JSON file which you can consume in CodeBuild as you desire.

[fab-mindflow]

@rix0rrr OK got it, thanks. That's what I did, dumping in multiple steps to an environment file.
However, it cannot work on latest CDK because of this:

Unfortunately, the #17074 has broken this workaround from CDK 1.129.0. See my #17074 (comment)

Edit: actually, this has already been fixed. See update in #17074 (comment)

[piotrekwitkowski]

I use the following workaround:

CodeBuildStep('StepId',
    commands=[
        'aws sts get-caller-identity', # you don't need this but it prints out the role for debugging purposes
        'export MY_VAR=$(aws cloudformation describe-stacks --query "Stacks[?StackName==\'stack-name\'][].Outputs[?OutputKey==\'my-output-key\'].OutputValue" --output text)',
        'echo $MY_VAR'
    ],
    role_policy_statements=[
        PolicyStatement(
            actions=['cloudFormation:DescribeStacks'],
            resources=['*'],
        ),
    ]
)

[github-actions]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[sthuber90]

Is there any specific reason this hasn't been resolved yet? What's the reason to limit the evaluation to 1000 characters?

For my pipeline I have a workaround but that's all it is, a workaround