(pipelines.CodePipeline): unable to use additional_inputs

[KnowSQL]

What is the problem?

No matter what name I supply for the additional inputs directory, it fails at:

[Container] 2021/10/28 23:30:47 Running command [[ ! -d "blah-1234" ]] || { echo 'additionalInputs: "blah-1234" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.'; exit 1; } && ln -s -- "$CODEBUILD_SRC_DIR_blahblah_blahblah_Source" "blah-1234"

followed by:

/codebuild/output/tmp/script.sh: 4: [[: not found

followed by:

additionalInputs: "blah-1234" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.

followed by:

[Container] 2021/10/28 23:30:47 Command did not exit successfully [[ ! -d "blah-1234" ]] || { echo 'additionalInputs: "blah-1234" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.'; exit 1; } && ln -s -- "$CODEBUILD_SRC_DIR_vblahblah_blahblah_Source" "blah-1234" exit status 1

causing the build to fail.

Reproduction Steps

My code simply looks like:

        source0 = CodePipelineSource.connection("blahblah/blahblah", "master",
            connection_arn="arn:aws:codestar-connections:us-east-1:<123456789012>:connection/<some-uuid>"
        )

        pipeline = CodePipeline(self, "Abcdef",
            synth=ShellStep("Synth",
                input=CodePipelineSource.connection("blablah/blahblah0", "master",
                    connection_arn="arn:aws:codestar-connections:us-east-1:<123456789012>:connection/<some-uuid>"
                ),
                additional_inputs={
                    "blah-1234": source0,
                },
                commands=[
                    "npm install -g aws-cdk",
                    "pip install -r requirements.txt",
                    "cdk synth"
                ]
            )
        )

I've put blah etc in place of real github sources are they are all private repositories. The code is pulled in fine, but the build/synth step fails for source0 above.

I've tried various combinations like ../blah-1234 (to create it in a sibling directory) etc. but all fail with similar error messages.

I'm suspicious of:

/codebuild/output/tmp/script.sh: 4: [[: not found

and think it's something in the aws script that's failing somewhere?

What did you expect to happen?

I expected the build to pass

What actually happened?

The build failed

CDK CLI Version

1.129.0

Framework Version

No response

Node.js Version

N/A

OS

Arch Linux

Language

Python

Language Version

No response

Other information

No response

[KnowSQL]

I just came across this commit. My repository name used as the source for additional_inputs does contain a hyphen. Not sure if it's anything to do with that.

Ignore that, see below

[ustulation]

/codebuild/output/tmp/script.sh: 4: [[: not found

Hmm this looks like script.sh is not being run as a bash script but just a shell script. [[ ]] is bash construct i think and pure posix shells will not recognise it. Maybe the shebang is #!/bin/sh or whatever it is it's not pointing to bash? Maybe bash is aliased to some pure posix shell?

[ustulation]

The BuildSpec produced by cdk looks something like this for the above:

"BuildSpec": "{\n  \"version\": \"0.2\",\n  \"phases\": {\n    \"install\": {\n      \"commands\": [\n        \"[[ ! -d \\\"../some-repo-2\\\" ]] || { echo 'additionalInputs: \\\"../some-repo-2\\\" must not exist yet. If you want to merge multiple artifacts, use a \\\"cp\\\" command.'; exit 1; } && ln -s -- \\\"$CODEBUILD_SRC_DIR_some_owner_some_repo_2_Source\\\" \\\"../some-repo-2\\\"\"\n      ]\n    },\n    \"build\": {\n      \"commands\": [\n        \"npm install -g aws-cdk\",\n        \"pip install -r requirements.txt\",\n        \"cdk synth\"\n      ]\n    }\n  },\n  \"artifacts\": {\n    \"base-directory\": \"cdk.out\",\n    \"files\": \"**/*\"\n  }\n}",
"Type": "CODEPIPELINE"

Is there a way to modify this from within cdk? Like try using POSIX compliant [ ! -d '../some-repo-2' ] || ... instead of [[ construct ?
Or is there a way to ask ShellStep in synth to use bash shell in-case it's using something else?

[ustulation]

OK I've made this deployment successful. It is the issue with the shell. We cannot use ShellStep if you have additional_inputs. I needed to use it's subclass CodeBuildStep instead and specify via partial_build_spec to use bash as the shell as opposed to whatever was getting used.

pipeline = CodePipeline(self, "...",
            synth=CodeBuildStep("Synth",
                partial_build_spec=BuildSpec.from_object({
                    "version": "0.2",
                    "env": {
                        "shell": "bash"
                    }
                }),
                input=...,
                additional_inputs={ ... },
                commands= [ ... ],
                ...

That solves it.

That also means that this should be treated as a bug using ShellStep seems to make it impossible to have additional_inputs. So either make ShellStep configure the underlying CodeBuild step to use bash by default or change the factory method in the source code to make it POSIX compliant (ie., drop [[ .. and use [ .. instead).

[NGL321]

Hey @ustulation,

It looks from your comments like you have figured out the problem independently. Can we consider this resolved, or does it still need attention from a team member?

[ustulation]

I'll leave that to yous but I would consider this to be a bug. The fact that ShellStep does not set the codebuild env to use bash but then creates a bash construct thus failing shell execution at runtime means it's unusable for additional_inputs

[abend-arg]

I experimented this error and ended downgrading to version 1.128. I needed to move forward with another change and could not verify why this was happening.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[HansFalkenberg-Visma]

I came here after googling the error message. Although the issue as originally reported has been fixed, the documentation on how to properly use additionalInputs could be better. I'll create a new issue for that.

[dennisvang]

@rix0rrr In a CodeBuildStep running a powershell script, using a codebuild.WindowsBuildImage.WIN_SERVER_CORE_2019_BASE_3_0 env, I'm seeing a similar issue:

...
[Container] 2024/09/09 12:04:08.605419 Running command [ ! -d "../my-content" ] || { echo 'additionalInputs: "../my-content" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.'; exit 1; } && ln -s -- "$CODEBUILD_SRC_DIR_fs_processing_Source" "../my-content"
...
The token '||' is not a valid statement separator in this version.
...

The additional input is a codecommit source, in this case.

Looks like it's trying to run bash commands in powershell. More specifically:

Powershell does have some bash-like aliases, but || only works in powershell 7, and not in powershell 5.1, which is the default for the WIN_SERVER_CORE_2019_BASE_3_0 image.

Tried the OP's workaround, using partialBuildSpec to set shell: 'powershell.exe', but that made no difference.

Any idea how to fix this?

Note:

If we specify the same additionalInputs in the pipeline's synth ShellStep, it works without issue. I assume this runs a linux image.