stepfunctions-tasks: Cannot use fromJsonPathAt when passing arg array values to ScriptBootstrapActionConfigProperty

[anthonysgro]

Describe the bug

When writing step functions, often I want to pass a value from the payload to an API that expects an array value like this:

bootstrapActions: [
    {
        name: 'MyBootstrapAction',
        scriptBootstrapAction: {
            path: 'my_path',
            "Args.$": '$.bootstrapArgs' // $.bootstrapArgs is an array value
        },
     },
],

or

bootstrapActions: [
    {
        name: 'MyBootstrapAction',
        scriptBootstrapAction: {
            path: 'my_path',
            args: sfn.TaskInput.fromJsonPathAt('$.bootstrapArgs').value
        },
     },
],

This works for EmrAddStepsProps args property, but not for the args property within ScriptBootstrapActionConfigProperty. This means I cannot pass custom arguments based on my payload to bootstrap arguments when creating clusters, and this behavior seems contradictory to other tasks I have used in the library.

Expected Behavior

I expected that I could dynamically pass a payload value to the args property within ScriptBootstrapActionConfigProperty inside of EmrCreateClusterProps. This should correspond to the task in the Step Functions console having an "Args.$" key instead of "Args". However, I cannot do this via CDK. I can do this via manual cloudformation and editing in the console, so this seems to be a CDK limitation.

Current Behavior

Instead, it fails:

TypeError: x.map is not a function

    at /Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/core/lib/runtime.js:1:1549

    at BootstrapActionConfigToJson (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions-tasks/lib/emr/private/cluster-utils.js:1:10250)

    at Array.map (<anonymous>)

    at /Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/core/lib/runtime.js:1:1549

    at EmrCreateCluster._renderTask (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.js:1:3931)

    at EmrCreateCluster.toStateJson (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/states/task-base.js:1:2233)

    at StateGraph.toGraphJson (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/state-graph.js:1:1984)

    at ChainDefinitionBody.bind (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/state-machine.js:1:11644)

    at new StateMachine (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/state-machine.js:1:6100)

    at PersistentClusterStateMachine.createPersistentClusterStateMachine (/Volumes/workplace/MyWorkplace/src/MyService/dist/lib/constructs/stepfunctions/persistentClusterStateMachine.js:47:16)

Reproduction Steps

In CDK, I have done this before with the following design pattern successfully:

const myTask = new sfn.Pass(this, 'FormatArgs', {
    parameters: {
        'emrStepArgs.$': `States.Array(
            'spark-submit',
            '--deploy-mode', 'cluster',
            '--class','${this.customValue}',
            $.payload.customValueFromPayload
            )',
        clusterId: sfn.TaskInput.fromJsonPathAt('$.clusterId').value,
        job: sfn.TaskInput.fromJsonPathAt('$.job').value
    }
}).next(
    new tasks.EmrAddStep(this, 'RunStep', {
        name: 'TaskName',
        clusterId: sfn.TaskInput.fromJsonPathAt('$.clusterId').value,
        jar: 'command-runner.jar',
        args: sfn.TaskInput.fromJsonPathAt('$.emrStepArgs').value
    })

If you notice, in EmrAddStepProps, args is expected to be an array, but allows this as is receiving this array from the input of the payload.

However, this does not work for the EmrCreateCluster api, specifically for the args property in ScriptBootstrapActionConfigProperty. Here is an example that fails to build:

const myTask = new sfn.Pass(this, 'FormatBootstrapArgs, {
    {
        parameters: {
            'payload.$': '$.payload',
            'bootstrapArgs.$': `States.Array(
                'string_literal',
                '${this.customValue}',
                $.payload.customValueFromPayload
            )',
        }
}).next(
    new tasks.EmrCreateCluster(this, 'CreateEmrCluster', {
        name: 'ClusterName',
        releaseLabel: '7.0.0',
        instances: ...
        clusterRole: ...
        serviceRole: ...
        applications: ....
        configuration: ...
        bootstrapActions: [
            {
                name: 'MyBootstrapAction',
                scriptBootstrapAction: {
                    path: 'my_path',
                    args: sfn.TaskInput.fromJsonPathAt('$.bootstrapArgs').value
                },
             },
         ],
         ....
});

I can actually bypass this error when doing:

bootstrapActions: sfn.TaskInput.fromObject([
            {
                name: 'MyBootstrapAction',
                scriptBootstrapAction: {
                    path: 'my_path',
                    'Args.$': '$.bootstrapArgs'
                },
             },
         ]),

But then the Args parameter is dropped on synthesis and it doesn't get published to the AWS account

Possible Solution

It seems that sfn.TaskInput.fromJsonPathAt creates a resolvable token that is a string under the hood. However, CDK is still trying to map over this token which is incorrect behavior. I think that a possible fix could be to ignore this mapping behavior on resolvable tokens.

No response

Additional Information/Context

No response

CDK CLI Version

2.126.0

Framework Version

No response

Node.js Version

v18.19.1

OS

Sonoma 14.3.1

Language

TypeScript

Language Version

No response

Other information

No response

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[anthonysgro]

I see this token is being mapped here:

aws-cdk/packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/emr/private/cluster-utils.ts

Lines 313 to 326 in a7de7fe

	/**
	* Render the BootstrapActionProperty as JSON
	*
	* @param property
	*/
	export function BootstrapActionConfigToJson(property: EmrCreateCluster.BootstrapActionConfigProperty) {
	return {
	Name: cdk.stringToCloudFormation(property.name),
	ScriptBootstrapAction: {
	Path: cdk.stringToCloudFormation(property.scriptBootstrapAction.path),
	Args: cdk.listMapper(cdk.stringToCloudFormation)(property.scriptBootstrapAction.args),
	},
	};
	}

[eeelnico]

+1

I am experiencing the same issue

[paulhcsun]

@anthonysgro I wasn't able to reproduce this error with the following stack, it deployed successfully for me:

    const myTask = new sfn.Pass(this, 'FormatBootstrapArgs', {
        parameters: {
            'payload.$': '$.payload',
            'bootstrapArgs.$': `States.Array(
                'string_literal',
                'com.example.spark.jobs.ProcessData',
                $.payload.customValueFromPayload
            )`,
        }
    }
        ).next(
            new tasks.EmrCreateCluster(this, 'CreateEmrCluster', {
                name: 'ClusterName',
                releaseLabel: 'emr-7.0.0',
                instances: {
                    masterInstanceType: 'm5.xlarge',
                    instanceCount: 2,
                },
                bootstrapActions:[
                    {
                        name: 'MyBootstrapAction',
                        scriptBootstrapAction: {
                            path: 'my_path',
                            args: sfn.TaskInput.fromJsonPathAt('$.bootstrapArgs').value
                        },
                    },
                ]
            }))

Could you provide another example of the stack that failed for you with the missing parameters filled in or let me know if there is some major difference with mine that mightve caused me to not run into the same error?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[eeelnico]

I will also try to reproduce in a small/minimal example.

[pahud]

@anthonysgro @eeelnico

Are you able to provide a full mini example that we can deploy and reproduce this issue?