new "Runtime" parameter is required for deploying Ray Glue Jobs

[jvmunhoz]

Describe the bug

When trying to deploy Ray Glue Jobs through CDK, they will fail because it is now required to provide the "Runtime" parameter, which, as far as I know, is currently not mapped (as of 2.81.0).

Expected Behavior

We should be able to set the "Runtime" parameter on Ray Glue Jobs so they can be properly deployed.

Current Behavior

 ❌  ETLStack failed: Error: The stack named ETLStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Runtime parameter is required for GlueRay command (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request ID: fd617e84-6f10-4068-bfe9-aebd7fce26f0; Proxy: null)
    at FullCloudFormationDeployment.monitorDeployment (/home/ec2-user/agent1/_work/17/s/node_modules/aws-cdk/lib/index.js:397:10236)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.deployStack2 [as deployStack] (/home/ec2-user/agent1/_work/17/s/node_modules/aws-cdk/lib/index.js:400:149977)
    at async /home/ec2-user/agent1/_work/17/s/node_modules/aws-cdk/lib/index.js:400:135508

Reproduction Steps

Trying to deploy any kind of Ray Glue Job will do.

Just cdk init in an empty folder, paste the entire code inside the app.py file created, create a hello_world.py file inside the root folder and cdk deploy

import os
from os import path

import aws_cdk as cdk
from aws_cdk import Stack
from aws_cdk import aws_glue_alpha as glue
from constructs import Construct


class RayStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # Create a Glue job with Ray engine
        glue_job = glue.Job(
            self,
            "ray-job",
            job_name='ray-job',
            worker_count=2,
            worker_type=glue.WorkerType.Z_2_X,
            executable=glue.JobExecutable.python_ray(
                glue_version=glue.GlueVersion.V4_0,
                python_version=glue.PythonVersion.THREE_NINE,
                script=glue.Code.from_asset(path.join(path.dirname(__file__), 'hello_world.py'))
            )
        )

app = cdk.App()

env = cdk.Environment(
    account=os.getenv('CDK_DEFAULT_ACCOUNT'),
    region=os.getenv('CDK_DEFAULT_REGION')
)

RayStack(app, "rayStack", env=env)
app.synth()

Possible Solution

Map the new "Runtime" parameter to the python executable props for Ray Jobs

Additional Information/Context

It was apparently a very recent change, since yesterday everything was fine.

CDK CLI Version

2.81.0

Framework Version

No response

Node.js Version

8.1.0

OS

Amazon Linux 2

Language

Python

Language Version

Python 3.9.14

Other information

No response

[peterwoodworth]

@jvmunhoz reproduction code would be helpful, but I'm not sure how the runtime property is supposed to be defined through CloudFormation or even the API in the first place. After looking into it a bit, this looks to me like it might be a bug with the service. I cannot see any options in the CloudFormation Job resource that would help with this. And, I cannot even see the Runtime prop on the DataType where it is supposed to be

[jvmunhoz]

@jvmunhoz reproduction code would be helpful, but I'm not sure how the runtime property is supposed to be defined through CloudFormation or even the API in the first place. After looking into it a bit, this looks to me like it might be a bug with the service. I cannot see any options in the CloudFormation Job resource that would help with this. And, I cannot even see the Runtime prop on the DataType where it is supposed to be

Just posted a snippet for creating a ray job. Just cdk init in an empty folder, paste the entire code inside the app.py file created, create a hello_world.py file inside the root folder and cdk deploy

[peterwoodworth]

It looks like it was just added to CloudFormation documentation https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-job-jobcommand.html

[peterwoodworth]

You can use escape hatches to fill in the necessary runtime like so until we find a solution for this:

job = Glue.job(self, ...)
cfn_job = job.node.default_child
cfn_job.add_property_override('Command.Runtime', runtime)

I can't find anywhere in the docs what the accepted runtime values are, all I can find is the regex. We might need more time for docs to come in until we're able to properly implement this

[jvmunhoz]

Any updates on the PR?

[shishircc]

By the way, the value of runtime in above workaround should be Ray2.4
cfn_job.add_property_override('Command.Runtime', 'Ray2.4')

From api reference - https://docs.aws.amazon.com/glue/latest/dg/ray-jobs-section.html

[moomindani]

The PR has been merged.

[peterwoodworth]

thanks

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.