feat(glue): add Job L2 construct #11128
Conversation
|
|
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
| The only required parameter is `script` which defines the relative path to your Python or Scala file. A `CustomResource` pushes your script to S3 and configures the `Job` with necessary permissions and the final location of the script. | ||
|
|
||
| ```ts | ||
| new glue.Job(stack, 'MyJob', { | ||
| script: './lib/scripts/etlJob.py' | ||
| }); | ||
| ``` | ||
|
|
||
| By default, an S3 bucket will be created to store the provided script but you can manually pass the `bucket` and `s3Prefix`: | ||
|
|
There was a problem hiding this comment.
I feel like it is more natural to use the s3-assets module here. This would work similar to how Lambda deals with assets.
There was a problem hiding this comment.
@adamelmore I agree with @tdikland that we should just use the regular asset mechanism.
Regarding the API, I want to provide something similar to what we have in Lambda and CloudWatch Synthetics, which also accept code from the user.
new glue.Job(this, 'Job', {
code: glue.Code.fromAsset(...) // glue.Code.fromInline(...)
});Is it even possible to use a directory or does it have to be a single script file?
There was a problem hiding this comment.
This all sounds good to me. And pretty certain only a single script is allowed.
There was a problem hiding this comment.
In that case the api shouldn't Code.fromAsset but rather Code.fromScript or Code.fromFile
|
@iliapolo have a chance to look at this? |
|
@adamelmore sorry its taking a while. I'll try to have a look by EOW. thanks for this! |
iliapolo
added
the
response-requested
|
This PR has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled. |
github-actions
bot
added
the
closing-soon
|
Dont close |
github-actions
bot
removed
closing-soon
|
@adamelmore What do you think about closing this in favor of #12506 ? Would love your input their as well. |
iliapolo
added
the
response-requested
Absolutely. I never did make time to work on an implementation, here. I do use my own Job construct, and have some opinions around the API, so I'll try to review that PR as I have time! |
|
Closing in favor of #12506 |
@iliapolo looking to start a conversation here around a
JobL2 construct inaws-glue. I'm using my ownJob,CrawlerandTriggerconstructs, and I think they could be useful here. Starting withJob, there's the need to push a local script to S3; sort of a lite version of Lambda bundling.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license