Add support for glue_version to resource aws_glue_job #10237

NSjogren · 2019-09-25T08:18:22Z

Closes #9524 , python_version support is already resolved.

This will enable you to use a newer versions of AWS Glue, also python 3 and Spark 2.4.

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Relates OR Closes #0000

Release note for CHANGELOG:

- Added support to change glue version in resource aws_glue_job

Output from acceptance testing:

$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSGlueJob_GlueVersion'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -parallel 20 -run=TestAccAWSGlueJob_GlueVersion -timeout 120m
=== RUN   TestAccAWSGlueJob_GlueVersion
=== PAUSE TestAccAWSGlueJob_GlueVersion
=== CONT  TestAccAWSGlueJob_GlueVersion

--- PASS: TestAccAWSGlueJob_GlueVersion (66.90s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	66.922s

...

Fixes hashicorp#9524

Add glue_version argument reference

bflad

Hi @NSjogren 👋 Thanks for submitting this -- its pretty good minus one issue with the schema and nit in the documentation. Please reach out if you have any questions or do not have time to implement the feedback items.

bflad · 2019-10-11T23:13:52Z

website/docs/r/glue_job.html.markdown

@@ -61,7 +61,8 @@ be removed in future releases, please use `max_capacity` instead.
 * `role_arn` – (Required) The ARN of the IAM role associated with this job.
 * `timeout` – (Optional) The job timeout in minutes. The default is 2880 minutes (48 hours).
 * `security_configuration` - (Optional) The name of the Security Configuration to be associated with the job.
-
+* `glue_version` - (Optional) The version of glue to use, for example "1.0".  For information about available versionse see [AWS Glue Release Notes](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html).


Nit: Can you please alphabetize this new argument with the others? Thanks.

bflad · 2019-10-11T23:17:24Z

aws/resource_aws_glue_job.go

@@ -66,6 +66,10 @@ func resourceAwsGlueJob() *schema.Resource {
 				Type:     schema.TypeString,
 				Optional: true,
 			},
+			"glue_version": {
+				Type:     schema.TypeString,
+				Optional: true,


This new argument is currently causing a difference for existing configurations since its set to perform drift detection and Glue assigns a default value, e.g.

--- PASS: TestAccAWSGlueJob_GlueVersion (31.34s) --- FAIL: TestAccAWSGlueJob_Command (31.54s) --- FAIL: TestAccAWSGlueJob_DefaultArguments (45.49s) --- FAIL: TestAccAWSGlueJob_MaxRetries (47.57s) --- FAIL: TestAccAWSGlueJob_Description (54.14s) --- FAIL: TestAccAWSGlueJob_SecurityConfiguration (56.34s) --- FAIL: TestAccAWSGlueJob_ExecutionProperty (59.86s) --- PASS: TestAccAWSGlueJob_Basic (65.38s) --- FAIL: TestAccAWSGlueJob_PythonShell (67.25s) --- FAIL: TestAccAWSGlueJob_MaxCapacity (74.10s) --- FAIL: TestAccAWSGlueJob_Timeout (74.40s) --- FAIL: TestAccAWSGlueJob_AllocatedCapacity (81.45s)

These failures look like:

--- FAIL: TestAccAWSGlueJob_Description (54.14s) testing.go:569: Step 1 error: After applying this step, the plan was not empty: DIFF: UPDATE: aws_glue_job.test ... omitted for clarity ... glue_version: "0.9" => "" ... omitted for clarity ...

While we could set Default: "0.9" on this attribute, I'm actually going to suggest Computed: true since I would imagine that over time, Glue will update the default version number assigned to Jobs that omit the parameter.

Suggested change

Optional: true,

Optional: true,

Computed: true,

This should get the existing testing passing.

bflad · 2019-10-30T00:47:28Z

Hi @NSjogren 👋 Since we haven't heard back about the feedback above, we have addressed it in a followup commit. After those changes:

Output from acceptance testing in AWS Commercial:

--- PASS: TestAccAWSGlueJob_AllocatedCapacity (52.40s)
--- PASS: TestAccAWSGlueJob_Timeout (52.61s)
--- PASS: TestAccAWSGlueJob_ExecutionProperty (57.79s)
--- PASS: TestAccAWSGlueJob_Description (61.93s)
--- PASS: TestAccAWSGlueJob_Command (62.62s)
--- PASS: TestAccAWSGlueJob_Basic (72.67s)
--- PASS: TestAccAWSGlueJob_DefaultArguments (74.90s)
--- PASS: TestAccAWSGlueJob_MaxRetries (81.59s)
--- PASS: TestAccAWSGlueJob_GlueVersion (88.02s)
--- PASS: TestAccAWSGlueJob_SecurityConfiguration (99.31s)
--- PASS: TestAccAWSGlueJob_PythonShell (105.36s)
--- PASS: TestAccAWSGlueJob_MaxCapacity (124.59s)

Output from acceptance testing in AWS GovCloud (US) (new test failure acceptable in this partition):

--- FAIL: TestAccAWSGlueJob_GlueVersion (34.67s)
    testing.go:615: Step 0 error: Check failed: Check 2/2 error: aws_glue_job.test: Attribute 'glue_version' expected "0.9", got ""
--- PASS: TestAccAWSGlueJob_Command (51.17s)
--- PASS: TestAccAWSGlueJob_DefaultArguments (67.58s)
--- PASS: TestAccAWSGlueJob_MaxRetries (68.45s)
--- PASS: TestAccAWSGlueJob_SecurityConfiguration (75.77s)
--- PASS: TestAccAWSGlueJob_PythonShell (81.86s)
--- PASS: TestAccAWSGlueJob_Description (86.03s)
--- PASS: TestAccAWSGlueJob_MaxCapacity (95.51s)
--- PASS: TestAccAWSGlueJob_Timeout (103.41s)
--- PASS: TestAccAWSGlueJob_Basic (120.65s)
--- PASS: TestAccAWSGlueJob_ExecutionProperty (134.29s)
--- PASS: TestAccAWSGlueJob_AllocatedCapacity (143.32s)

Reference: #10237 (review) Output from acceptance testing in AWS Commercial: ``` --- PASS: TestAccAWSGlueJob_AllocatedCapacity (52.40s) --- PASS: TestAccAWSGlueJob_Timeout (52.61s) --- PASS: TestAccAWSGlueJob_ExecutionProperty (57.79s) --- PASS: TestAccAWSGlueJob_Description (61.93s) --- PASS: TestAccAWSGlueJob_Command (62.62s) --- PASS: TestAccAWSGlueJob_Basic (72.67s) --- PASS: TestAccAWSGlueJob_DefaultArguments (74.90s) --- PASS: TestAccAWSGlueJob_MaxRetries (81.59s) --- PASS: TestAccAWSGlueJob_GlueVersion (88.02s) --- PASS: TestAccAWSGlueJob_SecurityConfiguration (99.31s) --- PASS: TestAccAWSGlueJob_PythonShell (105.36s) --- PASS: TestAccAWSGlueJob_MaxCapacity (124.59s) ``` Output from acceptance testing in AWS GovCloud (US) (new test failure acceptable in this partition): ``` --- FAIL: TestAccAWSGlueJob_GlueVersion (34.67s) testing.go:615: Step 0 error: Check failed: Check 2/2 error: aws_glue_job.test: Attribute 'glue_version' expected "0.9", got "" --- PASS: TestAccAWSGlueJob_Command (51.17s) --- PASS: TestAccAWSGlueJob_DefaultArguments (67.58s) --- PASS: TestAccAWSGlueJob_MaxRetries (68.45s) --- PASS: TestAccAWSGlueJob_SecurityConfiguration (75.77s) --- PASS: TestAccAWSGlueJob_PythonShell (81.86s) --- PASS: TestAccAWSGlueJob_Description (86.03s) --- PASS: TestAccAWSGlueJob_MaxCapacity (95.51s) --- PASS: TestAccAWSGlueJob_Timeout (103.41s) --- PASS: TestAccAWSGlueJob_Basic (120.65s) --- PASS: TestAccAWSGlueJob_ExecutionProperty (134.29s) --- PASS: TestAccAWSGlueJob_AllocatedCapacity (143.32s) ```

bflad · 2019-10-31T18:39:58Z

This has been released in version 2.34.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

ghost · 2020-03-29T17:39:41Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

NSjogren added 2 commits September 25, 2019 09:40

Add support for glue version on aws_glue_job

cb8453f

Fixes hashicorp#9524

Update resource aws_glue_job documentation

f875f4c

Add glue_version argument reference

NSjogren requested a review from a team September 25, 2019 08:18

kabeersvohra mentioned this pull request Sep 25, 2019

Add python_version for glue jobs #9409

Closed

ewbankkit mentioned this pull request Oct 2, 2019

Glue_version and python_version not working in terraform #10327

Closed

bflad added the enhancement Requests to existing resources that expand the functionality or scope. label Oct 11, 2019

bflad added this to the v2.33.0 milestone Oct 11, 2019

bflad suggested changes Oct 11, 2019

View reviewed changes

bflad added the waiting-response Maintainers are waiting on response from community or contributor. label Oct 11, 2019

nywilken modified the milestones: v2.33.0, v2.34.0 Oct 17, 2019

ewbankkit mentioned this pull request Oct 20, 2019

Add glue_version property to the aws_glue_job resource #10564

Closed

bflad merged commit f875f4c into hashicorp:master Oct 30, 2019

bflad added a commit that referenced this pull request Oct 30, 2019

Update CHANGELOG for #10237

a90ca3f

ghost locked and limited conversation to collaborators Mar 29, 2020

breathingdust removed the waiting-response Maintainers are waiting on response from community or contributor. label Sep 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for glue_version to resource aws_glue_job #10237

Add support for glue_version to resource aws_glue_job #10237

NSjogren commented Sep 25, 2019 •

edited

Loading

bflad left a comment

bflad Oct 11, 2019

bflad Oct 11, 2019

bflad commented Oct 30, 2019

bflad commented Oct 31, 2019

ghost commented Mar 29, 2020

Add support for glue_version to resource aws_glue_job #10237

Add support for glue_version to resource aws_glue_job #10237

Conversation

NSjogren commented Sep 25, 2019 • edited Loading

Community Note

bflad left a comment

Choose a reason for hiding this comment

bflad Oct 11, 2019

Choose a reason for hiding this comment

bflad Oct 11, 2019

Choose a reason for hiding this comment

bflad commented Oct 30, 2019

bflad commented Oct 31, 2019

ghost commented Mar 29, 2020

NSjogren commented Sep 25, 2019 •

edited

Loading