Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New Data Source: aws_glue_script #4481

Merged
merged 2 commits into from
May 22, 2018
Merged

New Data Source: aws_glue_script #4481

merged 2 commits into from
May 22, 2018

Conversation

bflad
Copy link
Contributor

@bflad bflad commented May 8, 2018

Closes #3879

Changes proposed in this pull request:

  • New data source that converts Glue DAG to a code script via CreateScript API call

Output from acceptance testing:

$ make testacc TEST=./aws TESTARGS='-run=TestAccDataSourceAWSGlueScript'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccDataSourceAWSGlueScript -timeout 120m
=== RUN   TestAccDataSourceAWSGlueScript_Language_Python
--- PASS: TestAccDataSourceAWSGlueScript_Language_Python (11.68s)
=== RUN   TestAccDataSourceAWSGlueScript_Language_Scala
--- PASS: TestAccDataSourceAWSGlueScript_Language_Scala (12.70s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	24.427s

@bflad bflad added new-data-source Introduces a new data source. service/glue Issues and PRs that pertain to the glue service. labels May 8, 2018
@bflad bflad requested a review from a team May 8, 2018 18:00
@ghost ghost added the size/L Managed by automation to categorize the size of a PR. label May 8, 2018
@bflad bflad added this to the v1.19.0 milestone May 10, 2018
@bflad bflad modified the milestones: v1.19.0, v1.20.0 May 17, 2018
tombuildsstuff
tombuildsstuff approved these changes May 22, 2018
View reviewed changes
Copy link
Contributor

@tombuildsstuff tombuildsstuff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bunch of mostly minor comments - but this otherwise LGTM 👍

aws/data_source_aws_glue_script.go Outdated

"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/glue"
"github.com/hashicorp/terraform/helper/schema"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really minor we can group these

aws/data_source_aws_glue_script.go
"language": {
Type: schema.TypeString,
Optional: true,
Default: glue.LanguagePython,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would this be better as Required with no default set?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It defaults to Python if you do not provide a value 😉

aws/data_source_aws_glue_script.go
"scala_code": {
Type: schema.TypeString,
Computed: true,
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could argue these could be one field, in case additional languages are added in the future?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why the Glue API distinguishes between the return values. It only returns one of them at a time currently 😕 Maybe it will return multiple in the future. We tend to opt for sticking with API rather than making our own choices for UX due to past maintenance issues. I'm tempted to leave this as-is hoping that AWS has a good reason for this API design decision.

aws/data_source_aws_glue_script.go Outdated

input := &glue.CreateScriptInput{
DagEdges: expandGlueCodeGenEdges(d.Get("dag_edge").([]interface{})),
DagNodes: expandGlueCodeGenNodes(d.Get("dag_node").([]interface{})),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor splitting these out into separate lines makes this easier when things crash

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update 👍

website/docs/d/glue_script.html.markdown Outdated
## Attributes Reference

* `python_script` - The Python script generated from the DAG. The `language` argument must be set to `PYTHON`.
* `scala_code` - The Scala code generated from the DAG. The `language` argument must be set to `SCALA`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd argue this would be better as:

* `scala_code` - The Scala code generated from the DAG when the `language` argument is set to `SCALA`.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update 👍

website/docs/d/glue_script.html.markdown Outdated

* `name` - (Required) The name of the argument or property.
* `param` - (Optional) Boolean if the value is used as a parameter. Defaults to `false`.
* `value` - (Required) The value of the argument or property.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor should we be sorting this Required -> Optional, then by Alphabetical?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call -- will update 👍

website/docs/d/glue_script.html.markdown Outdated
* `args` - (Required) Nested configuration an argument or property of a node. Defined below.
* `id` - (Required) A node identifier that is unique within the node's graph.
* `line_number` - (Optional) The line number of the node.
* `node_type` - (Required) The type of node this is.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor should we be sorting this Required -> Optional, then by Alphabetical?

website/docs/d/glue_script.html.markdown
]

dag_node = [
# ...
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'd be good to give a working example here imo

@ghost ghost added the size/L Managed by automation to categorize the size of a PR. label May 22, 2018
@bflad bflad merged commit 198b6db into master May 22, 2018
@bflad bflad deleted the f-aws_glue_script branch May 22, 2018 16:39
bflad added a commit that referenced this pull request May 22, 2018
@bflad
Update CHANGELOG for #4481
8943cd2
@bflad
Copy link
Contributor Author

bflad commented May 23, 2018

This has been released in version 1.20.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@ghost
Copy link

ghost commented Apr 5, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@tombuildsstuff tombuildsstuff tombuildsstuff approved these changes

Assignees
No one assigned
Labels
new-data-source Introduces a new data source. service/glue Issues and PRs that pertain to the glue service. size/L Managed by automation to categorize the size of a PR.
Projects
None yet
Milestone
v1.20.0
Development

Successfully merging this pull request may close these issues.

Feature Request: Data Source for generating AWS Glue Scripts
2 participants
@bflad @tombuildsstuff