New Resource: AWS Glue Catalog Database #2175

drewsonne · 2017-11-05T09:28:21Z

Implemented the AWS Glue Catalog database resource as described here http://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-databases.html to address #1416.

This is my first PR here so I want to keep it simple and only implement a single resource and will implement other resources for AWS glue once I've done the process once

Found the `resource.TestCheckResourceAttr` handles functionality I implemented manually. Have removed this custom code.

bflad

FYI the maintainers usually prefer to treat the vendor changes separately since they tend to be a fast moving target (not sure if this is specifically mentioned anywhere). This is to keep the version synchronized everywhere, prevent merge conflicts, and reduce PR changes.

drewsonne · 2017-11-05T13:23:37Z

@bflad thanks for the heads up!

Have changed this to a WIP PR, and added a new PR for the vendor changes #2177

# Conflicts: # vendor/vendor.json

# Conflicts: # aws/config.go

drewsonne · 2017-11-13T07:12:10Z

make testacc TEST=./aws TESTARGS='-run=TestAccAWSGlueCatalogDatabase_basic'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSGlueCatalogDatabase_basic -timeout 120m
=== RUN   TestAccAWSGlueCatalogDatabase_basic
--- PASS: TestAccAWSGlueCatalogDatabase_basic (21.46s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	21.488s

drewsonne · 2017-11-15T11:46:41Z

Is there anything else which is needed to get this PR in?

bflad

Hey @drewsonne -- thanks for your work here! I left you some initial feedback on your PR. Please take a look and let me know if you have any questions.

bflad · 2017-12-18T16:37:21Z

website/docs/r/glue_catalog_database.html.markdown

+  Provides a Glue Catalog Database.
+---
+
+# aws\_glue\_catalog\_database


Nitpick: (likely copypasta) antislashes in the documentation are no longer needed

bflad · 2017-12-18T16:41:09Z

website/docs/r/glue_catalog_database.html.markdown

+
+## Attributes Reference
+
+The following attributes are exported:


Two nitpicks:

(likely copypasta) this should read The following additional attributes are exported

We should list what the id attribute is here

bflad · 2017-12-18T16:41:41Z

website/docs/r/glue_catalog_database.html.markdown

+
+## Import
+
+Glue Catalog Databasess can be imported using the `name`, e.g.


Typo: Databases

bflad · 2017-12-18T16:45:19Z

aws/resource_aws_glue_catalog_database.go

+	glueconn := meta.(*AWSClient).glueconn
+	name := d.Get("name").(string)
+
+	input := &glue.CreateDatabaseInput{


In this form, it does not support this input parameter:

// The ID of the Data Catalog in which to create the database. If none is supplied, // the AWS account ID is used by default. CatalogId *string `min:"1" type:"string"`

I am wondering if we need to or should support this.

bflad · 2017-12-18T16:52:02Z

aws/resource_aws_glue_catalog_database.go

+		return fmt.Errorf("Error creating Catalogue Database: %s", err)
+	}
+
+	d.SetId(name)


Given the above about potentially supporting separate catalogs in an AWS account, does this resource ID need to support the CatalogId as well? e.g. fmt.Sprintf("%s:%s", meta.(*AWSClient).accountid, name)

bflad · 2017-12-18T16:53:39Z

aws/resource_aws_glue_catalog_database.go

+		},
+
+		Schema: map[string]*schema.Schema{
+			"description": &schema.Schema{


(likely copypasta) since Go 1.7 you do not need to duplicate the &schema.Schema for each of the attributes

bflad · 2017-12-18T16:55:37Z

aws/resource_aws_glue_catalog_database.go

+				Elem:     schema.TypeString,
+				Optional: true,
+			},
+			"create_time": &schema.Schema{


For most or all AWS resources we do not export create/update time attributes like these. Is there a specific use case for dealing with this attribute?

There is no specific use case, my motivation was simply to expose the Glue API for the Database object with as little logic as possible. If the convention within the AWS resources in terraform is not to expose this, I will remove it. Can you just confirm that hiding the CreateTime attribute would be part of the convention within the provider?

bflad · 2017-12-18T17:03:06Z

aws/resource_aws_glue_catalog_database_test.go

+		}
+		if _, err := conn.GetDatabase(input); err != nil {
+			//Verify the error is what we want
+			if ae, ok := err.(awserr.Error); ok && ae.Code() == "EntityNotFoundException" {


Two notes about this:

We have a helper function which can let you remove the awserr import from this file

I would recommend using the SDK constant here

isAWSErr(err, glue.ErrCodeEntityNotFoundException, "")

bflad · 2017-12-18T17:04:58Z

aws/resource_aws_glue_catalog_database_test.go

+	"github.com/hashicorp/terraform/terraform"
+)
+
+func TestAccAWSGlueCatalogDatabase_basic(t *testing.T) {


Can you please add a second resource.TestStep that updates the attributes and ensures the updates propagate properly to AWS/Terraform?

I have attempted to do this, but I struggling to get it working.
When I add a second step with a modified description field, I get:

testing.go:503: Step 0 error: After applying this step, the plan was not empty: DIFF: DESTROY/CREATE: aws_glue_catalog_database.test catalog_id: "440474553311" => "" (forces new resource) description: "A test catalog from terraform" => "A test catalog from terraform" location_uri: "my-location" => "my-location" name: "my_test_catalog_database_6335041086110733121" => "my_test_catalog_database_6335041086110733121" parameters.%: "3" => "3" parameters.param1: "value1" => "value1" parameters.param2: "1" => "1" parameters.param3: "50" => "50" STATE: aws_glue_catalog_database.test: ID = 440474553311:my_test_catalog_database_6335041086110733121 catalog_id = 440474553311 description = A test catalog from terraform location_uri = my-location name = my_test_catalog_database_6335041086110733121 parameters.% = 3 parameters.param1 = value1 parameters.param2 = 1 parameters.param3 = 50

When I take the second step out, it works, so I am not sure why it's saying the plan was not empty at the end if it succeeded with the first step. If there's only one step, does it not test for an empty plan?

I've had a look at https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_ecs_service_test.go to get an idea of a config change test, but I can't see why mine fails and that one works. @bflad , are you able to shed any light on this? I'm keen to get this test working, as it's a problem I ran across in my own provider. That is, how to test diffs successfully.

Try adding Computed: true to the optional catalog_id attribute in the schema. This should tell Terraform that its value is allowed to be set from the provider instead of just the configuration. 😄

tl;dr Computed: true should be set on any attributes that AWS might return to be set in the state that aren't required by writing Terraform configuration.

Great thanks! I was under the impression Computed: true was only for attributes which are computed, and not attributes which could be computed. This is working on my integration tests now.

$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSGlueCatalogDatabase_*' ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./aws -v -run=TestAccAWSGlueCatalogDatabase_* -timeout 120m === RUN TestAccAWSGlueCatalogDatabaseVault_importBasic --- PASS: TestAccAWSGlueCatalogDatabaseVault_importBasic (22.05s) === RUN TestAccAWSGlueCatalogDatabase_basic --- PASS: TestAccAWSGlueCatalogDatabase_basic (33.43s) PASS ok github.com/terraform-providers/terraform-provider-aws/aws 55.511s

drewsonne · 2017-12-18T21:28:08Z

Amazing thank you! I will have a work through those when I next get a chance. I based this off the glacier resource, so that's where the conventions came from.

Rather than passing in a very generic variable `meta` of type `interface{}`, require a specific type (string) for the account id which is being passed in.

# Conflicts: # aws/config.go # aws/provider.go # website/aws.erb

drewsonne · 2018-01-10T07:45:43Z

$ date; make testacc TEST=./aws TESTARGS='-run=TestAccAWSGlueCatalogDatabase'
Wed 10 Jan 2018 07:43:30 GMT
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSGlueCatalogDatabase -timeout 120m
=== RUN   TestAccAWSGlueCatalogDatabase_importBasic
--- PASS: TestAccAWSGlueCatalogDatabase_importBasic (26.66s)
=== RUN   TestAccAWSGlueCatalogDatabase_full
--- PASS: TestAccAWSGlueCatalogDatabase_full (57.37s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	84.055s

drewsonne · 2018-01-10T07:54:04Z

Also... if I wanted to add a list of things I have learned in this PR about how one goes about building a resource (code style, terraform-provider-aws idiomatic process, what type of tests, when to use computed), which .md file in the repo would I put it in?

radeksimko · 2018-01-10T10:00:41Z

if I wanted to add a list of things I have learned in this PR about how one goes about building a resource (code style, terraform-provider-aws idiomatic process, what type of tests, when to use computed), which .md file in the repo would I put it in?

https://github.com/terraform-providers/terraform-provider-aws/blob/master/.github/CONTRIBUTING.md is probably the best place for such things.

cemo · 2018-01-10T17:57:40Z

Hope that other resources will be added too. 👍 Thanks for the great PR.

drewsonne · 2018-01-11T07:24:41Z

Sigh.

Ok, the glue module in vendor has been deleted from the master branch, so I'm going to go through the vendor PR's and find out where it went.

drewsonne · 2018-01-11T07:27:45Z

Looks like it was deleted during an update to terraform 0.11.2. @bflad @radeksimko does this need a separate PR to pull this dependency back into master, or can I put it in this PR?

65b0601#diff-bd290170e2912d3e8694db1a151066e5L619

radeksimko · 2018-01-11T07:30:41Z

Ahh, sorry for this. Looks like we were a bit too excited in #2722 about removal of some "unused" packages...

I can track down the rest of missing packages - feel free to send a PR for glue - I can merge it straight away!

drewsonne · 2018-01-11T07:40:51Z

@radeksimko PR in #2933

radeksimko · 2018-01-11T07:49:02Z

@drewsonne merged - feel free to rebase.

drewsonne · 2018-01-11T08:14:37Z

@radeksimko Integration tests pass.

$ date; make testacc TEST=./aws TESTARGS='-run=TestAccAWSGlueCatalogDatabase'
Thu 11 Jan 2018 08:08:57 GMT
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -run=TestAccAWSGlueCatalogDatabase -timeout 120m
?   	github.com/terraform-providers/terraform-provider-aws	[no test files]
=== RUN   TestAccAWSGlueCatalogDatabase_importBasic
--- PASS: TestAccAWSGlueCatalogDatabase_importBasic (29.81s)
=== RUN   TestAccAWSGlueCatalogDatabase_full
--- PASS: TestAccAWSGlueCatalogDatabase_full (58.42s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	88.262s

So, @radeksimko, we think this is good to go now? :-)

drewsonne · 2018-01-12T07:07:11Z

Wee! Thanks @radeksimko! Will get another one for glue in soonish

bflad · 2018-01-12T14:31:23Z

This has been released in terraform-provider-aws version 1.7.0. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

ghost · 2020-04-08T17:52:14Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

Drew J. Sonne added 11 commits November 4, 2017 14:55

Updated vendor folder to include aws glue sdk

6afe6ea

Added initial test cases

4c79647

Added support resources

29286c5

Implemented glue service and database creation

cb12806

Expanded test cases to include parameters, description, and location_uri

da164c4

Implemented parameters attribute in database creation

b3de1ca

Added documentation

57a1230

Added import test

19ab44c

Fixed import not importing AWS Glue db parameters

bb596b8

Fixed formatting

4bfee75

Addded missing import documentation

dbfe0e7

drewsonne changed the title ~~Implemented AWS Glue Catalog Database~~ New Resource: AWS Glue Catalog Database Nov 5, 2017

Removed custom function

3fc9052

Found the `resource.TestCheckResourceAttr` handles functionality I implemented manually. Have removed this custom code.

bflad reviewed Nov 5, 2017

View reviewed changes

drewsonne mentioned this pull request Nov 5, 2017

Updated vendor to include AWS Glue #2177

Merged

drewsonne changed the title ~~New Resource: AWS Glue Catalog Database~~ [WIP] New Resource: AWS Glue Catalog Database Nov 5, 2017

Ninir added the enhancement Requests to existing resources that expand the functionality or scope. label Nov 5, 2017

Drew J. Sonne and others added 6 commits November 5, 2017 17:46

Merge branch 'master' into f-aws-glue

5a603d6

Merge branch 'master' into f-aws-glue

9a5ed37

# Conflicts: # vendor/vendor.json

Merge branch 'master' into f-aws-glue

5d717d7

# Conflicts: # aws/config.go

Removed merge artifacts

4ebc382

Fixed an error where the incorrect sha was attached to the iam vendor

5fec001

Removed duplicate glue vendor entry

8744d9b

drewsonne changed the title ~~[WIP] New Resource: AWS Glue Catalog Database~~ New Resource: AWS Glue Catalog Database Nov 13, 2017

radeksimko added the size/L Managed by automation to categorize the size of a PR. label Nov 15, 2017

bflad reviewed Dec 18, 2017

View reviewed changes

drewsonne added 7 commits January 10, 2018 07:16

Reduce the interface of createAwsGlueCatalogID

d27cc24

Rather than passing in a very generic variable `meta` of type `interface{}`, require a specific type (string) for the account id which is being passed in.

Make sure Glue Catalog Database only updates params which are provided.

7701582

Use the common syntac type as noted by @radeksimko

878627a

Check if the DB has been deleted and remove it from the state

8cdcebc

Merge branch 'master' into f-aws-glue

419eaa0

Merge branch 'master' into f-aws-glue

7d6e0dd

# Conflicts: # aws/config.go # aws/provider.go # website/aws.erb

Use American spelling

853efbd

Drew J. Sonne added 2 commits January 11, 2018 07:11

Remove un-used Computed attributes for schema vars

f285890

Merge branch 'master' into f-aws-glue

46302bf

Merge branch 'master' into f-aws-glue

15c87e0

radeksimko removed the waiting-response Maintainers are waiting on response from community or contributor. label Jan 11, 2018

radeksimko approved these changes Jan 11, 2018

View reviewed changes

radeksimko merged commit f1386fd into hashicorp:master Jan 11, 2018

drewsonne deleted the f-aws-glue branch January 11, 2018 20:01

bflad added this to the v1.7.0 milestone Jan 12, 2018

drewsonne mentioned this pull request Jan 14, 2018

[RFC] Code Style and Schema Structure #2986

Merged

ghost locked and limited conversation to collaborators Apr 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Resource: AWS Glue Catalog Database #2175

New Resource: AWS Glue Catalog Database #2175

drewsonne commented Nov 5, 2017

bflad left a comment

drewsonne commented Nov 5, 2017 •

edited

Loading

drewsonne commented Nov 13, 2017

drewsonne commented Nov 15, 2017

bflad left a comment

bflad Dec 18, 2017

bflad Dec 18, 2017 •

edited

Loading

bflad Dec 18, 2017

bflad Dec 18, 2017

bflad Dec 18, 2017

bflad Dec 18, 2017

bflad Dec 18, 2017

drewsonne Dec 19, 2017

bflad Dec 18, 2017

bflad Dec 18, 2017

drewsonne Dec 26, 2017

bflad Dec 26, 2017

bflad Dec 26, 2017

drewsonne Dec 26, 2017

drewsonne commented Dec 18, 2017

drewsonne commented Jan 10, 2018

drewsonne commented Jan 10, 2018

radeksimko commented Jan 10, 2018

cemo commented Jan 10, 2018

drewsonne commented Jan 11, 2018

drewsonne commented Jan 11, 2018

radeksimko commented Jan 11, 2018

drewsonne commented Jan 11, 2018 •

edited

Loading

radeksimko commented Jan 11, 2018

drewsonne commented Jan 11, 2018 •

edited

Loading

drewsonne commented Jan 12, 2018

bflad commented Jan 12, 2018

ghost commented Apr 8, 2020


		## Attributes Reference

		The following attributes are exported:


		## Import

		Glue Catalog Databasess can be imported using the `name`, e.g.

New Resource: AWS Glue Catalog Database #2175

New Resource: AWS Glue Catalog Database #2175

Conversation

drewsonne commented Nov 5, 2017

bflad left a comment

Choose a reason for hiding this comment

drewsonne commented Nov 5, 2017 • edited Loading

drewsonne commented Nov 13, 2017

drewsonne commented Nov 15, 2017

bflad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bflad Dec 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drewsonne commented Dec 18, 2017

drewsonne commented Jan 10, 2018

drewsonne commented Jan 10, 2018

radeksimko commented Jan 10, 2018

cemo commented Jan 10, 2018

drewsonne commented Jan 11, 2018

drewsonne commented Jan 11, 2018

radeksimko commented Jan 11, 2018

drewsonne commented Jan 11, 2018 • edited Loading

radeksimko commented Jan 11, 2018

drewsonne commented Jan 11, 2018 • edited Loading

drewsonne commented Jan 12, 2018

bflad commented Jan 12, 2018

ghost commented Apr 8, 2020

drewsonne commented Nov 5, 2017 •

edited

Loading

bflad Dec 18, 2017 •

edited

Loading

drewsonne commented Jan 11, 2018 •

edited

Loading

drewsonne commented Jan 11, 2018 •

edited

Loading