New resource - Glue crawlers

[darrenhaken]

Closes #3875

Changes proposed in this pull request:

• Add support for a new resource, Glue Crawlers.

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSGlueCrawler'

[darrenhaken]

@bflad I am still having issues having this work for me.

If you run the test do you also see failures around IAM and assuming the role?
I have been stumped by this and it has meant I haven't really come back to the PR to carry on with it.

Could you help fix this with me and I'll carry on with the rest of the PR?

[darrenhaken]

@bflad still not heard back, do you think it's something you can look at?

We're using the Crawlers quite a lot now and getting it integrated into TF would be fantastic.

[f0rk]

@darrenhaken I have resolved the role issue, you can see it here: darrenhaken#1

[darrenhaken]

@f0rk awesome! I'll give this a try tomorrow and confirm if it works. I appreciate the help.

[darrenhaken]

Looks like it's working @f0rk and I've included a few more of the fields to the crawler.

[jordanfowler]

How's this looking? We'd love to get this into our terraform stack.

[jordanfowler]

@mabushey 🌮

[darrenhaken]

@jordanfowler I've been on vacation but I'm picking this back up atm. I have the basic Crawler working and now I'm getting through all the optional fields.

You're welcome to work on it as well if you'd like.

I'm making good progress on it now though.

[darrenhaken]

@bflad thanks for getting back to me with the changes.

I'll do my best to change what I can today but if you can finish some of it off I'd certainly appreciate it. I'm quite swamped this week 👎

I have already pushed the rename and I'll do see what I can do on the rest.

[darrenhaken]

@bflad I have made a commit which fixes most of the issues you've raised. Do you think you could do the last few? I ran out of time.

[bflad]

@darrenhaken you rock 🥇 I'll add one last commit after yours to get this across the finish line. We should be able to release this today! More soon.

[darrenhaken]

Sweet! I've got a commit on my laptop to do the test with random names which I can commit in an hour or so. I didn't get chance to do the docs and if I missed anything then I'm sorry!

…

On Wed, 20 Jun 2018, 16:20 Brian Flad, ***@***.***> wrote: @darrenhaken <https://github.com/darrenhaken> you rock 🥇 I'll add one last commit after yours to get this across the finish line. We should be able to release this today! More soon. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4484 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6Me4ENoNRqw87nBlee_q-h9DyDiPL2ks5t-mhVgaJpZM4T3On5> .

[darrenhaken]

@bflad Sadly I had to revert my commit on the random acceptance test names. The basic test failed with a diff error and I do not have any time left today to take a look.

If you could finish the last bits off I'd really appreciate it. We will be really pleased when this goes in :)

Do you think you can?

P.S I didnt get chance to do the docs.

[darrenhaken]

@bflad sorry to mess you around with this but I did some more investigation.

The basic test was failing by adding in one of your suggested changes:

if err := d.Set("schema_change_policy", []map[string]string{schemaPolicy}); err != nil {
			return fmt.Errorf("error setting schema_change_policy: %s", schemaPolicy)
		}

I have commented this out, for now, see line 342.

The plus is that I have been able to keep the random AC test names in.

[darrenhaken]

@bflad any luck?

[bflad]

I have everything working in a commit after e5832d7:

make testacc TEST=./aws TESTARGS='-run=TestAccAWSGlueCrawler'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSGlueCrawler -timeout 120m
=== RUN   TestAccAWSGlueCrawler_JdbcTarget
--- PASS: TestAccAWSGlueCrawler_JdbcTarget (37.59s)
=== RUN   TestAccAWSGlueCrawler_JdbcTarget_Exclusions
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Exclusions (37.92s)
=== RUN   TestAccAWSGlueCrawler_JdbcTarget_Multiple
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Multiple (49.36s)
=== RUN   TestAccAWSGlueCrawler_S3Target
--- PASS: TestAccAWSGlueCrawler_S3Target (36.53s)
=== RUN   TestAccAWSGlueCrawler_S3Target_Exclusions
--- PASS: TestAccAWSGlueCrawler_S3Target_Exclusions (36.85s)
=== RUN   TestAccAWSGlueCrawler_S3Target_Multiple
--- PASS: TestAccAWSGlueCrawler_S3Target_Multiple (47.93s)
=== RUN   TestAccAWSGlueCrawler_Classifiers
--- PASS: TestAccAWSGlueCrawler_Classifiers (45.37s)
=== RUN   TestAccAWSGlueCrawler_Configuration
--- PASS: TestAccAWSGlueCrawler_Configuration (38.19s)
=== RUN   TestAccAWSGlueCrawler_Description
--- PASS: TestAccAWSGlueCrawler_Description (37.96s)
=== RUN   TestAccAWSGlueCrawler_Schedule
--- PASS: TestAccAWSGlueCrawler_Schedule (37.97s)
=== RUN   TestAccAWSGlueCrawler_SchemaChangePolicy
--- PASS: TestAccAWSGlueCrawler_SchemaChangePolicy (37.68s)
=== RUN   TestAccAWSGlueCrawler_TablePrefix
--- PASS: TestAccAWSGlueCrawler_TablePrefix (37.41s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	480.799s

[bflad]

Thanks so much for your work here @darrenhaken! All commits through e5832d7 are merged (sorry thought you were done pushing already earlier) and 6f41ed5 finishes the implementation including testing all the attributes individually with updates and documentation. I'll need to manually close this PR since GitHub will notice the last two commits in this branch missing from earlier today.

[bflad]

This has been released in version 1.24.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!