Feature Request: Resource for managing AWS Glue Crawlers

[bflad]

Split feature request from #1416. Please note there are relevant comments in #1416 about this feature request. For issues not immediately being worked on, please use 👍 upvotes on this original issue comment to help guage community interest.

Terraform Version

terraform 0.10+
terraform-provider-aws (new feature)

Affected Resource(s)

• aws_glue_crawler

Terraform Configuration Files

# New Resource - not implemented
resource "aws_glue_crawler" "example" {
  # ...
}

Expected Behavior

Create, read, update, and delete support for AWS Glue crawlers.

Actual Behavior

Resource not implemented.

References

• https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-crawling.html
• https://docs.aws.amazon.com/sdk-for-go/api/service/glue/#Glue.CreateCrawler

[bflad]

/cc @darrenhaken @drewsonne

[darrenhaken]

Thanks @bflad

Are you able to help further with the issue I was having a few weeks ago?

[darrenhaken]

@bflad Are you able to help with the failing test I have?

Context:

I have written the code to support a basic Glue Crawler with the required fields set. The acceptance test is failing with permissions issues. It complains that the service role for Glue cannot assume a role.

If you could help me unblock this then I'll be able to continue. I'm stumped!

[cloudyparts]

@darrenhaken are you still blocked? Do you have a fork somewhere?

[cloudyparts]

@darrenhaken - Found your fork.

I think you might have the wrong ARN for the AWSGlueServiceRole on line 167.

I think the ARN is arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole.

[darrenhaken]

@cloudyparts I tried changing the role as you suggested but I still get:

--- FAIL: TestAccAWSGlueCrawler_basic (20.37s)
	testing.go:518: Step 0 error: Error applying: 1 error(s) occurred:

		* aws_glue_catalog_crawler.test: 1 error(s) occurred:

		* aws_glue_catalog_crawler.test: error creating Glue crawler: InvalidInputException: Service is unable to assume role arn:aws:iam::697329683179:role/AWSGlueServiceRoleDefault. Please verify role's TrustPolicy
			status code: 400, request id: a4f55ff2-4b19-11e8-8491-a9b7762b818e

FYI I've pushed to my fork the latest changes I have done, it's on branch glue_crawlers

[cloudyparts]

@darrenhaken I don't see anything apparent that would cause an 'assume role' error.

That said I have had better luck with the data.aws_iam_policy_document resource as opposed to inline policies.

You could try this => https://gist.github.com/cloudyparts/f7a80e940cf9648eb815bca6a0d43788

Note: I created this gist with code from my own modules which has no issues. (terraform=v0.11.7, terraform-provider-aws=1.16.0)

[darrenhaken]

@cloudyparts I tried your gist but it still failed, I wonder if there's an issue on the account. Thoughts?
I had set up an empty sandbox account to do testing on this.

[cloudyparts]

@darrenhaken it is difficult to understand the extent of the issue. I would try creating a role manually in the console then referencing the role arn in your test. See if you can get the crawler created... - I am still puzzled by the error output...

Nothing should be assuming the Glue role except for the glue service when the crawler is invoked. You are only creating the crawler ... not invoking it.

[bflad]

IAM is an eventually consistent service so creating new roles/policies/attachments might take a minute or two to propagate through all AWS services. These tend to bubble up as seemingly fatal errors like "role not found" or "unable to assume role" initially, but will work after some time.

For a large portion of other services we tend to wrap these errors in a resource.Retry() loop for a minute. Here is some example code:

err := resource.Retry(1 * time.Minute, func() *resource.RetryError {
  _, err := conn.Example(input) // current call that is validating IAM parameters and failing
  if err != nil {
    if isAWSErr(err, "InvalidInputException", "Service is unable to assume role") {
      return resource.RetryableError(err)
    }
    return resource.NonRetryableError(err)
  }
  return nil
})
if err != nil {
  return fmt.Errorf("error doing XXX: %s", err)
}

[bflad]

@darrenhaken do you need any more help? Are you able to submit a pull request with what you have so far? We would love to get this feature request into the project.

[darrenhaken]

@bflad yeah I'm still having trouble getting this working around the assume role. I thought I'd already opened a PR, I'll do that now.

I'm having to let others contribute too of course :)

[darrenhaken]

See PR for WIP #4484

[bflad]

Kudos to @darrenhaken -- the new aws_glue_crawler resource is merged into master and will release later today. 🎉

[bflad]

This has been released in version 1.24.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[darrenhaken]

@bflad thanks for the updates! Seems to all be working OK :)

[rehevkor5]

I was experiencing the same error as @darrenhaken ("InvalidInputException: Service is unable to assume role arn:aws:iam::00000000:role/AWSGlueServiceRole-foo. Please verify role's TrustPolicy"), and eventually I realized that this was because I was following the example in the documentation which recommends that you use the ".name" property of the IAM Role in order to populate the ".role" property of the aws_glue_crawler. However, I happened to use a non-default "path" on my IAM Role. Unfortunately, "aws_iam_role.name" does not include the path. It started working when I switched to referring to the Role by ARN instead.

I made this improvement to the docs as a result: #6285

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!