Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

r/glue_crawler - add support for lineage_configuration, recrawl_policy and plan time validations. #16714

Merged
merged 11 commits into from
Jan 5, 2021
Merged

r/glue_crawler - add support for lineage_configuration, recrawl_policy and plan time validations. #16714

merged 11 commits into from
Jan 5, 2021

Conversation

DrFaust92
Copy link
Collaborator

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #16013

Release note for CHANGELOG:

resource_aws_glue_crawler - add support for `lineage_configuration` and `recrawl_policy`.
resource_aws_glue_crawler - add plan time validations to `name`, `description` and `table_prefix`.

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSGlueCrawler_'
--- PASS: TestAccAWSGlueCrawler_disappears (53.07s)
--- PASS: TestAccAWSGlueCrawler_S3Target_ConnectionName (89.49s)
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget (93.09s)
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget (98.45s)
--- PASS: TestAccAWSGlueCrawler_Description (98.51s)
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Exclusions (99.57s)
--- PASS: TestAccAWSGlueCrawler_S3Target (100.08s)
--- PASS: TestAccAWSGlueCrawler_JdbcTarget (100.08s)
--- PASS: TestAccAWSGlueCrawler_Configuration (100.90s)
--- PASS: TestAccAWSGlueCrawler_RemoveTablePrefix (105.02s)
--- PASS: TestAccAWSGlueCrawler_S3Target_Exclusions (105.25s)
--- PASS: TestAccAWSGlueCrawler_SecurityConfiguration (115.42s)
--- PASS: TestAccAWSGlueCrawler_recrawlPolicy (129.38s)
--- PASS: TestAccAWSGlueCrawler_S3Target_Multiple (130.18s)
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Multiple (132.92s)
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget_multiple (132.94s)
--- PASS: TestAccAWSGlueCrawler_lineageConfig (133.70s)
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget_scan_all (135.81s)
--- PASS: TestAccAWSGlueCrawler_Role_ARN_Path (46.39s)
--- PASS: TestAccAWSGlueCrawler_Role_Name_Path (45.99s)
--- PASS: TestAccAWSGlueCrawler_Role_ARN_NoPath (51.22s)
--- PASS: TestAccAWSGlueCrawler_CatalogTarget (153.98s)
--- PASS: TestAccAWSGlueCrawler_Tags (116.68s)
--- PASS: TestAccAWSGlueCrawler_TablePrefix (81.87s)
--- PASS: TestAccAWSGlueCrawler_SchemaChangePolicy (86.17s)
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget_scanRate (116.58s)
--- PASS: TestAccAWSGlueCrawler_Schedule (121.75s)
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget_scanAll (115.12s)
--- PASS: TestAccAWSGlueCrawler_Classifiers (115.62s)
--- PASS: TestAccAWSGlueCrawler_CatalogTarget_Multiple (240.51s)

@DrFaust92 DrFaust92 requested a review from a team as a code owner December 11, 2020 08:55
@ghost ghost added size/XL Managed by automation to categorize the size of a PR. documentation Introduces or discusses updates to documentation. service/glue Issues and PRs that pertain to the glue service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Dec 11, 2020
@DrFaust92 DrFaust92 added the enhancement Requests to existing resources that expand the functionality or scope. label Dec 11, 2020
ewbankkit
ewbankkit reviewed Dec 11, 2020
View reviewed changes
aws/resource_aws_glue_crawler.go Outdated
Comment on lines 595 to 598
if d.HasChanges(
"catalog_target", "classifiers", "configuration", "description", "dynamodb_target", "jdbc_target", "role",
"s3_target", "schedule", "schema_change_policy", "security_configuration", "table_prefix", "mongodb_target") {
"s3_target", "schedule", "schema_change_policy", "security_configuration", "table_prefix", "mongodb_target",
"lineage_configuration", "recrawl_policy") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest changing to d.HasChangesExcept("tags") so that we don't miss any non-tags attributes in future.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, wasnt aware of this one

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #16627.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added change and retested. Thanks for this.

ewbankkit
ewbankkit approved these changes Dec 11, 2020
View reviewed changes
Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSGlueCrawler_' ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 2 -run=TestAccAWSGlueCrawler_ -timeout 120m
=== RUN   TestAccAWSGlueCrawler_DynamodbTarget
=== PAUSE TestAccAWSGlueCrawler_DynamodbTarget
=== RUN   TestAccAWSGlueCrawler_DynamodbTarget_scanAll
=== PAUSE TestAccAWSGlueCrawler_DynamodbTarget_scanAll
=== RUN   TestAccAWSGlueCrawler_DynamodbTarget_scanRate
=== PAUSE TestAccAWSGlueCrawler_DynamodbTarget_scanRate
=== RUN   TestAccAWSGlueCrawler_JdbcTarget
=== PAUSE TestAccAWSGlueCrawler_JdbcTarget
=== RUN   TestAccAWSGlueCrawler_JdbcTarget_Exclusions
=== PAUSE TestAccAWSGlueCrawler_JdbcTarget_Exclusions
=== RUN   TestAccAWSGlueCrawler_JdbcTarget_Multiple
=== PAUSE TestAccAWSGlueCrawler_JdbcTarget_Multiple
=== RUN   TestAccAWSGlueCrawler_mongoDBTarget
=== PAUSE TestAccAWSGlueCrawler_mongoDBTarget
=== RUN   TestAccAWSGlueCrawler_mongoDBTarget_scan_all
=== PAUSE TestAccAWSGlueCrawler_mongoDBTarget_scan_all
=== RUN   TestAccAWSGlueCrawler_mongoDBTarget_multiple
=== PAUSE TestAccAWSGlueCrawler_mongoDBTarget_multiple
=== RUN   TestAccAWSGlueCrawler_S3Target
=== PAUSE TestAccAWSGlueCrawler_S3Target
=== RUN   TestAccAWSGlueCrawler_S3Target_ConnectionName
=== PAUSE TestAccAWSGlueCrawler_S3Target_ConnectionName
=== RUN   TestAccAWSGlueCrawler_S3Target_Exclusions
=== PAUSE TestAccAWSGlueCrawler_S3Target_Exclusions
=== RUN   TestAccAWSGlueCrawler_S3Target_Multiple
=== PAUSE TestAccAWSGlueCrawler_S3Target_Multiple
=== RUN   TestAccAWSGlueCrawler_CatalogTarget
=== PAUSE TestAccAWSGlueCrawler_CatalogTarget
=== RUN   TestAccAWSGlueCrawler_CatalogTarget_Multiple
=== PAUSE TestAccAWSGlueCrawler_CatalogTarget_Multiple
=== RUN   TestAccAWSGlueCrawler_disappears
=== PAUSE TestAccAWSGlueCrawler_disappears
=== RUN   TestAccAWSGlueCrawler_Classifiers
=== PAUSE TestAccAWSGlueCrawler_Classifiers
=== RUN   TestAccAWSGlueCrawler_Configuration
=== PAUSE TestAccAWSGlueCrawler_Configuration
=== RUN   TestAccAWSGlueCrawler_Description
=== PAUSE TestAccAWSGlueCrawler_Description
=== RUN   TestAccAWSGlueCrawler_Role_ARN_NoPath
=== PAUSE TestAccAWSGlueCrawler_Role_ARN_NoPath
=== RUN   TestAccAWSGlueCrawler_Role_ARN_Path
=== PAUSE TestAccAWSGlueCrawler_Role_ARN_Path
=== RUN   TestAccAWSGlueCrawler_Role_Name_Path
=== PAUSE TestAccAWSGlueCrawler_Role_Name_Path
=== RUN   TestAccAWSGlueCrawler_Schedule
=== PAUSE TestAccAWSGlueCrawler_Schedule
=== RUN   TestAccAWSGlueCrawler_SchemaChangePolicy
=== PAUSE TestAccAWSGlueCrawler_SchemaChangePolicy
=== RUN   TestAccAWSGlueCrawler_TablePrefix
=== PAUSE TestAccAWSGlueCrawler_TablePrefix
=== RUN   TestAccAWSGlueCrawler_RemoveTablePrefix
=== PAUSE TestAccAWSGlueCrawler_RemoveTablePrefix
=== RUN   TestAccAWSGlueCrawler_Tags
=== PAUSE TestAccAWSGlueCrawler_Tags
=== RUN   TestAccAWSGlueCrawler_SecurityConfiguration
=== PAUSE TestAccAWSGlueCrawler_SecurityConfiguration
=== RUN   TestAccAWSGlueCrawler_lineageConfig
=== PAUSE TestAccAWSGlueCrawler_lineageConfig
=== RUN   TestAccAWSGlueCrawler_recrawlPolicy
=== PAUSE TestAccAWSGlueCrawler_recrawlPolicy
=== CONT  TestAccAWSGlueCrawler_DynamodbTarget
=== CONT  TestAccAWSGlueCrawler_Classifiers
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget (35.17s)
=== CONT  TestAccAWSGlueCrawler_recrawlPolicy
--- PASS: TestAccAWSGlueCrawler_Classifiers (51.49s)
=== CONT  TestAccAWSGlueCrawler_lineageConfig
--- PASS: TestAccAWSGlueCrawler_recrawlPolicy (48.89s)
=== CONT  TestAccAWSGlueCrawler_SecurityConfiguration
--- PASS: TestAccAWSGlueCrawler_lineageConfig (44.50s)
=== CONT  TestAccAWSGlueCrawler_Tags
--- PASS: TestAccAWSGlueCrawler_SecurityConfiguration (39.35s)
=== CONT  TestAccAWSGlueCrawler_RemoveTablePrefix
--- PASS: TestAccAWSGlueCrawler_Tags (48.82s)
=== CONT  TestAccAWSGlueCrawler_TablePrefix
=== CONT  TestAccAWSGlueCrawler_SchemaChangePolicy
--- PASS: TestAccAWSGlueCrawler_RemoveTablePrefix (37.36s)
--- PASS: TestAccAWSGlueCrawler_TablePrefix (37.39s)
=== CONT  TestAccAWSGlueCrawler_Schedule
=== CONT  TestAccAWSGlueCrawler_Role_Name_Path
--- PASS: TestAccAWSGlueCrawler_SchemaChangePolicy (37.18s)
--- PASS: TestAccAWSGlueCrawler_Role_Name_Path (25.97s)
=== CONT  TestAccAWSGlueCrawler_Role_ARN_Path
--- PASS: TestAccAWSGlueCrawler_Schedule (49.77s)
=== CONT  TestAccAWSGlueCrawler_Role_ARN_NoPath
=== CONT  TestAccAWSGlueCrawler_Description
--- PASS: TestAccAWSGlueCrawler_Role_ARN_Path (25.45s)
--- PASS: TestAccAWSGlueCrawler_Role_ARN_NoPath (26.44s)
=== CONT  TestAccAWSGlueCrawler_Configuration
--- PASS: TestAccAWSGlueCrawler_Description (32.98s)
=== CONT  TestAccAWSGlueCrawler_mongoDBTarget_multiple
--- PASS: TestAccAWSGlueCrawler_Configuration (32.92s)
=== CONT  TestAccAWSGlueCrawler_disappears
--- PASS: TestAccAWSGlueCrawler_disappears (19.25s)
=== CONT  TestAccAWSGlueCrawler_CatalogTarget_Multiple
    resource_aws_glue_crawler_test.go:791: Step 1/4 error: Error running apply: 2020/12/11 09:40:58 [DEBUG] Using modified User-Agent: Terraform/0.12.26 HashiCorp-terraform-exec/0.10.0
        
        Error: error creating Glue crawler: InvalidInputException: Insufficient Lake Formation permission(s) on tf-acc-test-2203798892594810461_table_0 (Service: AmazonDataCatalog; Status Code: 400; Error Code: AccessDeniedException; Request ID: f0143276-9167-499e-8e9b-c0912bb8011e; Proxy: null)
        
        
--- FAIL: TestAccAWSGlueCrawler_CatalogTarget_Multiple (17.02s)
=== CONT  TestAccAWSGlueCrawler_CatalogTarget
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget_multiple (49.83s)
=== CONT  TestAccAWSGlueCrawler_S3Target_Multiple
=== CONT  TestAccAWSGlueCrawler_CatalogTarget
    resource_aws_glue_crawler_test.go:719: Step 1/3 error: Error running apply: 2020/12/11 09:41:15 [DEBUG] Using modified User-Agent: Terraform/0.12.26 HashiCorp-terraform-exec/0.10.0
        
        Error: error creating Glue crawler: InvalidInputException: Insufficient Lake Formation permission(s) on tf-acc-test-944679052379795349_table_0 (Service: AmazonDataCatalog; Status Code: 400; Error Code: AccessDeniedException; Request ID: 4ab5472b-0f7f-43af-a1cd-eac2f512986c; Proxy: null)
        
        
--- FAIL: TestAccAWSGlueCrawler_CatalogTarget (16.06s)
=== CONT  TestAccAWSGlueCrawler_S3Target_Exclusions
--- PASS: TestAccAWSGlueCrawler_S3Target_Multiple (44.18s)
=== CONT  TestAccAWSGlueCrawler_S3Target_ConnectionName
--- PASS: TestAccAWSGlueCrawler_S3Target_Exclusions (33.34s)
=== CONT  TestAccAWSGlueCrawler_S3Target
--- PASS: TestAccAWSGlueCrawler_S3Target_ConnectionName (31.52s)
=== CONT  TestAccAWSGlueCrawler_JdbcTarget_Exclusions
--- PASS: TestAccAWSGlueCrawler_S3Target (33.21s)
=== CONT  TestAccAWSGlueCrawler_mongoDBTarget_scan_all
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Exclusions (37.49s)
=== CONT  TestAccAWSGlueCrawler_mongoDBTarget
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget_scan_all (49.45s)
=== CONT  TestAccAWSGlueCrawler_JdbcTarget_Multiple
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget (38.82s)
=== CONT  TestAccAWSGlueCrawler_DynamodbTarget_scanRate
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Multiple (49.55s)
=== CONT  TestAccAWSGlueCrawler_JdbcTarget
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget_scanRate (48.29s)
=== CONT  TestAccAWSGlueCrawler_DynamodbTarget_scanAll
--- PASS: TestAccAWSGlueCrawler_JdbcTarget (37.74s)
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget_scanAll (48.47s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	581.072s
FAIL
GNUmakefile:27: recipe for target 'testacc' failed
make: *** [testacc] Error 1

Failures unrelated to these changes.

@DrFaust92 DrFaust92 force-pushed the r/glue_crawler_lineage branch from 0554c5d to f4e18d7 Compare December 20, 2020 08:15
@breathingdust
Copy link
Member

LGTM 🚀 Thanks @DrFaust92

Test failures are unrelated to change, will create a ticket to track. This may have been caused by adding LakeFormation support.

Verified Acceptance Tests in Commercial (us-west-2)

make testacc TESTARGS='-run=TestAccAWSGlueCrawler_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSGlueCrawler_ -timeout 120m
=== RUN   TestAccAWSGlueCrawler_DynamodbTarget
=== PAUSE TestAccAWSGlueCrawler_DynamodbTarget
=== RUN   TestAccAWSGlueCrawler_DynamodbTarget_scanAll
=== PAUSE TestAccAWSGlueCrawler_DynamodbTarget_scanAll
=== RUN   TestAccAWSGlueCrawler_DynamodbTarget_scanRate
=== PAUSE TestAccAWSGlueCrawler_DynamodbTarget_scanRate
=== RUN   TestAccAWSGlueCrawler_JdbcTarget
=== PAUSE TestAccAWSGlueCrawler_JdbcTarget
=== RUN   TestAccAWSGlueCrawler_JdbcTarget_Exclusions
=== PAUSE TestAccAWSGlueCrawler_JdbcTarget_Exclusions
=== RUN   TestAccAWSGlueCrawler_JdbcTarget_Multiple
=== PAUSE TestAccAWSGlueCrawler_JdbcTarget_Multiple
=== RUN   TestAccAWSGlueCrawler_mongoDBTarget
=== PAUSE TestAccAWSGlueCrawler_mongoDBTarget
=== RUN   TestAccAWSGlueCrawler_mongoDBTarget_scan_all
=== PAUSE TestAccAWSGlueCrawler_mongoDBTarget_scan_all
=== RUN   TestAccAWSGlueCrawler_mongoDBTarget_multiple
=== PAUSE TestAccAWSGlueCrawler_mongoDBTarget_multiple
=== RUN   TestAccAWSGlueCrawler_S3Target
=== PAUSE TestAccAWSGlueCrawler_S3Target
=== RUN   TestAccAWSGlueCrawler_S3Target_ConnectionName
=== PAUSE TestAccAWSGlueCrawler_S3Target_ConnectionName
=== RUN   TestAccAWSGlueCrawler_S3Target_Exclusions
=== PAUSE TestAccAWSGlueCrawler_S3Target_Exclusions
=== RUN   TestAccAWSGlueCrawler_S3Target_Multiple
=== PAUSE TestAccAWSGlueCrawler_S3Target_Multiple
=== RUN   TestAccAWSGlueCrawler_CatalogTarget
=== PAUSE TestAccAWSGlueCrawler_CatalogTarget
=== RUN   TestAccAWSGlueCrawler_CatalogTarget_Multiple
=== PAUSE TestAccAWSGlueCrawler_CatalogTarget_Multiple
=== RUN   TestAccAWSGlueCrawler_disappears
=== PAUSE TestAccAWSGlueCrawler_disappears
=== RUN   TestAccAWSGlueCrawler_Classifiers
=== PAUSE TestAccAWSGlueCrawler_Classifiers
=== RUN   TestAccAWSGlueCrawler_Configuration
=== PAUSE TestAccAWSGlueCrawler_Configuration
=== RUN   TestAccAWSGlueCrawler_Description
=== PAUSE TestAccAWSGlueCrawler_Description
=== RUN   TestAccAWSGlueCrawler_Role_ARN_NoPath
=== PAUSE TestAccAWSGlueCrawler_Role_ARN_NoPath
=== RUN   TestAccAWSGlueCrawler_Role_ARN_Path
=== PAUSE TestAccAWSGlueCrawler_Role_ARN_Path
=== RUN   TestAccAWSGlueCrawler_Role_Name_Path
=== PAUSE TestAccAWSGlueCrawler_Role_Name_Path
=== RUN   TestAccAWSGlueCrawler_Schedule
=== PAUSE TestAccAWSGlueCrawler_Schedule
=== RUN   TestAccAWSGlueCrawler_SchemaChangePolicy
=== PAUSE TestAccAWSGlueCrawler_SchemaChangePolicy
=== RUN   TestAccAWSGlueCrawler_TablePrefix
=== PAUSE TestAccAWSGlueCrawler_TablePrefix
=== RUN   TestAccAWSGlueCrawler_RemoveTablePrefix
=== PAUSE TestAccAWSGlueCrawler_RemoveTablePrefix
=== RUN   TestAccAWSGlueCrawler_Tags
=== PAUSE TestAccAWSGlueCrawler_Tags
=== RUN   TestAccAWSGlueCrawler_SecurityConfiguration
=== PAUSE TestAccAWSGlueCrawler_SecurityConfiguration
=== RUN   TestAccAWSGlueCrawler_lineageConfig
=== PAUSE TestAccAWSGlueCrawler_lineageConfig
=== RUN   TestAccAWSGlueCrawler_recrawlPolicy
=== PAUSE TestAccAWSGlueCrawler_recrawlPolicy
=== CONT  TestAccAWSGlueCrawler_DynamodbTarget
=== CONT  TestAccAWSGlueCrawler_Role_ARN_Path
=== CONT  TestAccAWSGlueCrawler_S3Target_ConnectionName
=== CONT  TestAccAWSGlueCrawler_S3Target_Exclusions
=== CONT  TestAccAWSGlueCrawler_recrawlPolicy
=== CONT  TestAccAWSGlueCrawler_lineageConfig
=== CONT  TestAccAWSGlueCrawler_SecurityConfiguration
=== CONT  TestAccAWSGlueCrawler_Tags
=== CONT  TestAccAWSGlueCrawler_RemoveTablePrefix
=== CONT  TestAccAWSGlueCrawler_TablePrefix
=== CONT  TestAccAWSGlueCrawler_SchemaChangePolicy
=== CONT  TestAccAWSGlueCrawler_Schedule
=== CONT  TestAccAWSGlueCrawler_Role_Name_Path
=== CONT  TestAccAWSGlueCrawler_disappears
=== CONT  TestAccAWSGlueCrawler_Role_ARN_NoPath
=== CONT  TestAccAWSGlueCrawler_Description
=== CONT  TestAccAWSGlueCrawler_Configuration
=== CONT  TestAccAWSGlueCrawler_Classifiers
=== CONT  TestAccAWSGlueCrawler_CatalogTarget
=== CONT  TestAccAWSGlueCrawler_CatalogTarget_Multiple
2021/01/04 16:03:44 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
2021/01/04 16:03:44 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2021/01/04 16:03:54 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
=== CONT  TestAccAWSGlueCrawler_CatalogTarget
    resource_aws_glue_crawler_test.go:719: Step 1/3 error: Error running apply: 2021/01/04 16:03:44 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.10.0

        Error: error creating Glue crawler: InvalidInputException: Insufficient Lake Formation permission(s) on tf-acc-test-8554921507484898476_table_0 (Service: AmazonDataCatalog; Status Code: 400; Error Code: AccessDeniedException; Request ID: 84766acd-6c7f-4bc2-8a96-676f445ea773; Proxy: null)


2021/01/04 16:03:58 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- FAIL: TestAccAWSGlueCrawler_CatalogTarget (23.23s)
=== CONT  TestAccAWSGlueCrawler_S3Target_Multiple
=== CONT  TestAccAWSGlueCrawler_CatalogTarget_Multiple
    resource_aws_glue_crawler_test.go:791: Step 1/4 error: Error running apply: 2021/01/04 16:03:44 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.10.0

        Error: error creating Glue crawler: InvalidInputException: Insufficient Lake Formation permission(s) on tf-acc-test-8011488058563598958_table_0 (Service: AmazonDataCatalog; Status Code: 400; Error Code: AccessDeniedException; Request ID: df2a67ec-5d0d-4700-99f6-54485b619ec7; Proxy: null)


2021/01/04 16:04:02 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- FAIL: TestAccAWSGlueCrawler_CatalogTarget_Multiple (29.17s)
=== CONT  TestAccAWSGlueCrawler_JdbcTarget_Multiple
--- PASS: TestAccAWSGlueCrawler_Role_Name_Path (31.46s)
=== CONT  TestAccAWSGlueCrawler_S3Target
--- PASS: TestAccAWSGlueCrawler_Role_ARN_Path (32.82s)
=== CONT  TestAccAWSGlueCrawler_mongoDBTarget_multiple
2021/01/04 16:04:08 [WARN] Truncating attribute path of 0 diagnostics for TypeSet
2021/01/04 16:04:08 [WARN] Truncating attribute path of 0 diagnostics for TypeSet
2021/01/04 16:04:08 [WARN] Truncating attribute path of 0 diagnostics for TypeSet
2021/01/04 16:04:08 [DEBUG] Reading Partition.
2021/01/04 16:04:08 [DEBUG] Setting AWS Partition to aws.
2021/01/04 16:04:08 [DEBUG] Setting AWS URL Suffix to amazonaws.com.
2021/01/04 16:04:08 [DEBUG] Getting IAM Policy: {
  PolicyArn: "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}
2021/01/04 16:04:08 [DEBUG] Waiting for state to become: [success]
--- PASS: TestAccAWSGlueCrawler_disappears (34.13s)
=== CONT  TestAccAWSGlueCrawler_mongoDBTarget_scan_all
2021/01/04 16:04:10 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- PASS: TestAccAWSGlueCrawler_Role_ARN_NoPath (34.98s)
=== CONT  TestAccAWSGlueCrawler_mongoDBTarget
2021/01/04 16:04:10 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2021/01/04 16:04:10 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2021/01/04 16:04:11 [DEBUG] Getting IAM Policy Version: {
  PolicyArn: "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole",
  VersionId: "v4"
}
2021/01/04 16:04:11 [DEBUG] Waiting for state to become: [success]
2021/01/04 16:04:14 [DEBUG] Reading Partition.
2021/01/04 16:04:14 [DEBUG] Setting AWS Partition to aws.
2021/01/04 16:04:14 [DEBUG] Setting AWS URL Suffix to amazonaws.com.
2021/01/04 16:04:14 [WARN] Truncating attribute path of 0 diagnostics for TypeSet
2021/01/04 16:04:14 [WARN] Truncating attribute path of 0 diagnostics for TypeSet
2021/01/04 16:04:14 [WARN] Truncating attribute path of 0 diagnostics for TypeSet
2021/01/04 16:04:18 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2021/01/04 16:04:22 [TRACE] Waiting 4s before next try
2021/01/04 16:04:23 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2021/01/04 16:04:24 [DEBUG] Getting IAM Policy Version: {
  PolicyArn: "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole",
  VersionId: "v4"
}
2021/01/04 16:04:24 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
--- PASS: TestAccAWSGlueCrawler_SchemaChangePolicy (50.41s)
=== CONT  TestAccAWSGlueCrawler_JdbcTarget
--- PASS: TestAccAWSGlueCrawler_S3Target_ConnectionName (50.62s)
=== CONT  TestAccAWSGlueCrawler_JdbcTarget_Exclusions
--- PASS: TestAccAWSGlueCrawler_RemoveTablePrefix (52.27s)
=== CONT  TestAccAWSGlueCrawler_DynamodbTarget_scanRate
--- PASS: TestAccAWSGlueCrawler_TablePrefix (56.03s)
=== CONT  TestAccAWSGlueCrawler_DynamodbTarget_scanAll
2021/01/04 16:04:35 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget (61.84s)
--- PASS: TestAccAWSGlueCrawler_SecurityConfiguration (62.29s)
2021/01/04 16:04:39 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2021/01/04 16:04:40 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- PASS: TestAccAWSGlueCrawler_Configuration (65.53s)
--- PASS: TestAccAWSGlueCrawler_S3Target_Exclusions (65.98s)
--- PASS: TestAccAWSGlueCrawler_Classifiers (70.11s)
--- PASS: TestAccAWSGlueCrawler_recrawlPolicy (74.75s)
--- PASS: TestAccAWSGlueCrawler_lineageConfig (78.36s)
--- PASS: TestAccAWSGlueCrawler_S3Target (49.44s)
2021/01/04 16:04:58 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget (51.75s)
2021/01/04 16:05:03 [DEBUG] deleting Glue crawler: tf-acc-test-3201453841987876955
--- PASS: TestAccAWSGlueCrawler_S3Target_Multiple (66.46s)
--- PASS: TestAccAWSGlueCrawler_Tags (93.43s)
--- PASS: TestAccAWSGlueCrawler_JdbcTarget (44.65s)
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Exclusions (44.63s)
--- PASS: TestAccAWSGlueCrawler_Schedule (97.54s)
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget_multiple (67.65s)
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Multiple (72.43s)
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget_scan_all (69.10s)
--- PASS: TestAccAWSGlueCrawler_Description (112.99s)
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget_scanAll (58.83s)
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget_scanRate (63.89s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	117.960s
FAIL
make: *** [testacc] Error 1

Verified Acceptance Tests in GovCloud (us-gov-west-1)

make testacc TESTARGS='-run=TestAccAWSGlueCrawler_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSGlueCrawler_ -timeout 120m
=== RUN   TestAccAWSGlueCrawler_DynamodbTarget
=== PAUSE TestAccAWSGlueCrawler_DynamodbTarget
=== RUN   TestAccAWSGlueCrawler_DynamodbTarget_scanAll
=== PAUSE TestAccAWSGlueCrawler_DynamodbTarget_scanAll
=== RUN   TestAccAWSGlueCrawler_DynamodbTarget_scanRate
=== PAUSE TestAccAWSGlueCrawler_DynamodbTarget_scanRate
=== RUN   TestAccAWSGlueCrawler_JdbcTarget
=== PAUSE TestAccAWSGlueCrawler_JdbcTarget
=== RUN   TestAccAWSGlueCrawler_JdbcTarget_Exclusions
=== PAUSE TestAccAWSGlueCrawler_JdbcTarget_Exclusions
=== RUN   TestAccAWSGlueCrawler_JdbcTarget_Multiple
=== PAUSE TestAccAWSGlueCrawler_JdbcTarget_Multiple
=== RUN   TestAccAWSGlueCrawler_mongoDBTarget
=== PAUSE TestAccAWSGlueCrawler_mongoDBTarget
=== RUN   TestAccAWSGlueCrawler_mongoDBTarget_scan_all
=== PAUSE TestAccAWSGlueCrawler_mongoDBTarget_scan_all
=== RUN   TestAccAWSGlueCrawler_mongoDBTarget_multiple
=== PAUSE TestAccAWSGlueCrawler_mongoDBTarget_multiple
=== RUN   TestAccAWSGlueCrawler_S3Target
=== PAUSE TestAccAWSGlueCrawler_S3Target
=== RUN   TestAccAWSGlueCrawler_S3Target_ConnectionName
=== PAUSE TestAccAWSGlueCrawler_S3Target_ConnectionName
=== RUN   TestAccAWSGlueCrawler_S3Target_Exclusions
=== PAUSE TestAccAWSGlueCrawler_S3Target_Exclusions
=== RUN   TestAccAWSGlueCrawler_S3Target_Multiple
=== PAUSE TestAccAWSGlueCrawler_S3Target_Multiple
=== RUN   TestAccAWSGlueCrawler_CatalogTarget
=== PAUSE TestAccAWSGlueCrawler_CatalogTarget
=== RUN   TestAccAWSGlueCrawler_CatalogTarget_Multiple
=== PAUSE TestAccAWSGlueCrawler_CatalogTarget_Multiple
=== RUN   TestAccAWSGlueCrawler_disappears
=== PAUSE TestAccAWSGlueCrawler_disappears
=== RUN   TestAccAWSGlueCrawler_Classifiers
=== PAUSE TestAccAWSGlueCrawler_Classifiers
=== RUN   TestAccAWSGlueCrawler_Configuration
=== PAUSE TestAccAWSGlueCrawler_Configuration
=== RUN   TestAccAWSGlueCrawler_Description
=== PAUSE TestAccAWSGlueCrawler_Description
=== RUN   TestAccAWSGlueCrawler_Role_ARN_NoPath
=== PAUSE TestAccAWSGlueCrawler_Role_ARN_NoPath
=== RUN   TestAccAWSGlueCrawler_Role_ARN_Path
=== PAUSE TestAccAWSGlueCrawler_Role_ARN_Path
=== RUN   TestAccAWSGlueCrawler_Role_Name_Path
=== PAUSE TestAccAWSGlueCrawler_Role_Name_Path
=== RUN   TestAccAWSGlueCrawler_Schedule
=== PAUSE TestAccAWSGlueCrawler_Schedule
=== RUN   TestAccAWSGlueCrawler_SchemaChangePolicy
=== PAUSE TestAccAWSGlueCrawler_SchemaChangePolicy
=== RUN   TestAccAWSGlueCrawler_TablePrefix
=== PAUSE TestAccAWSGlueCrawler_TablePrefix
=== RUN   TestAccAWSGlueCrawler_RemoveTablePrefix
=== PAUSE TestAccAWSGlueCrawler_RemoveTablePrefix
=== RUN   TestAccAWSGlueCrawler_Tags
=== PAUSE TestAccAWSGlueCrawler_Tags
=== RUN   TestAccAWSGlueCrawler_SecurityConfiguration
=== PAUSE TestAccAWSGlueCrawler_SecurityConfiguration
=== RUN   TestAccAWSGlueCrawler_lineageConfig
=== PAUSE TestAccAWSGlueCrawler_lineageConfig
=== RUN   TestAccAWSGlueCrawler_recrawlPolicy
=== PAUSE TestAccAWSGlueCrawler_recrawlPolicy
=== CONT  TestAccAWSGlueCrawler_DynamodbTarget
=== CONT  TestAccAWSGlueCrawler_SchemaChangePolicy
=== CONT  TestAccAWSGlueCrawler_SecurityConfiguration
=== CONT  TestAccAWSGlueCrawler_recrawlPolicy
=== CONT  TestAccAWSGlueCrawler_lineageConfig
=== CONT  TestAccAWSGlueCrawler_S3Target_Multiple
=== CONT  TestAccAWSGlueCrawler_Schedule
=== CONT  TestAccAWSGlueCrawler_Role_Name_Path
=== CONT  TestAccAWSGlueCrawler_Role_ARN_Path
=== CONT  TestAccAWSGlueCrawler_Role_ARN_NoPath
=== CONT  TestAccAWSGlueCrawler_Description
=== CONT  TestAccAWSGlueCrawler_Configuration
=== CONT  TestAccAWSGlueCrawler_Classifiers
=== CONT  TestAccAWSGlueCrawler_disappears
=== CONT  TestAccAWSGlueCrawler_CatalogTarget_Multiple
=== CONT  TestAccAWSGlueCrawler_CatalogTarget
=== CONT  TestAccAWSGlueCrawler_mongoDBTarget
=== CONT  TestAccAWSGlueCrawler_S3Target_Exclusions
=== CONT  TestAccAWSGlueCrawler_S3Target_ConnectionName
=== CONT  TestAccAWSGlueCrawler_S3Target
2021/01/05 11:42:54 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2021/01/05 11:42:55 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
=== CONT  TestAccAWSGlueCrawler_CatalogTarget
    resource_aws_glue_crawler_test.go:719: Step 1/3 error: Error running apply: 2021/01/05 11:42:55 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.10.0

        Error: error creating Glue crawler: InvalidInputException: Insufficient Lake Formation permission(s) on tf-acc-test-1887832137688294495_table_0 (Service: AmazonDataCatalog; Status Code: 400; Error Code: AccessDeniedException; Request ID: d6d1b74b-1111-4738-954d-494d9fcfe9bc; Proxy: null)


2021/01/05 11:43:07 [WARN] Got error running Terraform: 2021/01/05 11:42:55 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.10.0

Error: error creating Glue crawler: InvalidInputException: Insufficient Lake Formation permission(s) on tf-acc-test-5835294211335458174_table_0 (Service: AmazonDataCatalog; Status Code: 400; Error Code: AccessDeniedException; Request ID: a7c86f0d-bf72-43d5-9689-6dab86613d70; Proxy: null)


=== CONT  TestAccAWSGlueCrawler_CatalogTarget_Multiple
    resource_aws_glue_crawler_test.go:791: Step 1/4 error: Error running apply: 2021/01/05 11:42:55 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.10.0

        Error: error creating Glue crawler: InvalidInputException: Insufficient Lake Formation permission(s) on tf-acc-test-5835294211335458174_table_0 (Service: AmazonDataCatalog; Status Code: 400; Error Code: AccessDeniedException; Request ID: a7c86f0d-bf72-43d5-9689-6dab86613d70; Proxy: null)


--- FAIL: TestAccAWSGlueCrawler_CatalogTarget (19.72s)
=== CONT  TestAccAWSGlueCrawler_mongoDBTarget_multiple
--- FAIL: TestAccAWSGlueCrawler_CatalogTarget_Multiple (20.33s)
=== CONT  TestAccAWSGlueCrawler_mongoDBTarget_scan_all
2021/01/05 11:43:13 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- PASS: TestAccAWSGlueCrawler_disappears (28.02s)
=== CONT  TestAccAWSGlueCrawler_RemoveTablePrefix
2021/01/05 11:43:18 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2021/01/05 11:43:19 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- PASS: TestAccAWSGlueCrawler_Role_Name_Path (33.38s)
=== CONT  TestAccAWSGlueCrawler_Tags
--- PASS: TestAccAWSGlueCrawler_Role_ARN_Path (38.51s)
=== CONT  TestAccAWSGlueCrawler_JdbcTarget
2021/01/05 11:43:28 [TRACE] Waiting 2s before next try
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget (41.08s)
=== CONT  TestAccAWSGlueCrawler_JdbcTarget_Multiple
--- PASS: TestAccAWSGlueCrawler_S3Target_ConnectionName (41.25s)
=== CONT  TestAccAWSGlueCrawler_JdbcTarget_Exclusions
--- PASS: TestAccAWSGlueCrawler_S3Target (44.27s)
=== CONT  TestAccAWSGlueCrawler_DynamodbTarget_scanAll
2021/01/05 11:43:35 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2021/01/05 11:43:35 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- PASS: TestAccAWSGlueCrawler_Classifiers (50.90s)
=== CONT  TestAccAWSGlueCrawler_TablePrefix
2021/01/05 11:43:41 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- PASS: TestAccAWSGlueCrawler_Description (54.05s)
=== CONT  TestAccAWSGlueCrawler_DynamodbTarget_scanRate
2021/01/05 11:43:45 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- PASS: TestAccAWSGlueCrawler_S3Target_Exclusions (60.80s)
--- PASS: TestAccAWSGlueCrawler_SchemaChangePolicy (68.42s)
--- PASS: TestAccAWSGlueCrawler_lineageConfig (68.86s)
--- PASS: TestAccAWSGlueCrawler_recrawlPolicy (69.54s)
--- PASS: TestAccAWSGlueCrawler_JdbcTarget (35.22s)
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget_multiple (60.34s)
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Exclusions (39.36s)
--- PASS: TestAccAWSGlueCrawler_RemoveTablePrefix (52.74s)
--- PASS: TestAccAWSGlueCrawler_Role_ARN_NoPath (82.39s)
--- PASS: TestAccAWSGlueCrawler_S3Target_Multiple (83.13s)
--- PASS: TestAccAWSGlueCrawler_Tags (52.72s)
--- PASS: TestAccAWSGlueCrawler_mongoDBTarget_scan_all (70.35s)
--- PASS: TestAccAWSGlueCrawler_TablePrefix (45.34s)
--- PASS: TestAccAWSGlueCrawler_Schedule (97.73s)
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget_scanRate (45.11s)
--- PASS: TestAccAWSGlueCrawler_Configuration (100.89s)
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget_scanAll (61.34s)
--- PASS: TestAccAWSGlueCrawler_JdbcTarget_Multiple (64.84s)
--- PASS: TestAccAWSGlueCrawler_SecurityConfiguration (107.96s)
--- PASS: TestAccAWSGlueCrawler_DynamodbTarget (113.03s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	114.725s
FAIL
make: *** [testacc] Error 1

@breathingdust breathingdust merged commit 53d61a9 into hashicorp:master Jan 5, 2021
@github-actions github-actions bot added this to the v3.23.0 milestone Jan 5, 2021
breathingdust added a commit that referenced this pull request Jan 5, 2021
@breathingdust
Update CHANGELOG.md for #16714
2b14e0e
@ghost
Copy link

ghost commented Jan 8, 2021

This has been released in version 3.23.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented Feb 5, 2021

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked as resolved and limited conversation to collaborators Feb 5, 2021
@DrFaust92 DrFaust92 deleted the r/glue_crawler_lineage branch April 15, 2021 10:37
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@WarFox WarFox WarFox left review comments

@ewbankkit ewbankkit ewbankkit approved these changes

Assignees
No one assigned
Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. service/glue Issues and PRs that pertain to the glue service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v3.23.0
Development

Successfully merging this pull request may close these issues.

Support for recrawl-policy on aws_glue_crawler
4 participants
@DrFaust92 @breathingdust @WarFox @ewbankkit