r/aws_glue_catalog_table: Fix removal of Parameters when updating Iceberg table #33374

ewbankkit · 2023-09-08T16:02:21Z

Description

Add back any "managed" parameters removed during read when updating a Glue catalog table.

Relations

Closes #33370.
Relates #33157.

Output from Acceptance Testing

Before

% make testacc TESTARGS='-run=TestAccGlueCatalogTable_openTableFormat' PKG=glue
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/glue/... -v -count 1 -parallel 20  -run=TestAccGlueCatalogTable_openTableFormat -timeout 180m
=== RUN   TestAccGlueCatalogTable_openTableFormat
=== PAUSE TestAccGlueCatalogTable_openTableFormat
=== CONT  TestAccGlueCatalogTable_openTableFormat
    catalog_table_test.go:690: Step 1/2 error: After applying this test step, the plan was not empty.
        stdout:
        
        
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place
        
        Terraform will perform the following actions:
        
          # aws_glue_catalog_table.test will be updated in-place
          ~ resource "aws_glue_catalog_table" "test" {
                id            = "123456789012:tf-acc-test-6613294283571452933:tf-acc-test-6613294283571452933"
                name          = "tf-acc-test-6613294283571452933"
              ~ parameters    = {
                  - "metadata_location" = "s3://tf-acc-test-6613294283571452933/files/tf-acc-test-6613294283571452933.db/tf-acc-test-6613294283571452933/metadata/00000-dd3a1341-59cc-4266-84f1-fd23c7961e34.metadata.json" -> null
                  - "table_type"        = "ICEBERG" -> null
                }
                # (5 unchanged attributes hidden)
        
        
              ~ storage_descriptor {
                  + bucket_columns            = []
                  + parameters                = {}
                    # (4 unchanged attributes hidden)
        
                  ~ columns {
                        name       = "my_column_1"
                      + parameters = {}
                        # (2 unchanged attributes hidden)
                    }
                  ~ columns {
                        name       = "my_column_2"
                      + parameters = {}
                        # (2 unchanged attributes hidden)
                    }
                }
                # (1 unchanged block hidden)
            }
        
        Plan: 0 to add, 1 to change, 0 to destroy.
--- FAIL: TestAccGlueCatalogTable_openTableFormat (18.84s)
FAIL
FAIL	github.com/hashicorp/terraform-provider-aws/internal/service/glue	23.977s
FAIL
make: *** [testacc] Error 1

After

% make testacc TESTARGS='-run=TestAccGlueCatalogTable_' PKG=glue ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/glue/... -v -count 1 -parallel 2  -run=TestAccGlueCatalogTable_ -timeout 180m
=== RUN   TestAccGlueCatalogTable_basic
=== PAUSE TestAccGlueCatalogTable_basic
=== RUN   TestAccGlueCatalogTable_columnParameters
=== PAUSE TestAccGlueCatalogTable_columnParameters
=== RUN   TestAccGlueCatalogTable_full
=== PAUSE TestAccGlueCatalogTable_full
=== RUN   TestAccGlueCatalogTable_Update_addValues
=== PAUSE TestAccGlueCatalogTable_Update_addValues
=== RUN   TestAccGlueCatalogTable_Update_replaceValues
=== PAUSE TestAccGlueCatalogTable_Update_replaceValues
=== RUN   TestAccGlueCatalogTable_StorageDescriptor_emptyBlock
=== PAUSE TestAccGlueCatalogTable_StorageDescriptor_emptyBlock
=== RUN   TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_emptyBlock
=== PAUSE TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_emptyBlock
=== RUN   TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_updateValues
=== PAUSE TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_updateValues
=== RUN   TestAccGlueCatalogTable_StorageDescriptorSkewedInfo_emptyBlock
=== PAUSE TestAccGlueCatalogTable_StorageDescriptorSkewedInfo_emptyBlock
=== RUN   TestAccGlueCatalogTable_StorageDescriptor_schemaReference
=== PAUSE TestAccGlueCatalogTable_StorageDescriptor_schemaReference
=== RUN   TestAccGlueCatalogTable_StorageDescriptor_schemaReferenceARN
=== PAUSE TestAccGlueCatalogTable_StorageDescriptor_schemaReferenceARN
=== RUN   TestAccGlueCatalogTable_partitionIndexesSingle
=== PAUSE TestAccGlueCatalogTable_partitionIndexesSingle
=== RUN   TestAccGlueCatalogTable_partitionIndexesMultiple
=== PAUSE TestAccGlueCatalogTable_partitionIndexesMultiple
=== RUN   TestAccGlueCatalogTable_Disappears_database
=== PAUSE TestAccGlueCatalogTable_Disappears_database
=== RUN   TestAccGlueCatalogTable_targetTable
=== PAUSE TestAccGlueCatalogTable_targetTable
=== RUN   TestAccGlueCatalogTable_disappears
=== PAUSE TestAccGlueCatalogTable_disappears
=== RUN   TestAccGlueCatalogTable_openTableFormat
=== PAUSE TestAccGlueCatalogTable_openTableFormat
=== CONT  TestAccGlueCatalogTable_basic
=== CONT  TestAccGlueCatalogTable_StorageDescriptor_schemaReference
    catalog_table_test.go:467: Step 1/3 error: Error running apply: exit status 1
        
        Error: creating Glue Catalog Table (tf-acc-test-4894710199840325208): InvalidInputException: Lake Formation does not support tables with schemas from Glue Schema Registry
        {
          RespMetadata: {
            StatusCode: 400,
            RequestID: "a5ad7bc5-98ad-45b7-8640-582157c5bb66"
          },
          Message_: "Lake Formation does not support tables with schemas from Glue Schema Registry"
        }
        
          with aws_glue_catalog_table.test,
          on terraform_plugin_test.tf line 18, in resource "aws_glue_catalog_table" "test":
          18: resource "aws_glue_catalog_table" "test" {
        
--- FAIL: TestAccGlueCatalogTable_StorageDescriptor_schemaReference (14.84s)
=== CONT  TestAccGlueCatalogTable_StorageDescriptor_emptyBlock
--- PASS: TestAccGlueCatalogTable_basic (27.70s)
=== CONT  TestAccGlueCatalogTable_StorageDescriptorSkewedInfo_emptyBlock
--- PASS: TestAccGlueCatalogTable_StorageDescriptor_emptyBlock (22.97s)
=== CONT  TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_updateValues
--- PASS: TestAccGlueCatalogTable_StorageDescriptorSkewedInfo_emptyBlock (23.60s)
=== CONT  TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_emptyBlock
--- PASS: TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_emptyBlock (24.31s)
=== CONT  TestAccGlueCatalogTable_Disappears_database
--- PASS: TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_updateValues (48.50s)
=== CONT  TestAccGlueCatalogTable_openTableFormat
--- PASS: TestAccGlueCatalogTable_Disappears_database (24.84s)
=== CONT  TestAccGlueCatalogTable_disappears
--- PASS: TestAccGlueCatalogTable_disappears (23.14s)
=== CONT  TestAccGlueCatalogTable_targetTable
--- PASS: TestAccGlueCatalogTable_openTableFormat (52.38s)
=== CONT  TestAccGlueCatalogTable_Update_addValues
--- PASS: TestAccGlueCatalogTable_targetTable (28.43s)
=== CONT  TestAccGlueCatalogTable_Update_replaceValues
--- PASS: TestAccGlueCatalogTable_Update_addValues (43.56s)
=== CONT  TestAccGlueCatalogTable_full
--- PASS: TestAccGlueCatalogTable_Update_replaceValues (44.51s)
=== CONT  TestAccGlueCatalogTable_partitionIndexesSingle
--- PASS: TestAccGlueCatalogTable_full (28.63s)
=== CONT  TestAccGlueCatalogTable_partitionIndexesMultiple
--- PASS: TestAccGlueCatalogTable_partitionIndexesSingle (29.03s)
=== CONT  TestAccGlueCatalogTable_columnParameters
--- PASS: TestAccGlueCatalogTable_partitionIndexesMultiple (29.20s)
=== CONT  TestAccGlueCatalogTable_StorageDescriptor_schemaReferenceARN
    catalog_table_test.go:510: Step 1/2 error: Error running apply: exit status 1
        
        Error: creating Glue Catalog Table (tf-acc-test-4045031736026737163): InvalidInputException: Lake Formation does not support tables with schemas from Glue Schema Registry
        {
          RespMetadata: {
            StatusCode: 400,
            RequestID: "7f2df792-6904-4b79-b9e8-bb70087443cb"
          },
          Message_: "Lake Formation does not support tables with schemas from Glue Schema Registry"
        }
        
          with aws_glue_catalog_table.test,
          on terraform_plugin_test.tf line 18, in resource "aws_glue_catalog_table" "test":
          18: resource "aws_glue_catalog_table" "test" {
        
--- PASS: TestAccGlueCatalogTable_columnParameters (29.89s)
--- FAIL: TestAccGlueCatalogTable_StorageDescriptor_schemaReferenceARN (15.98s)
FAIL
FAIL	github.com/hashicorp/terraform-provider-aws/internal/service/glue	261.204s
FAIL
make: *** [testacc] Error 1

Failures are unrelated to this change.

…berg table.

github-actions · 2023-09-08T16:02:37Z

Community Note

Voting for Prioritization

Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

jar-b

LGTM 👍

% make testacc TESTARGS='-run=TestAccGlueCatalogTable_' PKG=glue
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/glue/... -v -count 1 -parallel 20  -run=TestAccGlueCatalogTable_ -timeout 180m

=== NAME  TestAccGlueCatalogTable_StorageDescriptor_schemaReference
    catalog_table_test.go:467: Step 1/3 error: Error running apply: exit status 1

        Error: creating Glue Catalog Table (tf-acc-test-330412134046413049): InvalidInputException: Lake Formation does not support tables with schemas from Glue Schema Registry
        {
          RespMetadata: {
            StatusCode: 400,
            RequestID: "fa6748e9-f718-46ff-bc7c-36a2408612e6"
          },
          Message_: "Lake Formation does not support tables with schemas from Glue Schema Registry"
        }

          with aws_glue_catalog_table.test,
          on terraform_plugin_test.tf line 18, in resource "aws_glue_catalog_table" "test":
          18: resource "aws_glue_catalog_table" "test" {

=== NAME  TestAccGlueCatalogTable_StorageDescriptor_schemaReferenceARN
    catalog_table_test.go:510: Step 1/2 error: Error running apply: exit status 1

        Error: creating Glue Catalog Table (tf-acc-test-5724533310826662585): InvalidInputException: Lake Formation does not support tables with schemas from Glue Schema Registry
        {
          RespMetadata: {
            StatusCode: 400,
            RequestID: "3d7416f1-8c10-4211-a953-27926c4aab93"
          },
          Message_: "Lake Formation does not support tables with schemas from Glue Schema Registry"
        }

          with aws_glue_catalog_table.test,
          on terraform_plugin_test.tf line 18, in resource "aws_glue_catalog_table" "test":
          18: resource "aws_glue_catalog_table" "test" {

--- FAIL: TestAccGlueCatalogTable_StorageDescriptor_schemaReference (29.31s)
--- FAIL: TestAccGlueCatalogTable_StorageDescriptor_schemaReferenceARN (29.72s)
--- PASS: TestAccGlueCatalogTable_Disappears_database (40.69s)
--- PASS: TestAccGlueCatalogTable_StorageDescriptorSkewedInfo_emptyBlock (43.41s)
--- PASS: TestAccGlueCatalogTable_StorageDescriptor_emptyBlock (45.21s)
--- PASS: TestAccGlueCatalogTable_disappears (46.16s)
--- PASS: TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_emptyBlock (47.04s)
--- PASS: TestAccGlueCatalogTable_basic (49.57s)
--- PASS: TestAccGlueCatalogTable_partitionIndexesMultiple (50.08s)
--- PASS: TestAccGlueCatalogTable_partitionIndexesSingle (50.10s)
--- PASS: TestAccGlueCatalogTable_full (50.46s)
--- PASS: TestAccGlueCatalogTable_columnParameters (50.52s)
--- PASS: TestAccGlueCatalogTable_targetTable (51.05s)
--- PASS: TestAccGlueCatalogTable_StorageDescriptorSerDeInfo_updateValues (63.77s)
--- PASS: TestAccGlueCatalogTable_Update_addValues (65.11s)
--- PASS: TestAccGlueCatalogTable_Update_replaceValues (65.16s)
--- PASS: TestAccGlueCatalogTable_openTableFormat (66.97s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-aws/internal/service/glue       70.261s

github-actions · 2023-09-11T12:13:45Z

This functionality has been released in v5.16.1 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2023-10-12T02:02:57Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

ewbankkit added 2 commits September 8, 2023 11:59

r/aws_glue_catalog_table: Fix removal of Parameters when updating Ice…

acefaf2

…berg table.

Add CHANGELOG entry.

17c7d55

terraform-aws-provider bot added the prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. label Sep 8, 2023

ewbankkit added 2 commits September 8, 2023 12:03

Correct CHANGELOG entry file name.

2c5eb6d

Tweak 'TestAccGlueCatalogTable_openTableFormat'.

0c372c8

github-actions bot added tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. service/glue Issues and PRs that pertain to the glue service. labels Sep 8, 2023

jar-b approved these changes Sep 8, 2023

View reviewed changes

ewbankkit merged commit 41bb39a into main Sep 8, 2023
52 checks passed

ewbankkit deleted the r-aws_glue_catalog_table-parameters-Iceberg branch September 8, 2023 17:31

github-actions bot added this to the v5.17.0 milestone Sep 8, 2023

ewbankkit modified the milestones: v5.17.0, v5.16.1 Sep 8, 2023

github-actions bot locked as resolved and limited conversation to collaborators Oct 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

r/aws_glue_catalog_table: Fix removal of Parameters when updating Iceberg table #33374

r/aws_glue_catalog_table: Fix removal of Parameters when updating Iceberg table #33374

ewbankkit commented Sep 8, 2023 •

edited

Loading

github-actions bot commented Sep 8, 2023

jar-b left a comment

github-actions bot commented Sep 11, 2023

github-actions bot commented Oct 12, 2023

r/aws_glue_catalog_table: Fix removal of Parameters when updating Iceberg table #33374

r/aws_glue_catalog_table: Fix removal of Parameters when updating Iceberg table #33374

Conversation

ewbankkit commented Sep 8, 2023 • edited Loading

Description

Relations

Output from Acceptance Testing

Before

After

github-actions bot commented Sep 8, 2023

Community Note

jar-b left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 11, 2023

github-actions bot commented Oct 12, 2023

ewbankkit commented Sep 8, 2023 •

edited

Loading