[Bug]: QuickSight dataset error: invalid transform in LogicalTableMap

[muratmozdemir]

Terraform Core Version

1.4.6

AWS Provider Version

5.3.0

Affected Resource(s)

• aws.service=QuickSight
• aws.operation=CreateDataSet

Expected Behavior

The dataset should be created successfully. I have viewed the HTTP request terraform makes and it appears that the JSON generated by Terraform is not formatted correctly, specifically within DataTransforms section.

I have enabled trace logs in Terraform and here is part of the JSON that is being sent in the HTTP request that fails (This also fails when I run this from Python (boto3). Below is what Terraform generates for LogicalTableMap

Terraform Generated LogicalTableMap

{
    "4559aa95-4a54-4998-800b-5fd03af260ea": {
        "Alias": "account_map",
        "Source": {
            "PhysicalTableId": "516bd20f-660c-49e0-95d3-0b5d21ef39f7"
        }
    },
    "6beb277c-4d74-47da-b425-1e255aa31be6": {
        "Alias": "Intermediate Table",
        "DataTransforms": [
            {
                "ProjectOperation": {
                    "ProjectedColumns": [
                        "year",
                        "month",
                        "billing_period",
                        "usage_date",
                        "payer_account_id",
                        "linked_account_id",
                        "resource_id",
                        "product_code",
                        "operation",
                        "region",
                        "charge_type",
                        "pricing_unit",
                        "usage_quantity",
                        "unblended_cost",
                        "public_cost",
                        "account_id",
                        "account_name"
                    ]
                },
                "TagColumnOperation": {
                    "ColumnName": "region",
                    "Tags": [
                        {
                            "ColumnGeographicRole": "STATE"
                        }
                    ]
                }
            }
        ],
        "Source": {
            "JoinInstruction": {
                "LeftOperand": "ea0d1482-1bf2-46ee-8dc4-5aaea2ab2209",
                "OnClause": "{linked_account_id} = {account_id}",
                "RightOperand": "4559aa95-4a54-4998-800b-5fd03af260ea",
                "Type": "LEFT"
            }
        }
    },
    "ea0d1482-1bf2-46ee-8dc4-5aaea2ab2209": {
        "Alias": "s3_view",
        "Source": {
            "PhysicalTableId": "7bb99428-5dfd-4599-8893-b4f57f0c689f"
        }
    }
}

If I use this JSON in boto3 however, the dataset is created successfully:

Correct JSON for LogicalTableMap

{
    "4559aa95-4a54-4998-800b-5fd03af260ea": {
        "Alias": "account_map",
        "Source": {
            "PhysicalTableId": "516bd20f-660c-49e0-95d3-0b5d21ef39f7"
        }
    },
    "6beb277c-4d74-47da-b425-1e255aa31be6": {
        "Alias": "Intermediate Table",
        "DataTransforms": [
            {
                "ProjectOperation": {
                    "ProjectedColumns": [
                        "year",
                        "month",
                        "billing_period",
                        "usage_date",
                        "payer_account_id",
                        "linked_account_id",
                        "resource_id",
                        "product_code",
                        "operation",
                        "region",
                        "charge_type",
                        "pricing_unit",
                        "usage_quantity",
                        "unblended_cost",
                        "public_cost",
                        "account_id",
                        "account_name"
                    ]
                }
            },
            {
                "TagColumnOperation": {
                    "ColumnName": "region",
                    "Tags": [
                        {
                            "ColumnGeographicRole": "STATE"
                        }
                    ]
                }
            }
        ],
        "Source": {
            "JoinInstruction": {
                "LeftOperand": "ea0d1482-1bf2-46ee-8dc4-5aaea2ab2209",
                "RightOperand": "4559aa95-4a54-4998-800b-5fd03af260ea",
                "Type": "LEFT",
                "OnClause": "{linked_account_id} = {account_id}"
            }
        }
    },
    "ea0d1482-1bf2-46ee-8dc4-5aaea2ab2209": {
        "Alias": "s3_view",
        "Source": {
            "PhysicalTableId": "7bb99428-5dfd-4599-8893-b4f57f0c689f"
        }
    }
}

• In the first JSON, the "ProjectOperation" and "TagColumnOperation" are contained within a single dictionary in the "DataTransforms" array.

• In the second JSON, the "ProjectOperation" and "TagColumnOperation" are separate dictionaries within the "DataTransforms" array.

Actual Behavior

Error: creating QuickSight Data Set: InvalidParameterValueException: Invalid transform in LogicalTableMap

Relevant Error/Panic Output Snippet

Message_: "Invalid transform in LogicalTableMap"
}" tf_resource_type=aws_quicksight_data_set @module=sdk.proto timestamp=2023-06-14T21:31:59.524Z
2023-06-14T21:31:59.525Z [TRACE] provider.terraform-provider-aws_v5.3.0_x5: Served request: tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_quicksight_data_set @caller=github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov5/tf5server/server.go:831 @module=sdk.proto tf_rpc=ApplyResourceChange tf_proto_version=5.3 tf_req_id=2e65eb2e-6624-33cb-4494-dd949c2001e5 timestamp=2023-06-14T21:31:59.524Z
2023-06-14T21:31:59.526Z [TRACE] maybeTainted: module.environments.module.sandbox.aws_quicksight_data_set.cid_dataset_s3_view encountered an error during creation, so it is now marked as tainted
2023-06-14T21:31:59.526Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.environments.module.sandbox.aws_quicksight_data_set.cid_dataset_s3_view
2023-06-14T21:31:59.526Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.environments.module.sandbox.aws_quicksight_data_set.cid_dataset_s3_view
2023-06-14T21:31:59.527Z [TRACE] evalApplyProvisioners: module.environments.module.sandbox.aws_quicksight_data_set.cid_dataset_s3_view is tainted, so skipping provisioning
2023-06-14T21:31:59.527Z [TRACE] maybeTainted: module.environments.module.sandbox.aws_quicksight_data_set.cid_dataset_s3_view was already tainted, so nothing to do
2023-06-14T21:31:59.527Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.environments.module.sandbox.aws_quicksight_data_set.cid_dataset_s3_view
2023-06-14T21:31:59.527Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.environments.module.sandbox.aws_quicksight_data_set.cid_dataset_s3_view
2023-06-14T21:31:59.528Z [ERROR] vertex "module.environments.module.sandbox.aws_quicksight_data_set.cid_dataset_s3_view" error: creating QuickSight Data Set: InvalidParameterValueException: Invalid transform in LogicalTableMap
{
RespMetadata: {
StatusCode: 400,
RequestID: "f1871995-b69a-4835-83e0-0a68198485e1"
},
Message_: "Invalid transform in LogicalTableMap"
}

Terraform Configuration Files

I have created an example repository which you can reproduce the bug here:

https://github.com/muratmozdemir/aws-quicksight-example

Steps to Reproduce

Here are some reproduction steps.

QuickSight Configuration

• Create a QuickSight Enterprise subscription and use the default settings.
  • Since Terraform will be creating our S3 buckets, we don't have to check Amazon S3 from the Allow access and autodiscovery for these resources option in the QuickSight signup page.
  • Unfortunately, it appears that you cannot modify the IAM role for a QuickSight subscription via Terraform or the API, so this is why the previous step is a bit manual.
• Once QuickSight is activated, modify the aws-quicksight-service-role-v0 IAM role. You can do this in the QuickSight settings but since the buckets do not exist yet, we can modify the role's policy directly.
• Create a new inline policy called AWSQuickSightS3Policy
  • You can find an example S3 IAM policy in the quicksight-example repository under resources/iam just be sure to replace <account-id> with the account ID that is hosting QuickSight in the policy document.

Terraform Configuration

• Now that we have QuickSight configured, we can run Terraform in our aws-quicksight-example repository. Go ahead and clone the repository.
• If using Terraform Cloud, you can modify the backend.tf file to match your organization's settings.
• Also modify config.tf to fit the needs for your environment. For my environment, I am assuming a role which I pass via the AWS_ASSUME_ROLE_ARN variable.
• If you run a terraform plan everything will look fine. The error happens when running terraform apply. If you look at the Expected Behavior section of this report, I have included the JSON being sent as part of the HTTP request from Terraform.
• You should get the invalid transform in LogicalTableMap error after running terraform apply

How to view the JSON that Terraform sends to AWS

• First you must enable debug or trace logging by setting the TF_LOG environment variable to DEBUG or TRACE
• Run terraform apply and look for the line that starts like this:

[DEBUG] provider.terraform-provider-aws_v5.3.0_x5: HTTP Request Sent: http.request.header.x_amz_date=20230614T204028Z tf_mux_provider=*schema.GRPCProviderServer aws.sdk=aws-sdk-go http.flavor=1.1 http.request.body="{"DataSetId":

• "{"DataSetId": is where the JSON begins you can grab.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[RuanRDM]

Hi Guys,

I faced the same problem and to solve just separate the data_transforms.

Example:

logical_table_map {
    logical_table_map_id = var.table_map_id
    alias                = var.table_name
    source {
      physical_table_id = var.table_map_id
    }
    data_transforms {
      project_operation {
        projected_columns = [
          "column1",
          "column2"
        ]
      }    
    }
    data_transforms {
      create_columns_operation {
        columns {
          column_id = "column-id"
          column_name = "column-name"
          expression = "'exp'"
        }
      }
    }
  }
}