AWS Classic OpenSearch Serverless Collection shows update when none have been made on pulumi up.

[martinpsz]

What happened?

I am experiencing an issue, when using OpenSearch Serverless Collection in AWS Classic, where I cannot run pulumi up without the preview showing updates to the resources I am using with the serverless collection. It shows updates even if none have been made.

Example

This code block below is my typical usage. when i run pulumi up, initially, the resources are created. Then, when I work on other resources and run pulumi up, updates shows up for the resources below. I include the pulumi preview --diff below the code block to lend insight.

const openSearchServerlessEncryptionPolicy = new aws.opensearch.ServerlessSecurityPolicy("encryption-policy", {
    type: "encryption",
    description: "encryption for member search data",
    policy: JSON.stringify({
        Rules: [{
            Resource: ["collection/member-search*"],
            ResourceType: "collection",
        }],
        AWSOwnedKey: true,
    }),
}, {provider: stack_acct});

const openSearchServerlessNetworkPolicy = new aws.opensearch.ServerlessSecurityPolicy("network-policy", {
    type: "network",
    description: "network policy for endpoint access",
    policy: JSON.stringify([
        {
            Rules: [
                {
                    ResourceType: "collection",
                    Resource: ["collection/member-search*"]
                }, 
                {
                    ResourceType: "dashboard",
                    Resource: ["collection/member-search*"]
                }
            ],
            AllowFromPublic: true,
        }
    ])
}, {provider: stack_acct})

const current = aws.getCallerIdentity({});
const openSearchServerlessAccessPolicy = new aws.opensearch.ServerlessAccessPolicy('open-search-access', {
    type: 'data',
    policy: current.arn.apply(current => JSON.stringify([{
        Rules: [
            {
                ResourceType: "index",
                Resource: ["index/member-search/*"],
                Permission: [
                    "aoss:ReadDocument",
                    "aoss:WriteDocument",
                    "aoss:CreateIndex",
                    "aoss:DeleteIndex",
                    "aoss:UpdateIndex",
                    "aoss:DescribeIndex"
                ]
            },
            {
                ResourceType: "collection",
                Resource: ["collection/member-search*"],
                Permission: [
                    "aoss:CreateCollectionItems",
                    "aoss:DeleteCollectionItems",
                    "aoss:UpdateCollectionItems",
                    "aoss:DescribeCollectionItems"
                ]
            }
        ],
        Principal: [current.arn]
    }]))
}, {provider: stack_acct});

const memberSearchService = new aws.opensearch.ServerlessCollection("member-search", {
    type: 'SEARCH', 
    tags,
}, {dependsOn: [openSearchServerlessEncryptionPolicy], provider: stack_acct})

pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:member-search-api::member-search-api::pulumi:pulumi:Stack::member-search-api-member-search-api]
    > pulumi:pulumi:StackReference: (read)
        [id=event-distrib-prod]
        [urn=urn:pulumi:member-search-api::member-search-api::pulumi:pulumi:StackReference::event-distrib-prod]
        name: "event-distrib-prod"
    ~ aws:opensearch/serverlessSecurityPolicy:ServerlessSecurityPolicy: (update)
        [id=network-policy-d108f6c]
        [urn=urn:pulumi:member-search-api::member-search-api::aws:opensearch/serverlessSecurityPolicy:ServerlessSecurityPolicy::network-policy]
        [provider=urn:pulumi:member-search-api::member-search-api::pulumi:providers:aws::stack::9cde4391-403e-49a5-8053-8417073f8c40]
        description: "network policy for endpoint access"
        name       : "network-policy-d108f6c"
        policy     : (json) [
            [0]: {
                AllowFromPublic: true
                Rules          : [
                    [0]: {
                        Resource    : [
                            [0]: "collection/member-search*"
                        ]
                        ResourceType: "collection"
                    }
                    [1]: {
                        Resource    : [
                            [0]: "collection/member-search*"
                        ]
                        ResourceType: "dashboard"
                    }
                ]
            }
        ]

        type       : "network"
    ~ aws:opensearch/serverlessSecurityPolicy:ServerlessSecurityPolicy: (update)
        [id=encryption-policy-9b616ba]
        [urn=urn:pulumi:member-search-api::member-search-api::aws:opensearch/serverlessSecurityPolicy:ServerlessSecurityPolicy::encryption-policy]
        [provider=urn:pulumi:member-search-api::member-search-api::pulumi:providers:aws::stack::9cde4391-403e-49a5-8053-8417073f8c40]
        description: "encryption for member search data"
        name       : "encryption-policy-9b616ba"
        policy     : (json) {
            AWSOwnedKey: true
            Rules      : [
                [0]: {
                    Resource    : [
                        [0]: "collection/member-search*"
                    ]
                    ResourceType: "collection"
                }
            ]
        }

        type       : "encryption"
    ~ aws:opensearch/serverlessAccessPolicy:ServerlessAccessPolicy: (update)
        [id=open-search-access-6806895]
        [urn=urn:pulumi:member-search-api::member-search-api::aws:opensearch/serverlessAccessPolicy:ServerlessAccessPolicy::open-search-access]
        [provider=urn:pulumi:member-search-api::member-search-api::pulumi:providers:aws::stack::9cde4391-403e-49a5-8053-8417073f8c40]
        name  : "open-search-access-6806895"
        policy: (json) [
            [0]: {
                Principal: [
                    [0]: "arn:aws:iam::941966628158:role/member-search-api-pipeline-role-0850f6a"
                ]
                Rules    : [
                    [0]: {
                        Permission  : [
                            [0]: "aoss:ReadDocument"
                            [1]: "aoss:WriteDocument"
                            [2]: "aoss:CreateIndex"
                            [3]: "aoss:DeleteIndex"
                            [4]: "aoss:UpdateIndex"
                            [5]: "aoss:DescribeIndex"
                        ]
                        Resource    : [
                            [0]: "index/member-search/*"
                        ]
                        ResourceType: "index"
                    }
                    [1]: {
                        Permission  : [
                            [0]: "aoss:CreateCollectionItems"
                            [1]: "aoss:DeleteCollectionItems"
                            [2]: "aoss:UpdateCollectionItems"
                            [3]: "aoss:DescribeCollectionItems"
                        ]
                        Resource    : [
                            [0]: "collection/member-search*"
                        ]
                        ResourceType: "collection"
                    }
                ]
            }
        ]

        type  : "data"
Resources:              
    ~ 3 to update

Output of pulumi about

Dependencies:
NAME VERSION
@pulumi/aws-native 0.94.0
@pulumi/aws 6.18.2
@pulumi/awsx 2.4.0
@pulumi/pulumi 3.102.0
@types/node 18.19.9

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[justinvp]

This looks like it's likely an issue in the AWS provider, so moving to the AWS provider repo for further triage.

[iwahbe]

Hi @martinpsz. Thanks for reporting this issue.

I was able to reproduce with the following program:

import * as aws from "@pulumi/aws";

new aws.opensearch.ServerlessSecurityPolicy("encryption-policy", {
    type: "encryption",
    policy: JSON.stringify({
        Rules: [{
            Resource: ["collection/member-search*"],
            ResourceType: "collection",
        }],
        AWSOwnedKey: true,
    }),
});

[ViktorCollin]

This is making Pulumi failed on update with the following error message. We had to add the current date to the description in order to not fail our CI build for all unrelated changes.

aws:opensearch:ServerlessSecurityPolicy (***-encryption-policy):
    error: updating Security Policy (***-encryption-policy): operation error OpenSearchServerless: UpdateSecurityPolicy, https response error StatusCode: 400, RequestID: *****, ValidationException: No changes detected in policy or policy description

[corymhall]

It looks like the diff might be caused by the ordering of the keys in the policy JSON document. Can you try to order the keys alphabetically and see if the diff goes away?

We are still investigating what is causing this ordering issue, but it may be a workaround in the meantime.

[corymhall]

It looks like this is technically an upstream issue.

If you are interested in the details

Even with Terraform if you try and create a security_policy using a string you will get an error message if the keys are not ordered.

resource "aws_opensearchserverless_security_policy" "example" {
  name   = "encryption-policy-023672f"
  type   = "encryption"
  policy = "{\"Rules\":[{\"Resource\":[\"collection/member-search*\"],\"ResourceType\":\"collection\"}],\"AWSOwnedKey\":true}"
}

aws_opensearchserverless_security_policy.example: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to aws_opensearchserverless_security_policy.example, provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value: .policy: was
│ cty.StringVal("{\"Rules\":[{\"Resource\":[\"collection/member-search*\"],\"ResourceType\":\"collection\"}],\"AWSOwnedKey\":true}"), but now
│ cty.StringVal("{\"AWSOwnedKey\":true,\"Rules\":[{\"Resource\":[\"collection/member-search*\"],\"ResourceType\":\"collection\"}]}").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

This is because the way that Terraform parses the json response from the CreateSecurityPolicy API call and stores the policy in state with ordered keys.

https://github.com/hashicorp/terraform-provider-aws/blob/961823a2be27dc1c943f8721962d189a0b4a8ca2/internal/service/opensearchserverless/security_policy.go#L264

The difference is that in Terraform you can use the jsonencode utility which produces a json object with ordered keys. In pulumi we rely on the language utilities like JSON.stringify() which does not order keys.

I think we may be able to fix this on our side by using a PreCheckCallback to order the keys before passing to create.

[t0yv0]

It is common to have JSON-valued attributes that need to ignore whitespace and reordering changes. The common way to handle it in the AWS provider for SDKv2 based resources is using something like this:

DiffSuppressFunc: verify.SuppressEquivalentJSONDiffs,
StateFunc: func(v interface{}) string {
    json, _ := structure.NormalizeJsonString(v)
    return json
},

For example, consider: https://github.com/hashicorp/terraform-provider-aws/blob/master/internal/service/networkmanager/core_network_policy_attachment.go#L57

I've searched briefly but couldn't find authoritative guidance on how to accomplish the same result for Plugin Framework based resources. It appears there was an idea to use custom types that redefine the equality operation (hashicorp/terraform-plugin-framework#803) that led to the creation of https://github.com/hashicorp/terraform-plugin-framework-jsontypes repository.

Looking further, there seems to be a precedent of using this:

https://github.com/hashicorp/terraform-provider-aws/blob/master/internal/service/opensearchserverless/access_policy.go#L42

[t0yv0]

Unlike access_policy, security_policy uses the String type and not jsontypes.Normalized:

Policy        jsontypes.Normalized                          `tfsdk:"policy"

This makes me suspect that:

1. the upstream provider may not be handling changes to security_policy in the desired way (that is, it may not be ignoring whitespace and reordering changes as the users might expect)

2. a possible fix could be editing upstream to transition to jsontypes.Normalized.

[corymhall]

Created an issue upstream hashicorp/terraform-provider-aws#38603

[corymhall]

My fix was merged upstream so we should be able to pull it in next release.

[corymhall]

v6.48.0 was just released and included the fix.