Nested TypeSet structures in avi_cloud resource always report as changed

[bartsqueezy]

Issue Description

The Avi provider is incorrectly reporting changes to avi_cloud resources when a nested TypeSet is defined in TF config files.

Steps to Reproduce

1. Create a Kubernetes cloud in the Avi Vantage controller.
2. Initialize the TF with the Avi provider and an (optional) remote backend

   terraform {
     backend "consul" {
       address = var.consul_address
       scheme = var.consul_scheme
       path = var.consul_path
     }
   }
   
   provider "avi" {
     avi_controller = var.avi_controller
     avi_tenant = "admin"
     avi_version = var.avi_version
   }

3. Using valid Avi credentials, import the existing Kubernetes cloud into TF state

   AVI_USERNAME=user AVI_PASSWORD=pass terraform import avi_cloud.default cloud-795da0ff-6758-42ce-b857-5989011c5bdc

4. Update TF config make a few changes to the newly imported Kubernetes cloud.

   resource "avi_cloud" "default" {
     name = "default"
     dhcp_enabled = true
     vtype = "CLOUD_OSHIFT_K8S"
   
     oshiftk8s_configuration {
       disable_auto_backend_service_sync = true
       master_nodes = [var.avi_master_node]
       service_account_token = "secret-key"
   
       se_include_attributes {
         attribute = "avi-se"
         value = "true"
       }
     }
   }

5. Apply the changes to the avi_cloud resource.

   AVI_USERNAME=user AVI_PASSWORD=pass terraform apply

6. After successful apply, immediately execute another plan without making code changes.

   AVI_USERNAME=user AVI_PASSWORD=pass terraform plan

Expected Result

Terraform should report no changes. Console output should be...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Actual Result

Terraform reports changes to the entire oshiftk8s_configuration dict.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # avi_cloud.default will be updated in-place
  ~ resource "avi_cloud" "default" {
        apic_mode                    = false
        autoscale_polling_interval   = 60
        dhcp_enabled                 = true
        dns_provider_ref             = "https://avi.local/api/ipamdnsproviderprofile/ipamdnsproviderprofile-e3db518d-274a-4df9-88e4-9c9ac189f23f"
        dns_resolution_on_se         = false
        enable_vip_static_routes     = false
        id                           = "https://avi.local/api/cloud/cloud-795da0ff-6758-42ce-b857-5989011c5bdc"
        ip6_autocfg_enabled          = false
        ipam_provider_ref            = "https://avi.local/api/ipamdnsproviderprofile/ipamdnsproviderprofile-9a15c2ef-adc6-45ef-8625-713770b46638"
        license_tier                 = "ENTERPRISE_18"
        license_type                 = "LIC_CORES"
        mtu                          = 1500
        name                         = "default"
        prefer_static_routes         = false
        state_based_dns_registration = true
        tenant_ref                   = "https://avi.local/api/tenant/admin"
        uuid                         = "cloud-795da0ff-6758-42ce-b857-5989011c5bdc"
        vtype                        = "CLOUD_OSHIFT_K8S"

      - oshiftk8s_configuration {
          - app_sync_frequency                   = 300 -> null
          - auto_assign_fqdn                     = true -> null
          - container_port_match_http_service    = true -> null
          - coredump_directory                   = "/var/lib/systemd/coredump" -> null
          - default_service_as_east_west_service = true -> null
          - disable_auto_backend_service_sync    = true -> null
          - disable_auto_frontend_service_sync   = false -> null
          - disable_auto_gs_sync                 = false -> null
          - disable_auto_se_creation             = false -> null
          - docker_endpoint                      = "/var/run/docker.sock" -> null
          - enable_event_subscription            = true -> null
          - feproxy_vips_enable_proxy_arp        = true -> null
          - http_container_ports                 = [] -> null
          - l4_health_monitoring                 = true -> null
          - master_nodes                         = [
              - "https://default.k8s.local",
            ] -> null
          - num_shards                           = 0 -> null
          - override_service_ports               = true -> null
          - sdn_overlay                          = true -> null
          - se_deployment_method                 = "SE_CREATE_POD" -> null
          - se_restart_batch_size                = 1 -> null
          - se_restart_force                     = false -> null
          - se_volume                            = "/var/lib/avi" -> null
          - secure_egress_mode                   = false -> null
          - service_account_token                = "<sensitive>" -> null
          - shared_virtualservice_namespace      = false -> null
          - sync_not_ready_addresses             = true -> null
          - use_controller_image                 = false -> null
          - use_resource_definition_as_ssot      = false -> null
          - use_scheduling_disabled_nodes        = false -> null
          - use_service_cluster_ip_as_ew_vip     = false -> null

          - avi_bridge_subnet {
              - mask = 16 -> null

              - ip_addr {
                  - addr = "172.18.0.1" -> null
                  - type = "V4" -> null
                }
            }

          - docker_registry_se {
              - private  = false -> null
              - registry = "avinetworks/se" -> null
            }

          - east_west_placement_subnet {
              - mask = 16 -> null

              - ip_addr {
                  - addr = "172.18.0.1" -> null
                  - type = "V4" -> null
                }
            }

          - se_include_attributes {
              - attribute = "avi-se" -> null
              - value     = "true" -> null
            }

          - se_pod_tolerations {
              - effect             = "NO_SCHEDULE" -> null
              - key                = "dedicated" -> null
              - operator           = "EQUAL" -> null
              - toleration_seconds = 0 -> null
              - value              = "se" -> null
            }

          - vip_default_gateway {
              - addr = "10.228.72.1" -> null
              - type = "V4" -> null
            }
        }
      + oshiftk8s_configuration {
          + app_sync_frequency                   = 300
          + auto_assign_fqdn                     = true
          + ca_tls_key_and_certificate_ref       = (known after apply)
          + client_tls_key_and_certificate_ref   = (known after apply)
          + cluster_tag                          = (known after apply)
          + container_port_match_http_service    = true
          + coredump_directory                   = "/var/lib/systemd/coredump"
          + default_service_as_east_west_service = true
          + disable_auto_backend_service_sync    = true
          + disable_auto_frontend_service_sync   = false
          + disable_auto_gs_sync                 = false
          + disable_auto_se_creation             = false
          + docker_endpoint                      = "/var/run/docker.sock"
          + enable_event_subscription            = true
          + feproxy_vips_enable_proxy_arp        = true
          + http_container_ports                 = []
          + l4_health_monitoring                 = true
          + master_nodes                         = [
              + "https://default.k8s.local",
            ]
          + node_availability_zone_label         = (known after apply)
          + num_shards                           = 0
          + override_service_ports               = true
          + sdn_overlay                          = true
          + se_deployment_method                 = "SE_CREATE_POD"
          + se_image_pull_secret                 = (known after apply)
          + se_priority_class                    = (known after apply)
          + se_restart_batch_size                = 1
          + se_restart_force                     = false
          + se_volume                            = "/var/lib/avi"
          + secure_egress_mode                   = false
          + service_account_token                = "secret-key"
          + shard_prefix                         = (known after apply)
          + shared_virtualservice_namespace      = false
          + ssh_user_ref                         = (known after apply)
          + sync_not_ready_addresses             = true
          + use_controller_image                 = false
          + use_resource_definition_as_ssot      = false
          + use_scheduling_disabled_nodes        = false
          + use_service_cluster_ip_as_ew_vip     = false

          + avi_bridge_subnet {
              + mask = (known after apply)

              + ip_addr {
                  + addr = (known after apply)
                  + type = (known after apply)
                }
            }

          + docker_registry_se {
              + password = (known after apply)
              + private  = (known after apply)
              + registry = (known after apply)
              + username = (known after apply)

              + oshift_registry {
                  + registry_namespace = (known after apply)
                  + registry_service   = (known after apply)

                  + registry_vip {
                      + addr = (known after apply)
                      + type = (known after apply)
                    }
                }
            }

          + east_west_placement_subnet {
              + mask = (known after apply)

              + ip_addr {
                  + addr = (known after apply)
                  + type = (known after apply)
                }
            }

          + se_include_attributes {
              + attribute = "avi-se"
              + value     = "true"
            }

          + vip_default_gateway {
              + addr = (known after apply)
              + type = (known after apply)
            }
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Analysis

As you can see, the Avi provider is not doing a proper diff of the nested oshiftk8s_configuration resource. After some research, it appears the issue is related to how the schema for this resource is defined. oshiftk8s_configuration is defined as TypeSet and includes other nested TypeSet attributes, such as avi_bridge_subnet, docker_registry_se, etc. Based on this comment, it appears the usage of nested TypeSet resources is a limitation of the current version of the Terraform SDK. In order to preserve the current config structure, it is recommended to use TypeList with MaxItems: 1 instead of TypeSet. I tried making this change locally, however, I ran into type conversion errors...

panic: interface conversion: interface {} is map[string]interface {}, not []interface {}

This error is thrown within SetDefaultsInAPIRes method in utils.go.

Considering this is common behavior in managing Avi resources via Terraform, I believe this is a critical bug.

[chaitanyaavi]

@bartsqueezy This is happening because of there are sensitive fields in the cloud configuration like password. Avi controller never returns the password in response. Hence in cloud configuration or other objects which has sensitive fields it always shows a change.

[bartsqueezy]

Hi @chaitanyaavi, thanks for the feedback. I completely understand the Avi API wanting to exclude sensitive information from response data. However, a new terraform plan should only show changes to those specific fields, not mark the entire oshiftk8s_configuration block as changed. This is most likely an issue with the way API response are converted to TF schemas. If that conversion was implemented based on the links I previously referenced then it should only show sensitive data fields as changed.

[chaitanyaavi]

@bartsqueezy The diff is showing the full nested object change is because of following issue on Terraform which is still open. hashicorp/terraform#21901

[yograjshisode]

The fix is available from version 20.1.7 provider