Graph issue when trying to remove a sec list

[cosmindev]

When trying to remove an existing security list that is attached to a subnet, tf fails as it should first to destroy the association between the sec list and subnet and only after the sec_list

To reproduce the issue I've developed the following snippet:

resource "oci_core_security_list" "test_security_list" {
  count = 0
  #Required
  compartment_id = "ocid1.compartment.oc1....."
  vcn_id         = "ocid1.vcn.oc1.eu-frankfurt-1......"

}

resource "oci_core_subnet" "test_subnet" {
  #Required
  cidr_block     = "10.0.85.0/24"
  compartment_id = "ocid1.compartment.oc1....."
  vcn_id         = "ocid1.vcn.oc1....."
  #Optional
  display_name = "test"

  security_list_ids = [for i in [/*default sec_list*/ "ocid1.securitylist.oc1....",
  /*optional sec_list*/ length(oci_core_security_list.test_security_list) > 0 ? oci_core_security_list.test_security_list[0].id : null] : i if i != null]
}

To reproduce the issue:

• Initially I run terraform apply with count = 1 for the security list resource. All is created fine
• After I run terraform apply with count = 0 for the security list resource. This is when it fails as it tries 1st to destroy the sec_list without destroying the sec_list - subnet association first: Error: Service error:IncorrectState. ocid1.securitylist.oc1.**** is associated with subnet security list association that is in use. http status code: 409. Opc request id: 34a5d892c5916271ea60c46182d0e68d/4FC52105B9A278BC880FB11879D095ED/AE3A7B83CABE5F63813640392F95DAB4

The complete error output:

$ terraform apply
oci_core_security_list.test_security_list[0]: Refreshing state... [id=ocid1.securitylist.oc1....]
oci_core_subnet.test_subnet: Refreshing state... [id=ocid1.subnet.oc1.eu-frankfurt-1.....]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
  - destroy

Terraform will perform the following actions:

  # oci_core_security_list.test_security_list[0] will be destroyed
  - resource "oci_core_security_list" "test_security_list" {
      - compartment_id = "ocid1.compartment.oc1......" -> null
      - defined_tags   = {} -> null
      - display_name   = "securitylist20191023085349" -> null
      - freeform_tags  = {} -> null
      - id             = "ocid1.securitylist.oc1....." -> null
      - state          = "AVAILABLE" -> null
      - time_created   = "2019-10-23 08:53:49.087 +0000 UTC" -> null
      - vcn_id         = "ocid1.vcn.oc1.eu-frankfurt-1....." -> null
    }

  # oci_core_subnet.test_subnet will be updated in-place
  ~ resource "oci_core_subnet" "test_subnet" {
        cidr_block                 = "10.0.85.0/24"
        compartment_id             = "ocid1.compartment.oc1......"
        defined_tags               = {}
        dhcp_options_id            = "ocid1.dhcpoptions.oc1.eu-frankfurt-1......."
        display_name               = "test-cotud"
        freeform_tags              = {}
        id                         = "ocid1.subnet.oc1.eu-frankfurt-1....."
        prohibit_public_ip_on_vnic = false
        route_table_id             = "ocid1.routetable.oc1.eu-frankfurt-1....."
      ~ security_list_ids          = [
            "ocid1.securitylist.oc1.eu-frankfurt-1.....",
          - "ocid1.securitylist.oc1.eu-frankfurt-1....",
        ]
        state                      = "AVAILABLE"
        time_created               = "2019-10-23 08:55:22.395 +0000 UTC"
        vcn_id                     = "ocid1.vcn.oc1.eu-frankfurt-1....."
        virtual_router_ip          = "10.0.85.1"
        virtual_router_mac         = "00:00:17:09:7D:B4"
    }

Plan: 0 to add, 1 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

oci_core_security_list.test_security_list[0]: Destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1...]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1....., 10s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1....., 20s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1...., 30s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1...., 40s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1...., 50s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1...., 1m0s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1..., 1m10s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1...., 1m20s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1...., 1m30s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1...., 1m40s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1...., 1m50s elapsed]
oci_core_security_list.test_security_list[0]: Still destroying... [id=ocid1.securitylist.oc1.eu-frankfurt-1..., 2m0s elapsed]

Error: Service error:IncorrectState. ocid1.securitylist.oc1.eu-frankfurt-1.... is associated with subnet security list association that is in use. http status code: 409. Opc request id: 34a5d892c5916271ea60c46182d0e68d/4FC52105B9A278BC880FB11879D095ED/AE3A7B83CABE5F63813640392F95DAB4

[alexng-canuck]

Hi @cosmindev ,

Do you have full debug logs for the repro with TF_LOG=DEBUG and OCI_GO_SDK_DEBUG=v variables enabled?

I wonder if Terraform did an update of the oci_core_subnet first before it tried to delete the oci_core_security_list? The error you’re getting sounds like Terraform didn’t do the update first; and so it’s not able to delete the security list.

[parrneet]

Hi @cosmindev - I tried the scenario and Terraform tries to delete the security list first. Since subnet has a reference to it so ideally subnet should be updated first. Explained here
This seems similar to https://github.com/terraform-providers/terraform-provider-oci/issues/728
and the Terraform limitations are tracked here.

If you are blocked, there is a work around mentioned in this comment.

[cosmindev]

Hi @parrneet,

Just wondering if this is a provider(OCI provider) or a TF issue. With other similar situations, things are working properly. For example, volumes and volume_groups:

resource "oci_core_volume" "test_volume" {
  count = 1
  #Required
  availability_domain = "uFjs:EU-FRANKFURT-1-AD-1"
  compartment_id      = "ocid1.compartment.oc1..aaaaaaaa......"

  #Optional

  display_name     = "test-cotud-${count.index}"

}

resource "oci_core_volume_group" "test_volume_group" {
  #Required
  availability_domain = "uFjs:EU-FRANKFURT-1-AD-1"
  compartment_id      = "ocid1.compartment.oc1..aaa........."
  source_details {
    #Required
    type       = "volumeIds"
    volume_ids = [for vol in oci_core_volume.test_volume : vol.id]
  }

  display_name     = "test-cotud"
}

• Run the above terraform apply with count = 2 for the volumes.
• Update count=1 from the volumes and run terraform apply again.
• You'll notice that first the volume group will be deleted, then the volume will be deleted and, at the end, the volume group will be re-created with a new list of volumes. This is acctually the diference between the 2 usescases: in the sec_list scenario the subnet resource is marked for update whereas in the volume_groups scenario, the volume_groups is maked for delete and re-create.

[cosmindev]

@parrneet
I do not think the proposed 2 step approach workaround is a feasible one. Frequent update of security rules is an extremely frequent use-case. There are lot of production CI/CD pipelines involving sec policy updates. I'll suggest prioritizing this issue.
Thanks

[parrneet]

@cosmindev - This is bug in Terraform code where update/delete on a resource and its dependencies are not handled in a correct order always.