Incorrect order of execution when deleting resource that participates in dependency, together with the dependency

[annakhm]

I feel like this is a very basic issue, but couldn't find similar existing issues. I think this is different that life cycle management of specific resource.
In the configuration sample below, if we remove both passive monitor and passive_monitor_id line from nsxt_lb_pool resource, terraform will first delete the passive monitor resource, and then update the dependency. This fails in provider, since on backend the monitor is being used in pool resource.

Terraform Version

Terraform v0.11.8
+ provider.nsxt (unversioned)

Terraform Configuration Files

resource "nsxt_lb_http_monitor" "test" {
  monitor_port          = 80
}

resource "nsxt_lb_passive_monitor" "test" {
}

resource "nsxt_lb_pool" "test" {
  member {
    ip_address                 = "7.7.7.7"
  }
  active_monitor_id = "${nsxt_lb_http_monitor.test.id}"
  passive_monitor_id = "${nsxt_lb_passive_monitor.test.id}"
}

Expected Behavior

Since terraform is aware of the dependency, I would expect lb_pool to be updated first, and then passive monitor resource to be deleted.

Actual Behavior

Passive monitor is deleted, and then lb_pool is updated.

Steps to Reproduce

1. terraform apply with above config
2. delete passive monitor and reference to it in lb_pool resource
3. terraform apply

[jcarrothers-sap]

I believe this is similar or the same issue as #8617. I have also run into exactly this problem. My understanding is that it is not that terraform is doing things in the reverse order to what we expect, it is that it is trying to complete the update and the delete in parallel.

[apparentlymart]

Hi @annakhm! Sorry for this weird behavior, and thanks for reporting it.

This does indeed seem to be similar to #8617. However, that issue is covering a number of different situations (running terraform destroy, replacing an object in-place without destroying things that depend on it, along with some issues where there are dependencies that Terraform can't automatically "see" from the configuration) so I'd like to dig into this a little more here before deciding whether to consolidate into that larger issue.

The intended behavior for Terraform is that it should record in the state file the dependencies that were present in the configuration when each resource instance is created. This information should then be used when dealing with resources that have been removed from configuration altogether, since without the configuration block there is no record of those dependencies left in the configuration.

From the reproduction steps here (which I'm afraid I haven't been able to run myself yet, since I don't have an nsxt cluster to test with) it sounds like Terraform would've needed to lean on the state record of dependencies in this case, so if you still have available the system where you encountered this it would be helpful to see what dependencies, if any, were recorded in the state after the initial terraform apply. With this being just a test configuration hopefully you'd be able to share the whole state file (e.g. in a gist) but if you'd rather be more specific I'm mainly looking for the dependency array that'd be stored in the state object for nsxt_lb_pool.test, which ought to be set to ["nsxt_lb_http_monitor.test", "nsxt_lb_passive_monitor.test"] for this configuration.

Whether these values are recorded correctly in the state will help to figure out if the problem is in the construction of the state record itself or if it's in the interpretation of those records.

Thanks, and sorry again for this confusing behavior.

[annakhm]

Hi @apparentlymart, thanks for your response!
I have the test setup and will be happy to provide more tests and outputs if needed.

The state file looks correct to me:

"nsxt_lb_pool.test": {
"type": "nsxt_lb_pool",
"depends_on": [
"nsxt_lb_http_monitor.test",
"nsxt_lb_passive_monitor.test"
],

Here is the full state file https://gist.github.com/annakhm/878961392356014df339cb8fcf9ed3c1
Apply output is here https://gist.github.com/annakhm/f8f15f1bbf957fec6864f9be5ff3531a

[apparentlymart]

Thanks for sharing those, @annakhm!

In our feature branch for the next major release we've slightly reworked how depends_on is handled, so I think what we'll probably do here is hold until the branch is feature complete, at which point it'll be merged down into master, and then we can test to see if this issue still arises with the new implementation.

However, it would still be nice to understand better what's going on here. From what you reported here it sounds like something isn't working quite right in the graph building phase, causing Terraform to "miss" those edges when it builds the graph to apply the plan.

If that is true, we may be able to see (or not see, rather) the missing edges in the terraform graph output:

• terraform apply
• delete the resources from configuration
• terraform plan -out=tfplan
• terraform graph tfplan

The final command above should produce a graphviz representation of the dependency graph for applying the given plan, which is supposed to have dependency edges reflecting the depends_on addresses in the state, but if my theory above is correct then those edges will be missing.

[annakhm]

Hi, the graph output is below:

digraph {
        compound = "true"
        newrank = "true"
        subgraph "root" {
                "[root] nsxt_lb_passive_monitor.test (destroy)" [label = "nsxt_lb_passive_monitor.test", shape = "box"]
                "[root] nsxt_lb_pool.test" [label = "nsxt_lb_pool.test", shape = "box"]
                "[root] provider.nsxt" [label = "provider.nsxt", shape = "diamond"]
               "[root] meta.count-boundary (count boundary fixup)" -> "[root] nsxt_lb_passive_monitor.test (destroy)"
                "[root] meta.count-boundary (count boundary fixup)" -> "[root] nsxt_lb_pool.test"
                "[root] nsxt_lb_passive_monitor.test (destroy)" -> "[root] provider.nsxt"
                "[root] nsxt_lb_pool.test" -> "[root] provider.nsxt"
                "[root] provider.nsxt (close)" -> "[root] nsxt_lb_passive_monitor.test (destroy)"
                "[root] provider.nsxt (close)" -> "[root] nsxt_lb_pool.test"
                "[root] root" -> "[root] meta.count-boundary (count boundary fixup)"
                "[root] root" -> "[root] provider.nsxt (close)"
        }
}

[apparentlymart]

Thanks! Here's the rendered graph:

Indeed it does seem like there's an edge missing between the two resources here, so at least we have an explanation for the behavior. We don't yet have a cause, but since (as I mentioned before) we have a reworked version of the graph builder in our development branch let's wait until that is in a more usable state and we can give this a try there to see what remains to be fixed there, if anything.

Thanks again for all the debugging help here! Once we're ready to test this in both 0.11 and the 0.12 development branch we'll see if we can reproduce this with the null_resource resource so that we can test it without access to NSXT.

[annakhm]

Thanks @apparentlymart!

[sloan-dog]

@annakhm Did you generate that graph manually, or programmatically? If the latter, please share your method 👍 I'm dying to visualize terraform dependency trees

[annakhm]

@annakhm Did you generate that graph manually, or programmatically? If the latter, please share your method 👍 I'm dying to visualize terraform dependency trees

@sloan-dog "terraform graph" generates the graph in dot format

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.