Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vcd_vapp: Wait for completion also if no guest_properties set. #649

Open
NilsBusche opened this issue Mar 2, 2021 · 7 comments
Open

vcd_vapp: Wait for completion also if no guest_properties set. #649

NilsBusche opened this issue Mar 2, 2021 · 7 comments
Assignees
@dataclouder @vbauzys

Comments

@NilsBusche
Copy link
Contributor

NilsBusche commented Mar 2, 2021 •
edited
Loading

Terraform Version

Terraform v0.13.5

Affected Resource(s)

  • vcd_vapp

Terraform Configuration Files

[...]

resource "vcd_vapp" "vapp" {
  name = var.hostname
}

resource "vcd_vapp_access_control" "vappacl" {
  vapp_id               = vcd_vapp.vapp.id
  shared_with_everyone  = true
  everyone_access_level = "ReadOnly"
}

resource "vcd_vapp_org_network" "vappnetattach" {
  vapp_name        = vcd_vapp.vapp.id == "always-not-equal" ? null : var.hostname
  org_network_name = var.network
}

resource "vcd_vapp_org_network" "vappbackupnetattach" {
  vapp_name        = vcd_vapp.vapp.id == "always-not-equal" ? null : var.hostname
  org_network_name = var.backup_network
}

[...]

Debug Output

I have added some debug lines into the provider in resource_vcd_vapp.go:

        vappstatus, err := vapp.GetStatus()
        log.Printf("[TRACE] State before if: %s", vappstatus)
        if _, ok := d.GetOk("guest_properties"); ok {

                vappstatus, err = vapp.GetStatus()
                log.Printf("[TRACE] State before blockwhilestatus: %s", vappstatus)

                // Even though vApp has a task and waits for its completion it happens that it is not ready
                // for operation just after provisioning therefore we wait for it to exit UNRESOLVED state
                err = vapp.BlockWhileStatus("UNRESOLVED", vcdClient.MaxRetryTimeout)
                if err != nil {
                        return fmt.Errorf("timed out waiting for vApp to exit UNRESOLVED state: %s", err)
                }

                vappstatus, err = vapp.GetStatus()
                log.Printf("[TRACE] State after blockwhilestatus: %s", vappstatus)

                guestProperties, err := getGuestProperties(d)
                if err != nil {
                        return fmt.Errorf("unable to convert guest properties to data structure")
                }

                log.Printf("[TRACE] Setting vApp guest properties")
                _, err = vapp.SetProductSectionList(guestProperties)
                if err != nil {
                        return fmt.Errorf("error setting guest properties: %s", err)
                }
        }

Terraform Trace Output without guest_properties set:

 2021-03-02T12:23:34.911+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:23:34 [TRACE] State before if: UNRESOLVED
2021-03-02T12:23:35.766+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:23:35 [TRACE] Setting empty properties into statefile because no properties were specified
2021-03-02T12:23:35.939+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:23:35 [DEBUG] Unlocking "xxxxxxxxxxxxxxxxxxxxx"
2021-03-02T12:23:35.939+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:23:35 [DEBUG] Unlocked "xxxxxxxxxxxxxxxxxxxxx"
2021/03/02 12:23:35 [WARN] Provider "registry.terraform.io/vmware/vcd" produced an unexpected new value for vcd_vapp.vapp, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .description: was null, but now cty.StringVal("")
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalMaybeTainted
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalWriteState
2021/03/02 12:23:35 [TRACE] EvalWriteState: recording 0 dependencies for vcd_vapp.vapp
2021/03/02 12:23:35 [TRACE] EvalWriteState: writing current state object for vcd_vapp.vapp
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalApplyProvisioners
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalMaybeTainted
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalWriteState
2021/03/02 12:23:35 [TRACE] EvalWriteState: recording 0 dependencies for vcd_vapp.vapp
2021/03/02 12:23:35 [TRACE] EvalWriteState: writing current state object for vcd_vapp.vapp
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalIf
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalIf
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalWriteDiff
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalApplyPost
2021/03/02 12:23:35 [TRACE] eval: *terraform.EvalUpdateStateHook
vcd_vapp.vapp: Creation complete after 2s [id=urn:vcloud:vapp:xxxxxxxxxxxxxxxxxxxxxxxx]

Terraform Trace Output with guest_properties set:

2021-03-02T12:17:27.601+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:17:27 [TRACE] State before if: RESOLVED
2021-03-02T12:17:27.661+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:17:27 [TRACE] State before blockwhilestatus: RESOLVED
2021-03-02T12:17:28.062+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:17:28 [TRACE] State after blockwhilestatus: RESOLVED
2021-03-02T12:17:28.062+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:17:28 [TRACE] Adding guest property: key=key1, value=value1 to object
2021-03-02T12:17:28.062+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:17:28 [TRACE] Setting vApp guest properties
2021/03/02 12:17:31 [TRACE] dag/walk: vertex "root" is waiting for "meta.count-boundary (EachMode fixup)"
2021/03/02 12:17:31 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/vmware/vcd\"] (close)" is waiting for "vcd_vapp_org_network.vappbackupnetattach"
2021/03/02 12:17:31 [TRACE] dag/walk: vertex "vcd_vapp_org_network.vappbackupnetattach" is waiting for "vcd_vapp.vapp"
2021/03/02 12:17:31 [TRACE] dag/walk: vertex "vcd_vapp_org_network.vappnetattach" is waiting for "vcd_vapp.vapp"
2021/03/02 12:17:31 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "vcd_vapp_access_control.vappacl"
2021/03/02 12:17:31 [TRACE] dag/walk: vertex "vcd_vapp_access_control.vappacl" is waiting for "vcd_vapp.vapp"
2021-03-02T12:17:31.971+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:17:31 [TRACE] Adding guest property: key=key1, value=value1 to object
2021-03-02T12:17:31.971+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:17:31 [TRACE] Updating vApp guest properties
2021-03-02T12:17:35.859+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:17:35 [TRACE] Setting empty properties into statefile because no properties were specified
2021-03-02T12:17:35.859+0100 [DEBUG] plugin.terraform-provider-vcd_v3.1.0: 2021/03/02 12:17:35 [TRACE] Setting properties into statefile
vcd_vapp.vapp: Creation complete after 9s [id=urn:vcloud:xxxxxxxxxxxxxxxxxxxxxxxx]

Expected Behavior

Terraform always should wait until vApp creation is completed.

Actual Behavior

In some cases the next steps depending on successfully created vApp start although the vApp creation is not completed and end in errors like this because Terraform does not wait for completion if optional guest_properties are not set:

Error: error creating vApp org network. &errors.errorString{s:"error updating vApp Network: API Error: 400: [ xxxxxxxxxxxxxxxxxxxxxx ] The entity Ref: com.vmware.vcloud.entity.vapp:xxxxxxxxxxxxxxxx is busy completing an operation VDC_COMPOSE_VAPP. VDC_COMPOSE_VAPP(com.vmware.vcloud.entity.task:xxxxxxxxxxxxxxxxxxxxxxxx)"}

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. Create configuration as described above
  2. terraform apply
  3. Add guest_properties section to the configuration
  4. terraform destroy
  5. terraform apply

Possible solution

Move the "wait" block out of the if-condition for guest_properties here:

terraform-provider-vcd/vcd/resource_vcd_vapp.go

Line 118 in 75876b5

err = vapp.BlockWhileStatus("UNRESOLVED", vcdClient.MaxRetryTimeout)

@jpbuecken
Copy link

jpbuecken commented Aug 24, 2021 •
edited
Loading

Any news on this one? This is still an issue with VCD 10.3.

This breaks our automated VM deployments.

vcd_vapp.vapp: Creation complete after 2s [id=urn:vcloud:vapp:fee69b93-6fa7-457c-87b4-fd3d36358081]
vcd_vapp_org_network.vappnetattach: Creating...
vcd_vapp_access_control.vappacl: Creating...
vcd_vapp_org_network.vappbackupnetattach[0]: Creating...
vcd_vapp_org_network.vappnetattach: Creation complete after 5s [id=urn:vcloud:network:f11b5312-a231-4ebb-ad85-e887049f1198]
vcd_vapp_access_control.vappacl: Creation complete after 6s [id=urn:vcloud:vapp:fee69b93-6fa7-457c-87b4-fd3d36358081]

Error: error creating vApp org network. &errors.errorString{s:"error updating vApp Network: API Error: 400: [ 4ea08867-49e1-4b74-a76b-676a199f6ab2 ] The entity Ref: com.vmware.vcloud.entity.vapp:fee69b93-6fa7-457c-87b4-fd3d36358081 is busy completing an operation VDC_COMPOSE_VAPP. VDC_COMPOSE_VAPP(com.vmware.vcloud.entity.task:b329b049-8133-4b22-aa17-ff4616964584)"}

Edit: I looks like the issue occur more often if you attach 2 vapp networks. My case uses vcd_vapp_org_network 2 times and a vcd_vapp_access_control.

@dataclouder
Copy link
Contributor

Reproducing this issue is quite difficult. I was able to do it by adding yet another network to the configuration, and running apply/destroy in a loop. After 18 iterations, it failed. We'll look into it.

@dataclouder
Copy link
Contributor

@NilsBusche does the fix you recommend (moving the wait block outside the guest property if) solve the issue for you?

@jpbuecken
Copy link

Hello @dataclouder : Since you was able to reproduce the issue with more vcd_vapp_org_network I have another theorie:

We had discussion in the past that "vCd allows only one action per vApp" [1]
Maybe this is valid for vcd_vapp_org_network as well?
Maybe a vcd_vapp_org_network makes a vapp busy?
Does the provider make sure only one vcd_vapp_org_network action is performed at a time instead of in parallel (similar to vapp_vm)?

[1] #507 (comment)

@dataclouder
Copy link
Contributor

@jpbuecken
Thanks for your analysis. This could be one of the contributing causes. However, all functions to create or modify vApp networks are properly locking access to the parent vApp.

There is a line of investigation that I can pursue, however. In vApp, there is a lock on Creation (which invokes Update), but not on update itself. Now, this is something that we must check carefully, because can create issues, but not in this specific test case, where the vApp must be properly created before other resources can start being processed.
It's more material to look into, with plenty of hot spots that can be the cause of this issue. We are looking into it

@MarioAlexis
Copy link

MarioAlexis commented Apr 29, 2022 •
edited
Loading

Any update on this?
We are facing the exact same issue on vCD version 10.3.3.19610595
Happening after attaching 2 vcd_vapp_org_network inside a for_each terraform loop.

FYI
This issue wasn't there on version 9.7.0 (9.7.0.14534864 )

@jpbuecken
Copy link

Hello,
I was monitoring
hashicorp/terraform-plugin-sdk#67 for a workaround.
Idea was I could set a parallism of 1 for resources, that modifies the vAPP (vcd_vapp_access_control , vcd_vapp_org_network)

They mentioned a solution, but it has to be implemented in the provider, not in terraform core:
hashicorp/terraform-plugin-sdk#67 (comment)

But I'm sure you are already aware of this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dataclouder dataclouder

@vbauzys vbauzys

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@MarioAlexis @dataclouder @jpbuecken @NilsBusche @vbauzys