Performance drop for remove VCH operation

[gopakumarr]

Uninstall/Remove VCH performance regression

From VIC engine build 19874 we are starting to observe Uninstall VIC command taking longer time neary +7 secs

Operation	19716	19874
Uninstall VCH	1.233 secs	8.134 secs

Deployment details

vic-machine-linux create --target $ESXHOSTIP --image-store $DATASTORE --name vic_$VERSION --compute-resource /ha-datacenter/host/*/Resources --no-tls --force --timeout 10m0s --user root --password mypassword

Uninstall VCH Command

vic-machine-linux delete --force --target $ESXHOSTIP --name vic_$VERSION --compute-resource /ha-datacenter/host/*/Resources --user root --password mypassword

From the logs it looks like time is mostly spent on removing portgroup

19716 logs snippet

INFO[0000] Removing volume stores
INFO[0000] Removing appliance VM network devices
INFO[0000] Removing Portgroup "vic_19716"
INFO[0001] Removing VirtualSwitch "vic_19716"
INFO[0001] Removing Resource Pool "vic_19716"

19874 logs snippet

INFO[0000] Removing volume stores
INFO[0000] Removing appliance VM network devices
INFO[0007] Removing Portgroup "vic_19874"
INFO[0007] Removing VirtualSwitch "vic_19874"
INFO[0008] Removing Resource Pool "vic_19874"

[zjs]

Could you attach debug-level logs for both versions? This would allow us to more precisely pinpoint a possible cause.

[hickeng]

The regression was almost certainly introduced as a legitimate change in build 19750. Unfortunately the CI network infrastructure issues (vmware/vic-tasks#69) prevented an updated build from publishing cleanly until 19874 so there's a set of 15 commits that present in 19874.

The change I think introduces this is #6943 which adds the polite guest shutdown path instead of direct power off, meaning we're waiting for the vic-engine components to finalize (including logging out of VC), OS services to shut down, and the OS to power off via ACPI. 8 seconds is in line with what I observed during testing for this change.

@gopakumarr apologies - I should have flagged this change to you as a known impact.

[zjs]

This may be worth release noting as it's an expected user-visible performance change. Perhaps the release note can include some information about #6943 as well, for context.

[stuclem]

@zjs, does this mean that this longer shutdown time is now the standard behaviour? If so, then I think that it needs to go into the core docs rather than in a release note.

[zjs]

does this mean that this longer shutdown time is now the standard behaviour?

Yes.

If so, then I think that it needs to go into the core docs rather than in a release note.

I couldn't find anything in the current docs about expected duration of the shutdown operation. (And that seems like a generally difficult thing to characterize, so it's probably best not to try to document it.)

It seemed like it may be worth noting in the context of, essentially: after you upgrade, this operation will take longer, and that's not a sign that anything is wrong.