Check for power off after kill regardless of startGuestProgram() return

If a container VM starts up and we stop it before VC detects the toolbox, startGuestProgram
returns an error 3016 even if the toolbox executed and the VM was powered off.  When this
happens, we try to power off the VM, throwing vSphere into an invalid state transition, which
it needs to recover from.  During this time, we have seen refreshing properties return invalid
powerstate value and prevents us from reading the exit code in portlayer's commit code.

To prevent this, we now wait to see if the container VM powered off after startGuestProgram,
regardless if it returns an error.

Resolves vmware#5803

lib/portlayer/exec/base.go

@@ -248,22 +248,25 @@ func (c *containerBase) kill(ctx context.Context) error {
 	log.Infof("sending kill -%s %s", sig, c.ExecConfig.ID)
 
 	err := c.startGuestProgram(timeout, "kill", sig)
-	if err == nil {
-		log.Infof("waiting %s for %s to power off", wait, c.ExecConfig.ID)
-		err := c.vm.WaitForPowerState(timeout, types.VirtualMachinePowerStatePoweredOff)
-		if err == nil {
-			return nil // VM has powered off
-		}
-
-		if timeout.Err() != nil {
-			log.Warnf("timeout (%s) waiting for %s to power off via SIG%s", wait, c.ExecConfig.ID, sig)
-		}
+	if err == nil && timeout.Err() != nil {
+		log.Warnf("timeout (%s) waiting for %s to power off via SIG%s", wait, c.ExecConfig.ID, sig)
 	}
-
 	if err != nil {
 		log.Warnf("killing %s attempt resulted in: %s", c.ExecConfig.ID, err)
 	}
 
+	// Even if startGuestProgram failed above, it may actually have executed.  If the container came up and then
+	// we kill it before VC gets a chance to detect the toolbox, vSphere can execute the kill but report an
+	// error 3016 indicating the guest toolbox wasn't found.  If we then try to poweroff, it may throw vSphere
+	// into an invalid transition and will need to recover.  If we try to grab properties at this time, the
+	// power state may be incorrect.  We work around this by waiting on the power state, regardless of error
+	// from startGuestProgram. https://github.com/vmware/vic/issues/5803
+	log.Infof("waiting %s for %s to power off", wait, c.ExecConfig.ID)
+	err = c.vm.WaitForPowerState(timeout, types.VirtualMachinePowerStatePoweredOff)
+	if err == nil {
+		return nil // VM has powered off
+	}
+
 	log.Warnf("killing %s via hard power off", c.ExecConfig.ID)
 
 	// stop wait time is not applied for the hard kill
@@ -296,7 +299,7 @@ func (c *containerBase) shutdown(ctx context.Context, waitTime *int32) error {
 
 		err := c.startGuestProgram(timeout, "kill", sig)
 		if err != nil {
-			return fmt.Errorf("%s: %s", msg, err)
+			log.Warnf("%s: %s", msg, err)
 		}
 
 		log.Infof("waiting %s for %s to power off", wait, c.ExecConfig.ID)