Updates to pre-upgrade container are dropped

[hickeng]

Build: https://ci-vic.vmware.com/vmware/vic/17605/8
Suite: 11-01-Upgrade

Failing output because it contains Exited (0)

CONTAINER ID        IMAGE                                                  COMMAND                  CREATED              STATUS                     PORTS                                                      NAMES
<snip>
d865d2abeaaa        busybox                                                "sh"                     3 minutes ago        Exited (0) 3 seconds ago                                                              vch-restart-test1

The test is attempting to assert that the container from pre-upgrade can be started. It can, however because of a configuration version difference the resetting of started field (and times, et al) is being dropped (see this code) and we therefore don't block waiting for the start to occur.
This was not detected by the test prior to 6144b31 because the call to start explicitly set the container state to Starting - the test should have been checking for Running, not checking for the abscence of Exited. However with that commit we now use the vSphere event to trigger the state update and the test therefore still sees the old state.

There is a second issue related to this migration behaviour (#7128) and discarding IP address assignments. The only correct fix for this is to finish implementing the migration logic - we have the portion to migrate old configurations forward for reading, but not the portion that migrates new configurations back for writing (and should return a hard error if a critical element cannot be migrated).

It should be comparatively simple to add the reverse path to the existing migration code, but I do not know without investigating whether it will entail refactoring. Estimate as a 5 due to unknowns.

Related: #7128 #5653

[mdubya66]

being seen on almost every CI run. changing to P0

[sgairo]

This has been commented out in the test.

[mdubya66]

This is existing behavior for upgrade, it's a p2 as @hickeng made it when filing. Resetting.

There is an integration test on this and when fixed needs to be revisited tests/test-cases/Group11-Upgrade/11-01-Upgrade.robot

[hickeng]

I'm adding this to 1.5 as it's now been hit by a customer. @dbarkelew may know a bugXXX number to link with.

[hickeng]

@YanzhaoLi To recreate this:
a. add a new plugin to lib/migration/plugins to force a data version migration after the upgrade (this is just a change for recreation - you don't want to ship the plugin unless there is an actual data format change).
b. start the container at least once before the upgrade - this is necessary to get the started field set to true.