[E2E] Improve admincli util functions to verify the business data

[shaominchen]

Due to issue #1411, the assertion statement in the test logic cannot really guarantee the operation is completed as expected. Before issue #1411 is addressed, we can have more verification - in addition to verify the ssh exit code (which doesn't work right now), but also verify the real business data to make sure the operation is completed as expected.

There are a lot of admincli util functions. Adding extra verification logic to all functions is a huge effort. For now let's improve the following 4 functions:

1. CreateVMgroup()
2. DeleteVMgroup()
3. ConfigInit
4. ConfigRemove

[shaominchen]

@shuklanirdesh82 @ashahi1

[msterin]

What is the "business data verification" in this context ? Can you give an example ?
admin CLI does not return proper code to shell, thus ssh does not report failures properly. The fix is trivial, but I am not sure if it would address this as well. Can you elaborate on business data here . please.

[shaominchen]

"business data verification" means to verify the data affected by the command execution. For example, after calling admincli.CreateVMgroup(), for now we just verify the exit code returned by shell. Since admin CLi does not return proper exit code to shell, the verification logic (i.e. the assert statement in the test cases) doesn't really work - it always passes. The intention here is to add one more step to verify the VMGroup is truly created by running "vmgroup ls" for instance.

I think the fix #1464 should address this issue as well.

[msterin]

yes, but that could (and should) happen totally unrelated to return codes. Just because the CLI returns proper code, does not mean anything is persisted :-)

[shaominchen]

True. If that happens, it means there's a product issue, and we should have separate test cases to cover that.

[shuklanirdesh82]

this issue is similar to #1411 .. closing this as duplicate. Please reopen if I have mistaken with more detail.

[shuklanirdesh82]

I guess we can improve verification to validate actual data which is persisted in the backend.