functional: more fleetctl unit action tests #1544

dongsupark · 2016-04-11T09:06:40Z

In order to make functional tests cover more fleetctl commands, do the following:

define util.UnitFileState for handling output of list-unit-files
aadd new cluster operations WaitForNUnits and WaitForNUnitFiles, wrappers based on util.WaitForState.
introduce a new test TestUnitCat for "fleetctl cat".
introduce a new test TestUnitStatus for "fleetctl status".
improve an existing test TestUnitSubmit.
introduce a new test TestUnitLoad for fleetctl load.

tixxdz · 2016-04-11T10:03:49Z

functional/unit_action_test.go

 		t.Fatalf("Unable to submit fleet unit: %v", err)
 	}
-	stdout, _, err := cluster.Fleetctl(m, "list-units", "--no-legend")


@dongsupark thanks for the patches!

Hmm something is wrong here, my understanding with fleet is submit only sends the units into the cluster, so it will show up when you do "fleetctl list-unit-files" but not with "fleetctl list-units" since the unit is not active.

Was this test wrong from the beginning ? we should not check the number of printed lines, we should just grep for the name of the unit, thoughts ?

Thanks!

@tixxdz Yes, this part has been wrong since the day 0. (5f1340b)
I'm not sure how the test has always succeeded. I'll try to improve it.

It wasn't "wrong", just not specific enough, right?

@jonboulle yes! by "wrong" I'm only referring to the functional tests and that perhaps there is a bug somewhere that allowed it to pass. For fleetctl commands they are clear.

@jonboulle yes not specific enough ;-)

@dongsu capturing the context, maybe we need a function runCheckListUnits() , runCheckListUnitFiles() and if you want to grep submitted units since they won't show up in other commands and you are not sure about the number of returned units... then make runGetListUnitFiles() return the units and grep it into the caller ? just quick thoughts.

thanks

@jonboulle no, it was indeed completely wrong: list-units returns an empty string; but that's still counts as one line, so the test "passed"...

Note that most of the existing tests only check the number of units, except for cases where there is actually a possibility for ambiguity -- so checking only the count would be fine here as well. It just needs to be fixed to use list-unit-files; and to properly check for a non-empty string...

I just saw that one of the existing tests in this very file (namely TestUnitRunnable()) actually test both the count and the name -- so feel free to copy that if you really think it's worthwhile...

@antrik

no, it was indeed completely wrong: list-units returns an empty string; but that's still counts as one line, so the test "passed"...

Yes, that was exactly what I wanted to say. That's why I said yesterday that this part has been wrong since the beginning. I think the underlying reason was that len(strings.Split("", "\n")) returns 1, not 0 as expected.

So my solution to get it working was adding additional conditions like "expectedUnits == 0 && len(outListUnitFiles) == 0" in each check function. Still somehow nasty, but it's working for now. Afterwards, we should do some code auditing in the entire tree to find such errors. But anyway that would be out of the scope of this PR.

dongsupark · 2016-04-11T14:09:29Z

Updated.

add new utilities util.SliceContains and util.GetListUnitNames.
add a new helper runListUnitFiles, a wrapper for fleetctl list-unit-files.
improve TestUnitSubmit to correctly check the output running by list-unit-files instead of list-units, as well as to parse output to compare the unit name with the expected name.
improve TestUnitLoad to parse output to compare the unit name with the expected name.

antrik · 2016-04-11T17:17:18Z

functional/unit_action_test.go

 	// submit a unit and assert it shows up
-	if _, _, err := cluster.Fleetctl(m, "submit", "fixtures/units/hello.service"); err != nil {
+	_, _, err = cluster.Fleetctl(m, "submit", unitFile)
+	if err != nil {


Why did you change the syntax here? There is no reason to have the assignment outside the if.

tixxdz · 2016-04-13T08:35:24Z

So a couple of changes then maybe merge this one ?

dongsupark · 2016-04-13T13:53:37Z

Updated.

define util.UnitFileState for handling output of list-unit-files
add new cluster operations WaitForNUnits and WaitForNUnitFiles
make TestUnitSubmit() use cluster.WaitForNUnitFiles
make TestUnitLoad() use cluster.WaitForNUnits
fix a bug in TestUnitStatus(), loading a unit before getting status.
drop unnecessary helpers like SliceContains, runListUnits, etc.
fix minor things like comments or coding style

I assume this PR should be merged prior to the existing #1509.
So I might also have to update functional tests in #1509 to avoid conflicts.

antrik · 2016-04-13T17:48:26Z

functional/unit_action_test.go

 		t.Fatalf("Failed to destroy unit: %v", err)
 	}
-	stdout, _, err = cluster.Fleetctl(m, "list-units", "--no-legend")
+	// wait until the unit gets submitted up to 15 seconds


I think this comment is still wrong here...

In addition to the existing UnitState, define UnitFileState to parse output of fleetctl list-unit-files, as well as its parse function ParseUnitFileStates(). The existing ParseUnitStates() parses Name, State, and Machine, including specific handling of machine strings. In contrast, ParseUnitFileStates() simply parses 3 fields: Name, DesiredState, and State.

…Files WaitForNUnits() runs fleetctl list-units to get a map of []UnitState (unit, active, machine). This operation should be called after fleetctl load or start. WaitForNUnitFiles() runs fleetctl list-unit-files to get a map of []UnitFileState (unit, desiredstate, state). This operation should be called after fleetctl submit.

A new functional test TestUnitCat simply tests if "fleetctl cat" works.

TestUnitStatus simply checks if "fleetctl status hello.service" works.

Improve TestUnitSubmit() like the following: * use list-unit-files instead of list-units, to correctly verify submitted units. * check that the output of list-unit-files contains the correct unit name, with help of a new helper cluster.WaitForNUnitFiles. That way the whole functional test could become less racy.

A new test TestUnitLoad verifies that "fleetctl {load,unload}" correctly works: load -> list-units -> unload -> list-units -> load

dongsupark · 2016-04-14T10:22:24Z

Done.

Fixed a bug in TestUnitStatus, by calling cluster.Fleetctl() correctly.
Removed unnecessary checking in cluster.WaitForN{Units,UnitFiles}()
Fixed comments

tixxdz · 2016-04-15T18:18:01Z

lgtm! thanks all.

dongsupark force-pushed the dongsu/fleetctl-unit-action-fxtests branch from 38ec1aa to 0313fa8 Compare April 11, 2016 09:23

tixxdz reviewed Apr 11, 2016
View reviewed changes

dongsupark force-pushed the dongsu/fleetctl-unit-action-fxtests branch from 0313fa8 to e6ba65f Compare April 11, 2016 13:57

antrik reviewed Apr 11, 2016
View reviewed changes

tixxdz mentioned this pull request Apr 13, 2016

fleetd:registry: improve Units() performance remove extra loops when fetching Jobs list from etcd #1515

Merged

tixxdz added this to the v0.13.0 milestone Apr 13, 2016

tixxdz added the priority/P1 label Apr 13, 2016

dongsupark force-pushed the dongsu/fleetctl-unit-action-fxtests branch from e6ba65f to 28ebafd Compare April 13, 2016 13:36

antrik reviewed Apr 13, 2016
View reviewed changes

Dongsu Park added 6 commits April 14, 2016 12:09

functional: introduce a new test TestUnitCat for fleetctl cat

0a5bee7

A new functional test TestUnitCat simply tests if "fleetctl cat" works.

functional: introduce a new test TestUnitStatus

24f2da0

TestUnitStatus simply checks if "fleetctl status hello.service" works.

functional: a new test TestUnitLoad to check fleetctl {load,unload}

7bcdb51

A new test TestUnitLoad verifies that "fleetctl {load,unload}" correctly works: load -> list-units -> unload -> list-units -> load

dongsupark force-pushed the dongsu/fleetctl-unit-action-fxtests branch from 28ebafd to 7bcdb51 Compare April 14, 2016 10:15

This was referenced Apr 14, 2016

functional: fleetctl submit multiple units #1541

Closed

fleet: add replace unit support #1509

Merged

functional: support testing multiple units #1552

Closed

functional: support testing multiple units #1553

Merged

tixxdz merged commit e2a8370 into coreos:master Apr 15, 2016

dongsupark deleted the dongsu/fleetctl-unit-action-fxtests branch April 18, 2016 16:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

functional: more fleetctl unit action tests #1544

functional: more fleetctl unit action tests #1544

dongsupark commented Apr 11, 2016

tixxdz Apr 11, 2016

dongsupark Apr 11, 2016

jonboulle Apr 11, 2016

tixxdz Apr 11, 2016

tixxdz Apr 11, 2016

tixxdz Apr 11, 2016

antrik Apr 11, 2016

antrik Apr 11, 2016

antrik Apr 11, 2016

dongsupark Apr 12, 2016

dongsupark commented Apr 11, 2016

antrik Apr 11, 2016

tixxdz commented Apr 13, 2016

dongsupark commented Apr 13, 2016

antrik Apr 13, 2016

dongsupark commented Apr 14, 2016

tixxdz commented Apr 15, 2016

functional: more fleetctl unit action tests #1544

functional: more fleetctl unit action tests #1544

Conversation

dongsupark commented Apr 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongsupark commented Apr 11, 2016

Choose a reason for hiding this comment

tixxdz commented Apr 13, 2016

dongsupark commented Apr 13, 2016

Choose a reason for hiding this comment

dongsupark commented Apr 14, 2016

tixxdz commented Apr 15, 2016