Enable test to be re-run in the same test suite

[damies13]

Is your feature request related to a problem? Please describe.
When running tests with Browser Library or SeleniumLibrary there is a large CPU overhead on the agent machines launching the browser and closing it each test iteration

Describe the solution you'd like
It would be good to have the option to tell the agent to simply re-run the test in the same test suite, that way the launching of the browser can be done in a suite setup and closing the browser in a suite tear down, and the actual website test case can be run over and over again without having to close the browser each time

There is an example of how to achieve this with the Robot Framework Listener interface in this forum post:
https://forum.robotframework.org/t/can-i-run-setup-test-and-teardown-separately/3878/14?u=damies13

[damies13]

As a workaround until this feature is implemented you can already do the following:

• include the TestRepeater.py from the forum post as a resource using Metadata File (see Support Files)
• add --listener TestRepeater.py to the Robot Options of the script (see Additional settings for test group image )

[damies13]

How I envisage this will work is somewhat similar to the way the agent reports the test results, the agent will create the listener and add it to the robot command line when running the robot as it does with RFSListener2.py.

This is a little mote complicated though, as we will also need to do the following:

• add option in the Manager GUI for the user to turn this behaviour on/off per script
• Potentially an update to the Manager <---> agent communication so the manager can tell the agent to enable this behaviour
• listener needs to know the iteration number for the current test and update the iteration number for the next test (${RFS_ITERATION})
• listener needs to communicate the iteration number back to the agent, so if the test fails and agent re-launches the robot it starts with the next iteration number, not the second iteration number created by the listener

[damies13]

Maybe we can avoid the need to communicate back to the agent by making the repeated tests a sub test of the initial test iteration

This could potentially look like

${RFS_ITERATION}    1
${RFS_ITERATION}    1.1
${RFS_ITERATION}    1.2
${RFS_ITERATION}    1.3
...
${RFS_ITERATION}    1.913
...
${RFS_ITERATION}    1.9880
...
${RFS_ITERATION}    13
${RFS_ITERATION}    13.1
${RFS_ITERATION}    13.2
${RFS_ITERATION}    13.3
...
${RFS_ITERATION}    13.913
...
${RFS_ITERATION}    13.9880

Just need work out how that would work with the data coming back to the Manager and also with Reporter

[damies13]

Reporter not working with data from tests with decimal iteration need to investigate

turns out I didn't break reporter, it was simply am insufficient data issue, 7 results is not enough to generate a 90 percentile, ran reporter again after running the test a bit longer and everything display's as expected.

[damies13]

Merged into v1.2.0 release branch

[damies13]

Found an issue with the iteration numbering

1
1.1    <-- A
1.2    <-- B
1.3
1.4
....
1.8
1.9
1.10  <-- A
1.11
....
1.19
1.20  <-- B
1.21

1.1 and 1.10 are treated as the same number, likewise 1.2 and 1.20.

[damies13]

I've thought a bit about how to solve this

• I could prefix the sub iteration with leading zero's but then how many? 3, 5, 8, 10? and what should the test repeater do if it reaches 1.999? or 1.99999, should it exit and go on to iteration 2? this might give unexpected behaviour.
• I could try making the iteration number a string and send back 1_1, 1_10, 1_20, 1_100, etc

[damies13]

Now resolved using 1_10,

Fix Merged into v1.2.0 release branch