[Core] Add reverse and random scenario execution order #1645

grasshopper7 · 2019-05-27T08:18:30Z

Summary

Allows scenarios to be executed in random or reverse order. Custom execution orders can also be created.

Details

All scenarios can be run in random order.
java <classpath> cucumber.api.cli.Main --order random -g <glue path> <feature file path>
The number of scenarios to be run in random can also be mentioned.
java <classpath> cucumber.api.cli.Main --order random --count <number> -g <glue> <feature file>

All scenarios can be run in reverse order.
java <classpath> cucumber.api.cli.Main --order reverse -g <glue path> <feature file path>

Custom orders can be created by implementing the PickleOrder interface in the cucumber.runtime.order package.

Motivation and Context

Scenarios can be executed in random order. Aids in finding out if their is any dependency between scenarios.

How Has This Been Tested?

Tests for RuntimeOptions are added in RuntimeOptionsTest for testing options in the command. OrderTypeFactoryTest tests the OrderTypeFactory class. RandomOrderTypeTest tests the RandomOrderType class. No test for ReverseOrderType as it will test the reverse method of Collections class.

Types of changes

Bug fix (non-breaking change which fixes an issue).
[x ] New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist:

[x ] I've added tests for my code.
My change requires a change to the documentation.
I have updated the documentation accordingly.

coveralls · 2019-05-27T08:33:36Z

Coverage increased (+0.006%) to 85.962% when pulling 6c19a59 on grasshopper7:scenario-execution-order into 4a9835f on cucumber:master.

mpkorstanje

Heya, looks pretty neat!

I'd say that my major objection is that different concerns are all united in the implementation of OrderType. I think it would be better to separate the CLI parsing from the implementation separate the implementation of --count and --order.

mpkorstanje · 2019-05-28T06:07:23Z

core/src/main/java/cucumber/runtime/RuntimeOptions.java

@@ -180,6 +184,10 @@ private void parse(List<String> args) {
                parsedJunitOptions.addAll(asList(arg.substring("--junit,".length()).split(",")));
            } else if (arg.equals("--wip") || arg.equals("-w")) {
                wip = true;
+            } else if (arg.equals("--order")) {
+            	orderTypeData.put(OrderType.TYPE_NAME, args.remove(0));


The names of the CLI commands are specific to the runtime options parser. It is not necessary for the OrderType to know what it is named on the CLI. It is okay to inline this string here.

Will change it.

mpkorstanje · 2019-05-28T06:11:15Z

core/src/main/java/cucumber/runtime/RuntimeOptions.java

@@ -214,6 +222,10 @@ private void parse(List<String> args) {
            junitOptions.clear();
            junitOptions.addAll(parsedJunitOptions);
        }
+        if(orderTypeData.containsKey(OrderType.TYPE_NAME)) {
+        	orderType = OrderTypeFactory.getOrderType(orderTypeData);
+        	orderType.checkVariableValues(orderTypeData);


The exact syntax of CLI commands is also specific to the runtime options parser. Checking if the input is valid should be done here.

Additionally any time when --count is used it's argument should be a number and should non-negative. Tying the validation to the order type makes this more complex then needed.

I think it might be better to make --count and --order independent variables and simply ignore --count when not relevant.

The --count is used only for the RandomOrderType thus the check was specific to that type. That required the test for number etc to be done later.

Idea of making --count applicable for all types, to be used if required, does simplifies things. If so then the orderTypeData map is also redundant and can be removed. Also no need of checkVariableValues method.

Later on if a future OrderType requires an additional variable from CLI it needs to be handled accordingly.

mpkorstanje · 2019-05-28T06:11:46Z

core/src/main/java/cucumber/runtime/order/OrderType.java

+
+public abstract class OrderType {
+
+	public static final String NONE_ORDER_TYPE = "None";


Type names should be lower case or case insensitive.

Will change it.

mpkorstanje · 2019-05-28T06:13:49Z

core/src/main/java/cucumber/runtime/order/OrderType.java

+
+import gherkin.events.PickleEvent;
+
+public abstract class OrderType {


I think this class can be turned into an interface. The default implementation for orderPickleEvents is only used by none order.

mpkorstanje · 2019-05-28T06:14:51Z

core/src/main/java/cucumber/runtime/order/OrderTypeFactory.java

+
+import cucumber.runtime.CucumberException;
+
+public class OrderTypeFactory {


This mapping belongs in RuntimeOptions. Additionally I think the different orders, could be turned into an enum that implements/extends the OrderType interface/class.

mpkorstanje · 2019-05-28T06:24:19Z

core/src/main/java/cucumber/runtime/order/RandomOrderType.java

+import cucumber.runtime.CucumberException;
+import gherkin.events.PickleEvent;
+
+public class RandomOrderType extends OrderType {


This should be an Enum that implements/extends OrderType.

https://stackoverflow.com/questions/2709593/why-would-an-enum-implement-an-interface

Will convert into enum. No need of Factory.

mpkorstanje · 2019-05-28T06:32:39Z

core/src/main/java/cucumber/runtime/order/RandomOrderType.java

+	@Override
+	public List<PickleEvent> orderPickleEvents(List<PickleEvent> pickleEvents) {	
+		Collections.shuffle(pickleEvents);
+		return pickleEvents.subList(0, ((count > pickleEvents.size() || count == 0) ? 


Nifty but way to hard to follow.

I think this class is doing too much. It's both ordering and selecting pickles. It should only order pickles. By making --order and --count independent of each other you can remove this complexity all together.

Yeah my bad. Though this logic needs to remain, as need to make sure that the count is less or equal to the size of the incoming PickleEvents to avoid exception.

No worries!

mpkorstanje · 2019-05-28T06:36:38Z

core/src/main/java/cucumber/runtime/Runtime.java

                }
            }
        }
+
+        final List<Future<?>> executingPickles = new ArrayList<>();
+        for(final PickleEvent pickleEvent : runtimeOptions.getOrderType().orderPickleEvents(filteredEvents)) {        	


Ideally the order is injected into the Runtime via the builder. This allows us and other implementations to create Runtime instances without passing in CLI based RuntimeOptions.

This will in the long run let us separate the parsing of CLI options from executing pickles.

For example:

List<PickleEvent> allPickles = new ArrayList<>(); for (CucumberFeature feature : features) { allPickles.addAll(feature.getPickles()); } List<PickleEvent> orderedPickles = pickleOrder.order(allPickles); List<PickleEvent> filteredPickles = pickleFilter.filter(orderedPickles); final List<Future<?>> executingPickles = new ArrayList<>(); for (PickleEvent filtredPickle : filteredPickles) { executingPickles.add(executor.submit(new Runnable() { @Override public void run() { runnerSupplier.get().runPickle(pickleEvent); } })); }

To implement the above the existing iteration for the filters will need to be moved to the Filters class. Are you suggesting above that the pickles need to be ordered first and then filtered?

Yes. It'd be much easier to implement this with a Java8 Stream but we don't have those at the moment. So I don't mind too much where the iteration ends up.

grasshopper7 · 2019-05-31T07:35:49Z

Implemented OrderType as an enumeration implementing OrderPickleEvents interface.

Separated --order and --count CLI options to work independently. The --order logic is handled by OrderType enum. While the --count is handled by the limitPickleEvents method in Filters class.

Updated the test classes accordingly.

mpkorstanje · 2019-06-04T18:45:29Z

Looks good! I'll need a bit more time to give this a proper once over.

mpkorstanje

Code looks good, just a few nit picks.

Suppose a test run fails, how would you reproduce this test run? The random order will be different on the next execution. Perhaps we can have the RANDOM option print out the seed use to shuffle the pickles? Then you could read it back again using --order random:<seed>.

If you can move the last bit of option parsing into RuntimeOptions we don't need to add that now though. It can be added later.

mpkorstanje · 2019-06-04T19:23:26Z

core/src/main/java/cucumber/runtime/filter/Filters.java


-    public Filters(FilterOptions filterOPtions) {
+    public Filters(FilterOptions filterOptions) {


mpkorstanje · 2019-06-07T08:27:10Z

core/src/main/java/cucumber/runtime/Runtime.java


    public Runtime(final Plugins plugins,
                   final RuntimeOptions runtimeOptions,
                   final EventBus bus,
                   final Filters filters,
                   final RunnerSupplier runnerSupplier,
                   final FeatureSupplier featureSupplier,
-                   final ExecutorService executor) {
+                   final ExecutorService executor,
+                   final OrderType orderType) {


Did you mean to use OrderPickleEvents here?

mpkorstanje · 2019-06-07T08:48:24Z

core/src/main/java/cucumber/runtime/order/OrderType.java

+
+     public static OrderType getOrderType(String name) {
+        for (OrderType orderType : OrderType.values()) {
+            if (name.equalsIgnoreCase(orderType.name())) {


Ideally we parse the different order names in runtime options too. This would avoid having to support both lower and uppercase options in future versions.

lock · 2020-06-24T23:18:29Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

grass-hopper-moc and others added 2 commits May 27, 2019 00:40

[Core] Scenario Execution Order

f707a12

Update USAGE.txt

8b6df43

mpkorstanje requested changes May 28, 2019

View reviewed changes

mpkorstanje mentioned this pull request May 29, 2019

User Streams to filter pickles in Runtime #1646

Closed

grass-hopper-moc added 2 commits May 31, 2019 12:14

[Core] Scenario Execution Order & Filter

dec7fac

[Core] Scenario Order Updated usage options

4e40c78

[Core]Fixed failing filter test

6c19a59

mpkorstanje reviewed Jun 7, 2019

View reviewed changes

mpkorstanje changed the title ~~Custom scenario execution order~~ [Core] Add reverse and random scenario execution order Jun 8, 2019

Tighten up naming and parsing

3d6c495

mpkorstanje merged commit b89e538 into cucumber:master Jun 8, 2019

grasshopper7 mentioned this pull request Jun 10, 2019

[Core] Added seed functionality to Random OrderType #1658

Merged

6 tasks

lock bot locked as resolved and limited conversation to collaborators Jun 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Add reverse and random scenario execution order #1645

[Core] Add reverse and random scenario execution order #1645

grasshopper7 commented May 27, 2019 •

edited by mpkorstanje

Loading

coveralls commented May 27, 2019 •

edited

Loading

mpkorstanje left a comment

mpkorstanje May 28, 2019

grasshopper7 May 29, 2019

mpkorstanje May 28, 2019

grasshopper7 May 29, 2019 •

edited

Loading

mpkorstanje May 28, 2019

grasshopper7 May 29, 2019

mpkorstanje May 28, 2019

mpkorstanje May 28, 2019

mpkorstanje May 28, 2019

mpkorstanje May 28, 2019

grasshopper7 May 29, 2019

mpkorstanje May 28, 2019

grasshopper7 May 29, 2019

mpkorstanje May 29, 2019

mpkorstanje May 28, 2019

mpkorstanje May 28, 2019 •

edited

Loading

grasshopper7 May 29, 2019

mpkorstanje May 29, 2019

grasshopper7 commented May 31, 2019

mpkorstanje commented Jun 4, 2019

mpkorstanje left a comment

mpkorstanje Jun 4, 2019

mpkorstanje Jun 7, 2019

mpkorstanje Jun 7, 2019 •

edited

Loading

lock bot commented Jun 24, 2020


		public abstract class OrderType {

		public static final String NONE_ORDER_TYPE = "None";


		import gherkin.events.PickleEvent;

		public abstract class OrderType {


		import cucumber.runtime.CucumberException;

		public class OrderTypeFactory {


		public Filters(FilterOptions filterOPtions) {
		public Filters(FilterOptions filterOptions) {

[Core] Add reverse and random scenario execution order #1645

[Core] Add reverse and random scenario execution order #1645

Conversation

grasshopper7 commented May 27, 2019 • edited by mpkorstanje Loading

Summary

Details

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

coveralls commented May 27, 2019 • edited Loading

mpkorstanje left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grasshopper7 May 29, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mpkorstanje May 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grasshopper7 commented May 31, 2019

mpkorstanje commented Jun 4, 2019

mpkorstanje left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mpkorstanje Jun 7, 2019 • edited Loading

Choose a reason for hiding this comment

lock bot commented Jun 24, 2020

grasshopper7 commented May 27, 2019 •

edited by mpkorstanje

Loading

coveralls commented May 27, 2019 •

edited

Loading

grasshopper7 May 29, 2019 •

edited

Loading

mpkorstanje May 28, 2019 •

edited

Loading

mpkorstanje Jun 7, 2019 •

edited

Loading