[Core] Parallel cukes #1389

boaty82 · 2018-06-09T11:51:23Z

Notice

This PR replaces Parallel support #1357 and relies on Refactor Runtime #1367

Outline

I’ve previously used another implementation to deliver parallel execution of cucumber tests, however it basically wrapped up the creation and execution of Runtime’s, followed by the need to merge the multiple JSON/HTML etc reports that followed.

I reviewed the current implementation and also took note of comments in Support concurrent executions of scenarios also I perused all tickets related to running in parallel cukes for other insights

Main changes

Added `--threads` argument to runtime options

Allows users to specify the (max) number of threads to be used to run the tests

The actual number of threads used will be the smaller of the given --threads value and the number of features to be executed

Tests are placed into a Queue and consumed by each thread when needed, resulting in a more performant usage of the threads & less deterministic order of Features being run

New formatter introduced `TimelineFormatter`

Which produces reports using vsjis.org timeline to highlight which feature was run on which Thread and when.
Note: the resources (js, html etc) are currently within cucumber-core if the PR is accepted then I’d imagine these would have to be moved out into its own project, similarly to cucumber-html.

example.zip

Types of changes

Bug fix (non-breaking change which fixes an issue).
New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).
~~- Runtime constructor now private Runtime.Builder introduce to hide away boiler plate setup/instantiation code~~

Checklist:

I've added tests for my code.
My change requires a change to the documentation.
I have updated the documentation accordingly.

mpkorstanje · 2018-07-05T17:16:28Z

Preliminary release notes:

Cucumber can be run in parallel. Use --threads fro the CLI or consult the documentation on how to run JUnit/TestNG in parallel.
JUnit can now be run in parallel. Will only run Features in parallel (for now). Changing this to pickles will result in the loss of the outline.
TestNg can now be run in parallel. Will run pickles in parallel.
With the exception of the CLI Formatters will now receive all test events after the run is complete. When running from the CLI formatters will receive all test events after the run is complete only when running in parallel.
Implement concurrent event listener to receive events in real time, but keep in mind that TestCaseEvents may interleave.
We have a new timeline formatter which looks pretty.

Anything I missed?

mpkorstanje · 2018-07-05T17:17:48Z

I would like to update the examples with some configuration that makes them run in parallel but that can be done after merging.

boaty82 · 2018-07-05T19:04:42Z

RE: Release notes. Nope can’t think of anything, other than the massive refactoring ;o)

aslakhellesoy · 2018-07-06T11:53:00Z

Hi @boaty82,

Thanks for your making your first contribution to Cucumber, and welcome to the Cucumber committers team! You can now push directly to this repo and all other repos under the cucumber organization! 🍾

In return for this generous offer we hope you will:

✅ Continue to use branches and pull requests. When someone on the core team approves a pull request (yours or someone else's), you're welcome to merge it yourself.
💚 Commit to setting a good example by following and upholding our code of conduct in your interactions with other collaborators and users.
💬 Join the community Slack channel to meet the rest of the team and make yourself at home.
ℹ️ Don't feel obliged to help, just do what you can if you have the time and the energy.
🙋 Ask if you need anything. We're looking for feedback about how to make the project more welcoming, so please tell us!

On behalf of the Cucumber core team,
Aslak Hellesøy
Creator of Cucumber

mpkorstanje · 2018-07-06T12:19:15Z

I've merged this a bit earlier then planned to unclog some stuff. Welcome aboard Boaty.

aslakhellesoy · 2018-07-06T15:10:48Z

Very exciting stuff!!!!

@boaty82 if you haven't already, please have a look at the roadmap.

I think it would be great to extract the timeline formatter to a standalone command line program at some point, when Cucumber-JVM can report results as protobuf messages. That will make the formatter easier to maintain, but more importantly, it will allow any Cucumber implementation to use it.

boaty82 · 2018-07-06T16:03:50Z

If you let me know what branch to work from etc then I’ll happily carry on contributing to this and other parts where and when I can help

jdowneyEverest · 2018-08-14T13:29:05Z

@boaty82 Thanks for your work on this!
I have a couple questions. How are the number of threads to be used determined? Does this only work when running with junit or testNG?

boaty82 · 2018-10-11T10:21:37Z

@aslakhellesoy

I think it would be great to extract the timeline formatter to a standalone command line program at some point, when Cucumber-JVM can report results as protobuf messages. That will make the formatter easier to maintain, but more importantly, it will allow any Cucumber implementation to use it.

Does this still need doing? If so, I'll happily crack on with this in the next few days - just been too busy recently with work and managing to hurt myself 🤕

aslakhellesoy · 2018-10-11T20:46:59Z

Does this still need doing?

Yes - we don't want any formatters to be implemented in Cucumber-{jvm,js,ruby,xxx}. As described in the roadmap, all formatters should be standalone command line programs (preferrably go) that can be used by all implementations using the message protocol.

aslakhellesoy · 2018-10-11T20:47:50Z

Have a look at the dots-formatter to get an idea of a very simple implementation.

boaty82 · 2018-10-16T13:33:17Z

Unless I am mistaken it looks like the data defined in messages protocol will need expanding to include the thread id the test ran within.

mpkorstanje · 2018-10-16T13:54:00Z

Maybe call it executorId. The thread is an implementation detail.

james-bjss · 2018-10-24T22:50:03Z

Unless I am mistaken it looks like the data defined in messages protocol will need expanding to include the thread id the test ran within.

Is this something that would be considered for the current events?

Use case:
I have been looking at Allure recently which has httpclient filters for capturing requests, these are correlated with the current executing step in it's formatter. Unfortunately it doesn't work with Cucumber4/parallel running as the formatter doesn't receive the events until after the test run.

If the thread(executorId) was available in the the event messages it should be possible to match these up after the run (or during if using a concurrent version of the formatter).

Happy to raise this in a new thread if this isn't the right place for this convo.

mpkorstanje · 2018-10-25T11:27:05Z

It'd be better to raise a new thread for that.

boaty82 · 2018-10-25T15:21:41Z

@aslakhellesoy I cannot get the dotsformatter to build, tried various things and all I get is

can't load package: package github.com/cucumber/cucumber-messages-go: no Go files in /Users/xxx/go/src/github.com/cucumber/cucumber-messages-go

kutzi · 2019-05-10T11:07:27Z

core/src/main/java/cucumber/runtime/Runtime.java

+        for (CucumberFeature feature : features) {
+            for (final PickleEvent pickleEvent : compiler.compileFeature(feature)) {
+                if (filters.matchesFilters(pickleEvent)) {
+                    executor.execute(new Runnable() {


If I see this correctly, that will simply swallow any exceptions thrown while executing a pickle - i.e it will log them, dispose the executor thread and then execute the next pickle in a new executor thread.
I have seen this behaviour because there's a bug in my ConcurrentEventListener. It doesn't make the cucumber run fail!

Can you make an issue for that? Ideally we catch all exceptions in the runner and fail the pickle.

lock · 2020-05-20T21:36:22Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

lock · 2020-05-20T21:37:09Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

mpkorstanje and others added 30 commits May 21, 2018 23:44

Extract backend supplier class

4a8937b

Restore api

1acf5cd

Inline constructor

3fcb0f4

Inline constructor

decdc6f

Inline constructor

2849aa4

Inline constructor

d5b73f0

Extract glue supplier

cf7d65f

Inline method

a9c7575

Rename and move RuntimeGlueSupplier to upper level

6142371

Fix compile errors

b13927f

Fix compile tests

1cf94fe

Extract a Runner provider

6353c94

Lift up RunnerSupplier creation to constructors

57a9ae4

Remove unused constructor parameters

6a26cc2

Inline constructor

47c9b8a

Inline constructor

15b32b1

Use local bus instead of Runtime.getBus

0d0d3e5

Move RunnerSupplier to top level

f289b6a

Nitpicking

1e416b1

Remove now unused event bus getter

6245f4e

Move TestRunStartEvent to same level as TestRunFinished

f8923ba

Extract FeatureCompiler

fc51069

Remove runtime from runner test

ebc0b19

Remove runtime from java step definition test

2241333

Remove runtime from hook test

47b623f

Fix compile error

47104cf

Remove getGlue from Runtime

11cdf72

Fix android

a17b215

Initialize plugins before sending start event.

a01604f

Extract getting filters from RuntimeOptions

10a6ce3

aslakhellesoy mentioned this pull request Jul 5, 2018

Parallel Threaded Execution of Scenarios cucumber/common#89

Closed

mpkorstanje added 3 commits July 6, 2018 11:21

Merge branch 'master' into parallel-cukes

8cacb05

Merge branch 'master' into parallel-cukes

f2fc90d

Move resources to io.cucumber package

22e10f9

mpkorstanje merged commit f840358 into cucumber:master Jul 6, 2018

mpkorstanje mentioned this pull request Jul 6, 2018

Support for parallel cukes cucumber/cucumber-android#4

Closed

mpkorstanje added a commit that referenced this pull request Aug 21, 2018

Update changelog w.r.t #1389

1c33961

mpkorstanje mentioned this pull request Aug 28, 2018

Maven BUILD SUCCESSFUL despite exceptions thrown in @Before #1403

Closed

derwasp mentioned this pull request May 2, 2019

Consider parallelizing inside codebuild slaves albumprinter/junit-synnefo#25

Open

kutzi reviewed May 10, 2019

View reviewed changes

kutzi mentioned this pull request May 10, 2019

Cucumber Runtime swallows exceptions in Executor pool threads when run with --threads > 1 #1628

Closed

lock bot locked as resolved and limited conversation to collaborators May 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Parallel cukes #1389

[Core] Parallel cukes #1389

boaty82 commented Jun 9, 2018 •

edited

Loading

mpkorstanje commented Jul 5, 2018

mpkorstanje commented Jul 5, 2018

boaty82 commented Jul 5, 2018 •

edited

Loading

aslakhellesoy commented Jul 6, 2018

mpkorstanje commented Jul 6, 2018

aslakhellesoy commented Jul 6, 2018

boaty82 commented Jul 6, 2018

jdowneyEverest commented Aug 14, 2018

boaty82 commented Oct 11, 2018

aslakhellesoy commented Oct 11, 2018

aslakhellesoy commented Oct 11, 2018

boaty82 commented Oct 16, 2018

mpkorstanje commented Oct 16, 2018

james-bjss commented Oct 24, 2018

mpkorstanje commented Oct 25, 2018

boaty82 commented Oct 25, 2018

kutzi May 10, 2019

mpkorstanje May 10, 2019

kutzi May 10, 2019

lock bot commented May 20, 2020

lock bot commented May 20, 2020

[Core] Parallel cukes #1389

[Core] Parallel cukes #1389

Conversation

boaty82 commented Jun 9, 2018 • edited Loading

Notice

Outline

Main changes

Added --threads argument to runtime options

New formatter introduced TimelineFormatter

Types of changes

Checklist:

mpkorstanje commented Jul 5, 2018

mpkorstanje commented Jul 5, 2018

boaty82 commented Jul 5, 2018 • edited Loading

aslakhellesoy commented Jul 6, 2018

mpkorstanje commented Jul 6, 2018

aslakhellesoy commented Jul 6, 2018

boaty82 commented Jul 6, 2018

jdowneyEverest commented Aug 14, 2018

boaty82 commented Oct 11, 2018

aslakhellesoy commented Oct 11, 2018

aslakhellesoy commented Oct 11, 2018

boaty82 commented Oct 16, 2018

mpkorstanje commented Oct 16, 2018

james-bjss commented Oct 24, 2018

mpkorstanje commented Oct 25, 2018

boaty82 commented Oct 25, 2018

kutzi May 10, 2019

Choose a reason for hiding this comment

mpkorstanje May 10, 2019

Choose a reason for hiding this comment

kutzi May 10, 2019

Choose a reason for hiding this comment

lock bot commented May 20, 2020

lock bot commented May 20, 2020

boaty82 commented Jun 9, 2018 •

edited

Loading

Added `--threads` argument to runtime options

New formatter introduced `TimelineFormatter`

boaty82 commented Jul 5, 2018 •

edited

Loading