Support some level of parallelization in junit-vintage-engine #2229

sarod · 2020-03-31T09:53:38Z

Goal

Add support for some level of parallelization inside the Junit Vintage engine.
Ideally the level of configuration should be similar to maven surefire parallel options.

Why

When including junit 5 tests in a project containing junit4 tests the natural path is to use junit 5 platform with the jupiter and vintage engines.

However this causes problem with parallelization.
In a project with Junit4 tests only maven surefire level parallelization works fine. (See https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html).
For Junit5 tests the jupiter engine parallel execution works great.

However when a project contains a mix of Junit4 and Junit 5 there is no great way to configure parallelism:

Surefire parallel does not work with Junit Platform Provider so it's not an option
Surefire forkCount or gradle maxParallelForks can be used but:
- It's not available at all when using tycho surefire
- It introduces some memory and execution time overhead compared to surefire parallel or junit 5 parallel.
- It provides less control than Junit 5 parallel execution options.
Combining fork + junit 5 parallel execution leads to more parallel executions than expected when executing junit 5 tests because parallelization is done at two level.

Another option is to configure two distinct test tasks one for junit 4 using surefire parallel and one for junit 5 but it kind of defeats the point of junit platform.

Deliverables

Not sure

koretzki · 2020-04-16T08:51:56Z

thanks, @sarod for rising this.
I tried to fully embrace Junit 5 but had issues working it using the maven failsafe plugin 2.22.2-3.0.0-M4 (Junit 5.3-5.6.2), but I never could make parallelism to work even with settings that worked in maven surefire (in 3 ways to set them).
The issue was the mixed Junit 4,5 testing environment. I moved them all to Junit 5 and it worked like magic in maven failsafe 2.2.22 with parallel execution.

vab2048 · 2020-08-29T13:13:56Z

Are there any known workarounds to achieve this currently?
I would like to run the junit4 classes in parallel and methods in the same thread.

marcphilipp · 2020-09-13T11:31:11Z

I would like to run the junit4 classes in parallel and methods in the same thread.

That should be fairly straightforward to implement. Basically, one would have to call RunnerExecutor concurrently and read a few configuration parameters to enable parallel execution and configure the number of threads:

junit5/junit-vintage-engine/src/main/java/org/junit/vintage/engine/VintageTestEngine.java

Lines 77 to 84 in 3f7fed6

    
           private void executeAllChildren(RunnerExecutor runnerExecutor, TestDescriptor engineDescriptor) { 
        
           	// @formatter:off 
        
           	engineDescriptor.getChildren() 
        
           			.stream() 
        
           			.map(RunnerTestDescriptor.class::cast) 
        
           			.forEach(runnerExecutor::execute); 
        
           	// @formatter:on 
        
           }

This change updates the JUnit Vintage Test Engine to read the parallel configuration parameters, and if parallel execution is enabled, creates a thread pool for executing the test descriptors concurrently. The approach for configuring the thread pool emulates the logic used by the Jupiter Test Engine. Issue: junit-team#2229

This updates the user guide and release notes with details on concurrent test execution. Issue: junit-team#2229

This change updates the JUnit Vintage Test Engine to read the parallel configuration parameters, and if parallel execution is enabled, creates a thread pool for executing the test descriptors concurrently. The approach for configuring the thread pool emulates the logic used by the Jupiter Test Engine. Issue: junit-team#2229

This updates the user guide and release notes with details on concurrent test execution. Issue: junit-team#2229

This addresses code review feedback. The configuration property prefix is now `junit.vintage.execution.parallel`. The user guide reflects this. The `ForkJoinPool` that supports concurrent execution more closely matches the one used by the JUnit platform engine. Lastly code review feedback was incorporated to remove unnecesary or redundant code and to conform better to coding standards. Issue: junit-team#2229

This corrects the asciidoc formatting for the Vintage Parallelization usage guide. Issue: junit-team#2229

This change updates the JUnit Vintage Test Engine to read the parallel configuration parameters, and if parallel execution is enabled, creates a thread pool for executing the test descriptors concurrently. The approach for configuring the thread pool emulates the logic used by the Jupiter Test Engine. Issue: junit-team#2229

This updates the user guide and release notes with details on concurrent test execution. Issue: junit-team#2229

This addresses code review feedback. The configuration property prefix is now `junit.vintage.execution.parallel`. The user guide reflects this. The `ForkJoinPool` that supports concurrent execution more closely matches the one used by the JUnit platform engine. Lastly code review feedback was incorporated to remove unnecesary or redundant code and to conform better to coding standards. Issue: junit-team#2229

This corrects the asciidoc formatting for the Vintage Parallelization usage guide. Issue: junit-team#2229

This commit _attempts_ to fix concurrency unit test failures observed on Windows Server 2019 10.0 amd64 with JVM 15.0.1 (Azul Systems, Inc. 15.0.1+8) and Linux 5.4.0-1031-azure amd64 with JVM 15 (Oracle Corporation 15+36-1562). It does this by using a `CyclicBarrier` to ensure test methods run concurrently when expected to. This is the same logic used by `ParallelExecutionIntegrationTests`. Note, I was not able to reproduce the build failures locally. Issue: junit-team#2229

This change updates the JUnit Vintage Test Engine to read the parallel configuration parameters, and if parallel execution is enabled, creates a thread pool for executing the test descriptors concurrently. The approach for configuring the thread pool emulates the logic used by the Jupiter Test Engine. Issue: junit-team#2229

This updates the user guide and release notes with details on concurrent test execution. Issue: junit-team#2229

This addresses code review feedback. The configuration property prefix is now `junit.vintage.execution.parallel`. The user guide reflects this. The `ForkJoinPool` that supports concurrent execution more closely matches the one used by the JUnit platform engine. Lastly code review feedback was incorporated to remove unnecesary or redundant code and to conform better to coding standards. Issue: junit-team#2229

This corrects the asciidoc formatting for the Vintage Parallelization usage guide. Issue: junit-team#2229

This commit _attempts_ to fix concurrency unit test failures observed on Windows Server 2019 10.0 amd64 with JVM 15.0.1 (Azul Systems, Inc. 15.0.1+8) and Linux 5.4.0-1031-azure amd64 with JVM 15 (Oracle Corporation 15+36-1562). It does this by using a `CyclicBarrier` to ensure test methods run concurrently when expected to. This is the same logic used by `ParallelExecutionIntegrationTests`. Note, I was not able to reproduce the build failures locally. Issue: junit-team#2229

This addresses code review feedback. Specifically, it extracts a factory for creating `ForkJoinPool` instances. It also consolidates the documentation on customising pool creation. Lastly, the unit tests are refactored to simplify future enhancements. Issue: junit-team#2229

kishoretak · 2022-05-27T10:04:34Z

@marcphilipp As you suggested running test classes in parallel with Vintage Engine is straightforward and the same is working fine. Is there any suggestion for running the test methods from the same class in parallel? I tried the solution mentioned here but it only supports test classes in parallel.

nedtwigg · 2023-08-31T22:58:03Z

I might be missing something, but so far as I can tell, it is not possible to run tests in parallel with junit-vintage. At the very least, junit.jupiter.execution.parallel.enabled=true does not seem to be effective. The PR #2449 looks small and high quality, I hope it or something similar gets merged someday.

If implementing the feature is very difficult, it might be nice if junit-vintage looked for junit.jupiter.execution.parallel.enabled=true and warned that it has no effect.

nedtwigg · 2023-09-01T20:07:29Z

For anybody else who wants this, I was able to get #2449 working on JitPack, took some tricks, you can use it like so:

maven { url 'https://jitpack.io' }

// testImplementation "org.junit.jupiter:junit-jupiter-api:${VER_junit5}"
// testImplementation "org.junit.vintage:junit-vintage-engine:${VER_junit5}"
testImplementation('com.github.diffplug.l0s-junit5:junit-jupiter-api:7786d92011e4c68255434b6f0f31cfd03841f24b')
testImplementation('com.github.diffplug.l0s-junit5:junit-vintage-engine:7786d92011e4c68255434b6f0f31cfd03841f24b')

test {
  useJUnitPlatform {}
  systemProperty 'junit.vintage.execution.parallel.enabled', 'true'
}

Swapping that little dep took a painpoint testsuite from 60 seconds to 10 seconds.

YongGoose · 2024-11-12T16:47:51Z

@marcphilipp

I’m interested in this issue.
If it’s still open for contributions, would it be okay for me to work on it?

marcphilipp · 2024-11-13T17:10:09Z

Yes, but let's discuss the plan first. What did you have in mind? Do sth. like I outlined in #2229 (comment)?

YongGoose · 2024-11-16T15:46:15Z

Yes, but let's discuss the plan first. What did you have in mind? Do sth. like I outlined in #2229 (comment)?

Yes, I think moving forward with the approach outlined in #2229 (comment) is a good idea.

I’ve implemented it in my personal workspace. Could you please take a look and confirm if the direction I took is correct?
Here’s the commit : YongGoose@e6bf957

In summary, parallelization has been applied to the executeAllChildren method:
- Parallelism was implemented using ExecutorService, and CountDownLatch was used to wait for the completion of all threads.
Additionally, logs have been generated using a logger to capture key events and situations.

WDYT?

I haven’t yet worked on the part that reads configuration parameters, such as whether parallel execution should be enabled or the number of threads.

I plan to address that once the design is finalized.🙂

marcphilipp · 2024-11-17T16:23:25Z

I’ve implemented it in my personal workspace. Could you please take a look and confirm if the direction I took is correct?

Yes, that's the right place to add this. I think it'll be easier to discuss details such as the following on a draft PR.

I don't think using a ForkJoinPool as the Executor has any benefits here, does it?
Shouldn't we just wait for all futures (e.g. using CompletableFuture.allOf() rather than using a CountDownLatch?

YongGoose · 2024-11-18T13:56:49Z

I don't think using a ForkJoinPool as the Executor has any benefits here, does it?

Hmm... I think ForkJoinPool is more suitable for cases where tasks can be divided into smaller units.
However, in the current code, since the tests are executed independently, there’s no way to split the tasks further.
That's why I understood there would be no benefit to using ForkJoinPool.

Did I understand it correctly? 🤔

I’ve changed it to use newFixedThreadPool as the Executor instead.

Shouldn't we just wait for all futures (e.g. using CompletableFuture.allOf() rather than using a CountDownLatch?

Sounds Good!!, I think that's a good idea.

YongGoose · 2024-11-18T15:10:18Z

@marcphilipp

I think it'll be easier to discuss details such as the following on a draft PR.

I've created a draft PR to make it easier to discuss details!

Support some level of parallelization in junit-vintage-engine #4135

marcphilipp · 2024-11-19T07:50:23Z

Did I understand it correctly?

Yes, you did.

I've created a draft PR to make it easier to discuss details!

Thanks! 👍

Issue: junit-team#2229 Signed-off-by: yongjunhong <kevin0928@naver.com>

sbrannen added component: Vintage theme: concurrency type: new feature labels Mar 31, 2020

marcphilipp added the status: waiting-for-interest label Apr 9, 2020

marcphilipp mentioned this issue Aug 11, 2020

Support parallel execution of legacy Junit Tests #515

Closed

marcphilipp added the up-for-grabs label Sep 13, 2020

marcphilipp added this to the General Backlog milestone Sep 13, 2020

l0s mentioned this issue Oct 15, 2020

Support parallelization in junit-vintage-engine #2449

Closed

7 tasks

l0s added a commit to l0s/junit5 that referenced this issue Oct 18, 2020

Document Vintage parallel execution support

28c96e7

This updates the user guide and release notes with details on concurrent test execution. Issue: junit-team#2229

l0s added a commit to l0s/junit5 that referenced this issue Oct 26, 2020

Document Vintage parallel execution support

a2ed511

This updates the user guide and release notes with details on concurrent test execution. Issue: junit-team#2229

l0s added a commit to l0s/junit5 that referenced this issue Oct 26, 2020

Fix doc format for Vintage Parallelization

14085ec

This corrects the asciidoc formatting for the Vintage Parallelization usage guide. Issue: junit-team#2229

l0s added a commit to l0s/junit5 that referenced this issue Oct 27, 2020

Document Vintage parallel execution support

d493f11

This updates the user guide and release notes with details on concurrent test execution. Issue: junit-team#2229

l0s added a commit to l0s/junit5 that referenced this issue Oct 27, 2020

Fix doc format for Vintage Parallelization

1ba962c

This corrects the asciidoc formatting for the Vintage Parallelization usage guide. Issue: junit-team#2229

l0s added a commit to l0s/junit5 that referenced this issue Oct 28, 2020

Document Vintage parallel execution support

52e28ef

This updates the user guide and release notes with details on concurrent test execution. Issue: junit-team#2229

l0s added a commit to l0s/junit5 that referenced this issue Oct 28, 2020

Fix doc format for Vintage Parallelization

a5fb73f

This corrects the asciidoc formatting for the Vintage Parallelization usage guide. Issue: junit-team#2229

marcphilipp linked a pull request Oct 30, 2020 that will close this issue

Support parallelization in junit-vintage-engine #2449

Closed

7 tasks

marcphilipp modified the milestones: General Backlog, 5.8 M1 Oct 30, 2020

marcphilipp modified the milestones: 5.8 M1, 5.8 M2/RC1 Feb 7, 2021

mpkorstanje mentioned this issue Jun 6, 2021

Parallel execution with surefire and failsafe doesn't work with junit-vintage-engine cucumber/docs#627

Open

marcphilipp removed this from the 5.8 RC1 milestone Aug 15, 2021

marcphilipp added status: in progress and removed status: waiting-for-interest labels Nov 7, 2021

SheliakLyr mentioned this issue May 23, 2024

Parallel suites/tests execution scalatest/scalatestplus-junit5#13

Open

marcphilipp added this to Sovereign Tech Fund Jul 19, 2024

github-project-automation bot moved this to Todo in Sovereign Tech Fund Jul 19, 2024

marcphilipp removed the status: in progress label Nov 13, 2024

YongGoose linked a pull request Nov 18, 2024 that will close this issue

Support some level of parallelization in junit-vintage-engine #4135

Draft

6 tasks

marcphilipp added the status: in progress label Nov 22, 2024

YongGoose added a commit to YongGoose/junit5 that referenced this issue Dec 5, 2024

Add configuration parameter

e0312cc

Issue: junit-team#2229 Signed-off-by: yongjunhong <kevin0928@naver.com>

YongGoose added a commit to YongGoose/junit5 that referenced this issue Dec 5, 2024

Stored the interrupted state in a local variable

dfeb1cd

Issue: junit-team#2229 Signed-off-by: yongjunhong <kevin0928@naver.com>

YongGoose added a commit to YongGoose/junit5 that referenced this issue Dec 6, 2024

Remove try-catch

f9af925

Issue: junit-team#2229 Signed-off-by: yongjunhong <kevin0928@naver.com>

YongGoose added a commit to YongGoose/junit5 that referenced this issue Dec 15, 2024

Add integration test

de8fbce

Issue: junit-team#2229 Signed-off-by: yongjunhong <kevin0928@naver.com>

YongGoose added a commit to YongGoose/junit5 that referenced this issue Dec 15, 2024

Add integration test

1484347

Issue: junit-team#2229 Signed-off-by: yongjunhong <kevin0928@naver.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support some level of parallelization in junit-vintage-engine #2229

Support some level of parallelization in junit-vintage-engine #2229

sarod commented Mar 31, 2020 •

edited by marcphilipp

Loading

koretzki commented Apr 16, 2020 •

edited

Loading

vab2048 commented Aug 29, 2020

marcphilipp commented Sep 13, 2020

kishoretak commented May 27, 2022

nedtwigg commented Aug 31, 2023

nedtwigg commented Sep 1, 2023

YongGoose commented Nov 12, 2024

marcphilipp commented Nov 13, 2024

YongGoose commented Nov 16, 2024 •

edited

Loading

marcphilipp commented Nov 17, 2024

YongGoose commented Nov 18, 2024

YongGoose commented Nov 18, 2024

marcphilipp commented Nov 19, 2024

Support some level of parallelization in junit-vintage-engine #2229

Support some level of parallelization in junit-vintage-engine #2229

Comments

sarod commented Mar 31, 2020 • edited by marcphilipp Loading

Goal

Why

Deliverables

koretzki commented Apr 16, 2020 • edited Loading

vab2048 commented Aug 29, 2020

marcphilipp commented Sep 13, 2020

kishoretak commented May 27, 2022

nedtwigg commented Aug 31, 2023

nedtwigg commented Sep 1, 2023

YongGoose commented Nov 12, 2024

marcphilipp commented Nov 13, 2024

YongGoose commented Nov 16, 2024 • edited Loading

marcphilipp commented Nov 17, 2024

YongGoose commented Nov 18, 2024

YongGoose commented Nov 18, 2024

marcphilipp commented Nov 19, 2024

sarod commented Mar 31, 2020 •

edited by marcphilipp

Loading

koretzki commented Apr 16, 2020 •

edited

Loading

YongGoose commented Nov 16, 2024 •

edited

Loading