Parallelism value ignored for the fixed strategy #2273

rbok78 · 2020-04-25T10:35:15Z

Steps to reproduce

git clone git@github.com:rbok78/framework-template.git
cd framework-template
mvn clean test -P parallel-execution

Actual outcome

All 4 test methods are kicked off at the same time.

Expected outcome

I use the following configuration for the Maven Surefire Plugin. I expect the number of test methods running in parallel is the same as the parallelism value. I've tried to set it to 1, 2, and 3 but still all 4 test methods run in parallel.

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = concurrent
junit.jupiter.execution.parallel.config.strategy = fixed
junit.jupiter.execution.parallel.config.fixed.parallelism = 1

Context

Linux 5.3.0-46-generic x86_64
Ubuntu 19.10
OpenJDK 11.0.7
Apache Maven 3.6.1
Maven Surefire Plugin 3.0.0-M4
Jupiter 5.7.0-M1
IntelliJ IDEA 2020.1 (Community Edition)

Deliverables

n/a

The text was updated successfully, but these errors were encountered:

marcphilipp · 2020-04-25T11:07:13Z

Thanks for reporting the issue and providing a reproducer! 👍

The parallelism setting is no guarantee that there will be at most one test executing simultaneously. It has the same semantics as ForkJoinPool.getParallelism(), i.e. a potentially blocking operation may lead to another thread being used in order to ensure progress. If you don't want your test methods to run in parallel, please use the following configuration:

junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent

rbok78 · 2020-05-05T04:28:09Z

Thanks for the answer. I've done some reading about the ForkJoinPool and I understand that the number of threads created can be higher than the parallelism setting. Still I would expect that that number of running tests (concurrently running threads) would follow the parallelism setting. I plan to use JUnit5 for my UI automation where I can have hundreds of tests (test methods that can be within a single class or multiple classes). I can't run all at the same time as it would exhaust all hardware resources hence I need to be able to control how many tests run concurrently. I stripped my code to the bare minimum and discovered a weird behaviour. For the parallelism setting:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = concurrent
junit.jupiter.execution.parallel.config.strategy = fixed
junit.jupiter.execution.parallel.config.fixed.parallelism = 2

Without creating an instance of the ChromeDriver anywhere in my code, the number of test methods that run concurrently follows the parallelism setting:

11:58:19.135 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Set up completed for pageTitleTest
11:58:19.135 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Set up completed for failTestWithScreenshot
11:58:19.164 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Tear down completed for failTestWithScreenshot
11:58:19.203 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Tear down completed for pageTitleTest
11:58:19.305 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Set up completed for waitForRequestTest
11:58:19.305 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Set up completed for matchTemplateTest
11:58:19.319 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Tear down completed for matchTemplateTest
11:58:19.322 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Tear down completed for waitForRequestTest

However, when I create an instance of the ChromeDriver e.g. in the method annotated with BeforeEach or in the test method directly, I can see all test methods are kicked off at the same time:

12:07:13.622 [ForkJoinPool-1-worker-13] INFO  org.example.utils.BaseTest - Set up completed for getDomHighResTimeStampTest
12:07:13.619 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Set up completed for pageTitleTest
12:07:13.620 [ForkJoinPool-1-worker-7] INFO  org.example.utils.BaseTest - Set up completed for linkTextTest
12:07:13.615 [ForkJoinPool-1-worker-5] INFO  org.example.utils.BaseTest - Set up completed for matchTemplateTest
12:07:13.621 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Set up completed for failTestWithScreenshot
12:07:13.693 [ForkJoinPool-1-worker-15] INFO  org.example.utils.BaseTest - Set up completed for waitForNetworkIdleTest
12:07:13.698 [ForkJoinPool-1-worker-9] INFO  org.example.utils.BaseTest - Set up completed for paragraphTextTest
12:07:13.704 [ForkJoinPool-1-worker-11] INFO  org.example.utils.BaseTest - Set up completed for waitForRequestTest
12:07:13.707 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Tear down completed for failTestWithScreenshot
12:07:13.736 [ForkJoinPool-1-worker-15] INFO  org.example.utils.BaseTest - Tear down completed for waitForNetworkIdleTest
12:07:13.736 [ForkJoinPool-1-worker-13] INFO  org.example.utils.BaseTest - Tear down completed for getDomHighResTimeStampTest
12:07:13.794 [ForkJoinPool-1-worker-11] INFO  org.example.utils.BaseTest - Tear down completed for waitForRequestTest
12:07:13.818 [ForkJoinPool-1-worker-7] INFO  org.example.utils.BaseTest - Tear down completed for linkTextTest
12:07:13.823 [ForkJoinPool-1-worker-5] INFO  org.example.utils.BaseTest - Tear down completed for matchTemplateTest
12:07:13.828 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Tear down completed for pageTitleTest
12:07:13.830 [ForkJoinPool-1-worker-9] INFO  org.example.utils.BaseTest - Tear down completed for paragraphTextTest

The only way to fix this, which I have found so far, is to use a custom strategy. With this one in place, I can control the number of running tests reliably. Here is the parallelism setting:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = concurrent
junit.jupiter.execution.parallel.config.strategy = custom
junit.jupiter.execution.parallel.config.custom.class = org.example.CustomStrategy

And here is the custom strategy with the parallelism set to 2:

package org.example;

import org.junit.platform.engine.ConfigurationParameters;
import org.junit.platform.engine.support.hierarchical.ParallelExecutionConfiguration;
import org.junit.platform.engine.support.hierarchical.ParallelExecutionConfigurationStrategy;

public class CustomStrategy implements ParallelExecutionConfiguration, ParallelExecutionConfigurationStrategy {
    @Override
    public int getParallelism() {
        return 2;
    }

    @Override
    public int getMinimumRunnable() {
        return 2;
    }

    @Override
    public int getMaxPoolSize() {
        return 2;
    }

    @Override
    public int getCorePoolSize() {
        return 2;
    }

    @Override
    public int getKeepAliveSeconds() {
        return 30;
    }

    @Override
    public ParallelExecutionConfiguration createConfiguration(final ConfigurationParameters configurationParameters) {
        return this;
    }
}

I will try to reach out to the ChromeDriver developers if they can help to explain the current behaviour.

marcphilipp · 2020-05-05T11:50:57Z

Thanks for the detailed analysis and posting the workaround!

My only hunch is that the ChromeDriver instantiation somehow behaves differently when invoked from a ForkJoinWorkerThread. If ForkJoinPool.managedBlock is used somehow behind the scenes the ForkJoinPool will preventively start another thread to prevent blocking.

kobynet · 2021-02-09T16:47:46Z

I can confirm we are having the same issue, not using any ChromeDriver at all. using OpenJDK 11.0.9, Jupiter 5.7.0 and Maven Surefire Plugin 3.0.0-M5

I can confirm the workaround showed in #2273 (comment) works for us as well.

Our confguration:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent
junit.jupiter.execution.parallel.config.strategy = fixed
junit.jupiter.execution.parallel.config.fixed.parallelism = 2

What we see is all classes are triggered at once and not just 2 at a time.

titusfortner · 2021-04-05T19:05:50Z

@rbok78 / @kobynet were you by chance using Selenium 4 when you had this issue?

shs96c · 2021-04-27T13:35:22Z

If I understand the above comments correctly, does this mean that if any code under test (either first or via dependency) calls ForkJoinPool.managedBlock the parallelism count used by junit is basically ignored? If that's the case, then I'd suggest we reopen this issue.

Put another way, something in Selenium's deps may have tickled this issue, but it's something that could affect any JUnit 5 user.

shs96c · 2021-04-27T13:38:24Z

Having a hunt in the JDK, this should impact anyone who's using Process.onExit(), or CompleteableFuture.get(), the latter of which is used pretty widely.

kobynet · 2021-04-28T08:04:20Z

@titusfortner We're note using any selenium deps.

titusfortner · 2021-04-28T14:18:45Z

Of course, you even said that in the first line of your comment; sorry. :)

I guess I should also say that the proposed solution only seems to work if I want to limit the number of parallel executions to a very small number. When increasing it to a reasonable number, it does not work.

Here's a sample project showing the problem, with the relevant config values: https://github.com/titusfortner/bug9359/blob/main/pom.xml#L62-L79

I really want to be able to recommend JUnit 5 to my clients because I love the new syntax/functionality, but I can't until this gets sorted.

marcphilipp · 2021-05-13T12:15:33Z

I'm all ears if anyone has a proposal how to resolve this. The only thing I've come up with would be to wrap the execution of each test in another, non-ForkJoinPool thread but that would double the number of required threads and have performance implications.

titusfortner · 2021-05-13T14:37:45Z

Out of curiosity, how does JUnit 4 / maven surefire approach it? That has always worked extremely well for our purposes.

titusfortner · 2021-07-13T15:46:45Z

@marcphilipp would it work to just use something like AtomicInteger to limit the total number of running threads?

marcphilipp · 2021-07-14T17:50:58Z

You can already limit the number of threads by implementing a custom ParallelExecutionConfigurationStrategy that returns a ParallelExecutionConfiguration where getMaxPoolSize() returns the limit. However, since a ForkJoinPool is used in the background a small number might lead to blockages in combination with @ResourceLock and other synchronization mechanisms.

titusfortner · 2021-07-14T21:03:52Z

well, damn it all. I was using a custom strategy and it wasn't working in Selenium beta 3, but guess what? It works in Selenium beta 4.

Thanks for your reply, @marcphilipp !

sangameshwar-balur · 2022-02-09T16:16:39Z

@rbok78 / @kobynet were you by chance using Selenium 4 when you had this issue?

I am having same issue when I am using Selenium 4, here what could be the issue with Selenium 4. Any suggestion to resolve the issue with Selenium 4.

titusfortner · 2022-02-09T16:40:36Z

So, it turns out that it works with Java 11, but not Java 8 or Java 17. Discussion from Selenium here: SeleniumHQ/selenium#10113
Which references open JUnit 5 issues: #2787 & #2792

harrietwamala · 2023-02-08T21:09:12Z

In my case junit parameters did not work, what worked was only the cucumber parameters below:
cucumber.execution.parallel.enabled = true
cucumber.execution.parallel.mode.default = same_thread
cucumber.execution.parallel.mode.classes.default = concurrent
cucumber.execution.parallel.config.strategy = fixed
cucumber.execution.parallel.config.fixed.parallelism = 2

titusfortner · 2023-02-09T22:47:47Z

fwiw, I got this working for today on Java 8 with JUnit 4.9.2
I'm not sure how/why it works on Java 8, but 🤷‍♂️
Method level parallelization, not just class level.

<configurationParameters>
    junit.jupiter.execution.parallel.enabled = true
    junit.jupiter.execution.parallel.mode.default = concurrent
    junit.jupiter.execution.parallel.config.strategy = fixed
    junit.jupiter.execution.parallel.config.fixed.parallelism = 10
    junit.jupiter.execution.parallel.config.fixed.max-pool-size = 10
</configurationParameters>

BalurQA · 2023-02-13T19:27:03Z

@titusfortner : 4.9.2 is the version of Junit 5 or Junit4 . ?
I do not see this version 4.9.2 in Maven repository for either. Could you please help with maven gradle dependency details from maven repo.

titusfortner · 2023-02-15T13:11:26Z

Sorry, 5.9.2

BalurQA · 2023-02-15T15:10:48Z

@titusfortner I did try with 5.9.2 unfortunately it is not working for me. If possible could you please share list of dependencies used in the project or working git repo.

Below are list dependencies used in my project framework. Kindly let me know of any concerns with dependencies used

dependencies {
//    testImplementation 'org.junit.jupiter:junit-jupiter:5.9.2'
//    testImplementation('org.junit.jupiter:junit-jupiter:5.5.1')
    testImplementation(platform('org.junit:junit-bom:5.9.2'))
    testImplementation('org.junit.jupiter:junit-jupiter')
// https://mvnrepository.com/artifact/io.github.bonigarcia/selenium-jupiter
    implementation 'io.github.bonigarcia:selenium-jupiter:4.3.1'
    testImplementation 'org.hamcrest:hamcrest:2.1'
    testImplementation 'org.hamcrest:hamcrest-library:2.1'
//    testCompile 'org.junit.jupiter:junit-jupiter-api:5.9.2'
//	testRuntime 'org.junit.jupiter:junit-jupiter-engine:5.9.2'
    testRuntime 'org.junit.platform:junit-platform-launcher:1.4.2'
    testRuntime 'org.junit.jupiter:junit-jupiter-engine:5.9.2'
    compile group: 'org.jsoup', name: 'jsoup', version: '1.9.2'
    compile group: 'org.apache.poi', name: 'poi', version: '3.9'
    compile group: 'org.apache.poi', name: 'poi-contrib', version: '3.6'
//    compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version: '3.141.59'
    // https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java
    implementation 'org.seleniumhq.selenium:selenium-java:4.8.0'
    compile group: 'io.qameta.allure', name: 'allure-junit5', version: '2.11.0'
    compile group: 'org.apache.pdfbox', name: 'pdfbox', version: '2.0.16'
    compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.17.1'
    compile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.17.1'
    compile group: 'io.qameta.allure', name: 'allure-gradle', version: '2.7.0'
    compile group: 'org.apache.poi', name: 'poi-ooxml', version: '3.7'
    compile group: 'org.apache.xmlbeans', name: 'xmlbeans', version: '4.0.0'
    compile group: 'com.automation-remarks', name: 'video-recorder-junit5', version: '2.+'
    compile('com.assertthat:selenium-shutterbug:0.9.2')
    compile 'org.slf4j:slf4j-nop:1.7.25'
//	compile 'com.microsoft.ews-java-api:ews-java-api:2.0'
    compile group: 'org.jsoup', name: 'jsoup', version: '1.9.2'
    compile group: 'com.surevine', name: 'ews-java-api', version: '2.3'
    compile group: 'ru.yandex.qatools.ashot', name: 'ashot', version: '1.5.4'
//    implementation group: 'io.github.bonigarcia', name: 'webdrivermanager', version: '4.3.1'
    implementation group: 'javax.mail', name: 'mail', version: '1.4.7'
    implementation group: 'javax.mail', name: 'javax.mail-api', version: '1.6.2'
    runtimeClasspath group: 'javax.mail', name: 'javax.mail-api', version: '1.6.2'
    implementation group: 'org.json', name: 'json', version: '20201115'
    implementation group: 'com.googlecode.json-simple', name: 'json-simple', version: '1.1.1'
    implementation group: 'com.microsoft.sqlserver', name: 'mssql-jdbc', version: '7.2.2.jre8'
    implementation 'net.java.dev.jna:jna-platform:5.9.0'
    // https://mvnrepository.com/artifact/org.junit.platform/junit-platform-engine
    testImplementation 'org.junit.platform:junit-platform-engine:1.9.0'
    compile group: 'com.epam.healenium', name: 'healenium-web', version: '3.2.5'
    // https://mvnrepository.com/artifact/com.fasterxml.jackson/jackson-bom
    implementation 'com.fasterxml.jackson:jackson-bom:2.14.1'
    compile group: 'org.mongodb', name: 'mongo-java-driver', version: '3.12.4'
}

Kolesnikov-Vladislav · 2023-05-30T09:27:16Z

@BalurQA

У меня с 5.9.2 работает =)

Мои зависимости:

            "com.codeborne:selenide:6.12.4",
            "org.junit.jupiter:junit-jupiter:5.9.2",
            "org.aspectj:aspectjweaver:1.9.19",
            "org.slf4j:slf4j-simple:2.0.7",
            "org.aeonbits.owner:owner:1.0.12",
            "com.github.javafaker:javafaker:1.0.2",
            "io.qameta.allure:allure-selenide:2.21.0",
            "io.qameta.allure:allure-rest-assured:2.21.0",
            "org.testng:testng:7.1.0",
            "com.codeborne:selenide-proxy:6.12.4"

paajake · 2024-04-08T13:59:41Z

fwiw, I got this working for today on Java 8 with JUnit 4.9.2 I'm not sure how/why it works on Java 8, but 🤷‍♂️ Method level parallelization, not just class level.
<configurationParameters>
    junit.jupiter.execution.parallel.enabled = true
    junit.jupiter.execution.parallel.mode.default = concurrent
    junit.jupiter.execution.parallel.config.strategy = fixed
    junit.jupiter.execution.parallel.config.fixed.parallelism = 10
    junit.jupiter.execution.parallel.config.fixed.max-pool-size = 10
</configurationParameters>

This worked for me using JDK 21, with JUnit 5.10.2, the fixed.parallelism value still seems ignored, but the fixed.max-pool-size variable seems to keep things in check

mbuchner · 2024-05-09T00:35:10Z

I can confirm that this seams to be a Java 17 issue - upgraded now to Java 21 and used the following configuration to run always 2 classes in parallel:

    -Djunit.jupiter.execution.parallel.enabled=true 
    -Djunit.jupiter.execution.parallel.mode.default=same_thread 
    -Djunit.jupiter.execution.parallel.mode.classes.default=concurrent 
    -Djunit.jupiter.execution.parallel.config.fixed.max-pool-size=2 
    -Djunit.jupiter.execution.parallel.config.strategy=fixed 
    -Djunit.jupiter.execution.parallel.config.fixed.parallelism=2

Quarkus, jUnit 3.8.2, Selenide 7.2.3, Selenium Grid with 2 Chrome nodes (Concurrency 1)

marcphilipp closed this as completed Apr 25, 2020

sbrannen added the status: works-as-designed label Apr 25, 2020

titusfortner mentioned this issue Apr 5, 2021

Can no longer limit the number of parallel sessions from JUnit 5 SeleniumHQ/selenium#9359

Closed

pujagani mentioned this issue May 21, 2021

Selenium 4.0.0-beta-3 - Parallel Threads with JUnit5 and JDK11/16 does not work SeleniumHQ/selenium#9475

Closed

big-andy-coates mentioned this issue May 13, 2022

Speed up the build with a bit of parallel testing... creek-service/creek-json-schema-gradle-plugin#10

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelism value ignored for the fixed strategy #2273

Parallelism value ignored for the fixed strategy #2273

rbok78 commented Apr 25, 2020

marcphilipp commented Apr 25, 2020

rbok78 commented May 5, 2020

marcphilipp commented May 5, 2020

kobynet commented Feb 9, 2021 •

edited

Loading

titusfortner commented Apr 5, 2021

shs96c commented Apr 27, 2021

shs96c commented Apr 27, 2021

kobynet commented Apr 28, 2021

titusfortner commented Apr 28, 2021

marcphilipp commented May 13, 2021

titusfortner commented May 13, 2021

titusfortner commented Jul 13, 2021

marcphilipp commented Jul 14, 2021

titusfortner commented Jul 14, 2021

sangameshwar-balur commented Feb 9, 2022

titusfortner commented Feb 9, 2022

harrietwamala commented Feb 8, 2023

titusfortner commented Feb 9, 2023

BalurQA commented Feb 13, 2023

titusfortner commented Feb 15, 2023

BalurQA commented Feb 15, 2023 •

edited by sbrannen

Loading

Kolesnikov-Vladislav commented May 30, 2023

paajake commented Apr 8, 2024

mbuchner commented May 9, 2024

Parallelism value ignored for the fixed strategy #2273

Parallelism value ignored for the fixed strategy #2273

Comments

rbok78 commented Apr 25, 2020

Steps to reproduce

Actual outcome

Expected outcome

Context

Deliverables

marcphilipp commented Apr 25, 2020

rbok78 commented May 5, 2020

marcphilipp commented May 5, 2020

kobynet commented Feb 9, 2021 • edited Loading

titusfortner commented Apr 5, 2021

shs96c commented Apr 27, 2021

shs96c commented Apr 27, 2021

kobynet commented Apr 28, 2021

titusfortner commented Apr 28, 2021

marcphilipp commented May 13, 2021

titusfortner commented May 13, 2021

titusfortner commented Jul 13, 2021

marcphilipp commented Jul 14, 2021

titusfortner commented Jul 14, 2021

sangameshwar-balur commented Feb 9, 2022

titusfortner commented Feb 9, 2022

harrietwamala commented Feb 8, 2023

titusfortner commented Feb 9, 2023

BalurQA commented Feb 13, 2023

titusfortner commented Feb 15, 2023

BalurQA commented Feb 15, 2023 • edited by sbrannen Loading

Kolesnikov-Vladislav commented May 30, 2023

paajake commented Apr 8, 2024

mbuchner commented May 9, 2024

kobynet commented Feb 9, 2021 •

edited

Loading

BalurQA commented Feb 15, 2023 •

edited by sbrannen

Loading