Parallelism option ignored when using own ForkJoinPool
#1858
Comments
|
Are the 4 tests all in the same test class or different test classes? Could you please provide a small sample project? |
|
Yes all 4 tests are in the same classes. As I can't share the original test code I wrote a small sample snippet which tried to reproduce the problem. This is when I found the actual problem. This isn't related to tags but more to concurrency and the meaning of the @Test @Tag("main")
void done1() throws InterruptedException {
System.out.println("BEGIN1");
System.out.println(Thread.currentThread().getName());
ForkJoinPool customThreadPool = new ForkJoinPool(4);
RecursiveAction a = new RecursiveAction() {
@Override
protected void compute() {
while (true) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(Thread.currentThread().getName());}
}
};
customThreadPool.invoke(a);
System.out.println("DONE1");
}
@Test @Tag("main")
void done2() throws InterruptedException {
System.out.println("BEGIN2");
System.out.println(Thread.currentThread().getName());
String name = Thread.currentThread().getName();
threadNames.add(name);
ForkJoinPool customThreadPool = new ForkJoinPool(4);
RecursiveAction a = new RecursiveAction() {
@Override
protected void compute() {
while (true) {
try {When setting My knowledge about ForkJoinPools is unfortunately limited as I didn't work with them before but it seems that when worker-1 is forced to wait on the result of the invoke, another worker (worker-0) is spawned to start with the second test. This essentially means that all tests in my class would run in parallel. So i guess the If this is not what happens I would be happy if you could help me understand what exactly is going on here. In any case this behavior surprised me as my intuition would be that the tests that are executed in parallel must actually complete before another test can be started. While I can see that there might be reasons why one does not want to do this, is there a way to enforce the behavior I expect (i.e that I can control how many tests are started in parallel) ? |
|
The idea behind @fredericOfTestfabrik Do your actual tests also start operations that use |
|
First of all thank you for your explanation. Regarding your Question: Yes the actual test create fork join pools to let long running operations execute in parallel. My tests use selenium to test a website. The test code is built in such a way that it can be run in parallel for different browsers. Each test thus runs the selenium code in parallel for 2 different browsers. When I don't use junit5s parallelism, this runs fine. Now want I want to do is to run multiple testcases at the same time. However the total amount of available browsers is limited. E.g I have 4 browsers available so I want to run 2 testcases at most at the same time, because each of the testcase again needs 2 browsers. My guess is that at some point the testcode needs to yield (because it waits on a response or there is a thread sleep needed to ensure some stability). And this is when the additional tests are run. So what I need would be a setting or functionality to ensure that no more than 2 tests are run in parallel even if one test case is suspended for a certain amount of time. Edit: I'm sorry for the mess. I am writing this from my phone and accidentally hit the close button |
|
Thanks for the explanation! I'll take a closer look in the next few days. |
|
Is there an update on this? Do you need additional Information? |
|
@fredericOfTestfabrik I should know better than to make promises like that, sorry! Unfortunately, I haven't had a chance to investigate further, yet. |
|
Created https://github.com/sormuras/junit5-1858-fixed-parallelism to investigate this issue. Some notes:
|
sormuras
changed the title
|
Changed the title to reflect that the underlying issue doesn't need tags or Maven to show up. |
|
Using an Seems like multiple Does anybody know a way to prevent this? By tagging/marking tasks as belonging to a specific pool only? |
|
Using |
work-aroundUsing a custom strategy that also fixes the maximum (and core?) pool size to same value as the The default |
|
IIRC the problem with that is that the |
|
@fredericOfTestfabrik Please give the workaround posted by @sormuras above a try and let us know if that works for you. |
|
There is no (good) fix for Java 8 aside from serializing tests. (Maybe you can work arround this issue by e.g. running your selenium-tests on Java 11/17 but still compile against language level 8.) |
|
@christophs78 I am using Gradle instead maven. Do you recommend any specific version for Gradle to be used to mitigate this issue. From your comment you mean to say that below setting should work fine without any issue. It works based configurations in junit-platform.properties file. Java version - > 11 |
|
Gradle-version IMO should not matter for this. On my day job we are using Gradle 7.x |
|
@christophs78 : Can you please help to resolve this. I am using JAVA 11 and below version for each but still I see undefined number threads are opening Parallel setting looks like this: Note : I am using WebDriverManager also |
|
Hello @sangameshwar-balur! after @christophs78 's suggestion, I tried to compile with Java 11 and 17, but still no luck with the fixed parallelism strategy :( However, after a long investigation with many pokes in the dark... I found that setting a Custom Strategy works for me: maven.compiler.source/target -> 1.8 / 11 / 17 (I did not find any difference... IMPORTANT! as long as you execute them using at least java11 as @christophs78 said. I mean, you can compile against 8, but maven needs to point to java 11) For these config parameters, I moved from the pom.xml to an external \src\main\resources\junit-platform.properties I included a new class with this content: https://github.com/sormuras/junit5-1858-fixed-parallelism/blob/master/src/test/java/CustomStrategy.java Changing the value of the returned overrides, you will probably manage to limit the number of parallel threads. The only disadvantage is that you cannot define the number of threads dynamically/externally from maven/jenkins |
The JUnit Platform now supports saturating the ForkJoinPool via a property. Enabling this property will limit the number of concurrently executing tests will to the configured parallelism even when some are blocked. For JUnit Jupiter this can be enabled by through the `junit.jupiter.execution.parallel.config.dynamic.saturate` and `junit.jupiter.execution.parallel.config.fixed.saturate` configuration properties. Fixes: junit-team#2545 Fixes: junit-team#1858 Fixes: junit-team#3026
The JUnit Platform now supports saturating the ForkJoinPool via a property. Enabling this property will limit the number of concurrently executing tests will to the configured parallelism even when some are blocked. For JUnit Jupiter this can be enabled by through the `junit.jupiter.execution.parallel.config.dynamic.saturate` and `junit.jupiter.execution.parallel.config.fixed.saturate` configuration properties. Fixes: junit-team#2545 Fixes: junit-team#1858 Fixes: junit-team#3026
The JUnit Platform now supports saturating the ForkJoinPool via a property. Enabling this property will limit the number of concurrently executing tests will to the configured parallelism even when some are blocked. For JUnit Jupiter this can be enabled by through the `junit.jupiter.execution.parallel.config.dynamic.saturate` and `junit.jupiter.execution.parallel.config.fixed.saturate` configuration properties. Fixes: junit-team#2545 Fixes: junit-team#1858 Fixes: junit-team#3026
The JUnit Platform now supports saturating the ForkJoinPool via a property. Enabling this property will limit the number of concurrently executing tests will to the configured parallelism even when some are blocked. For JUnit Jupiter this can be enabled by through the `junit.jupiter.execution.parallel.config.dynamic.saturate` and `junit.jupiter.execution.parallel.config.fixed.saturate` configuration properties. Fixes: junit-team#2545 Fixes: junit-team#1858 Fixes: junit-team#3026
The JUnit Platform now supports saturating the ForkJoinPool via a property. Enabling this property will limit the number of concurrently executing tests will to the configured parallelism even when some are blocked. For JUnit Jupiter this can be enabled by through the `junit.jupiter.execution.parallel.config.dynamic.saturate` and `junit.jupiter.execution.parallel.config.fixed.saturate` configuration properties. Fixes: junit-team#2545 Fixes: junit-team#1858 Fixes: junit-team#3026
JUnit Jupiter (and The JUnit Platform) now support limiting the
maximum number of concurrently executing tests via the
`junit.jupiter.execution.parallel.config.fixed.max-pool-size` property.
With Java 9+ the `ForkJoinPool` used by JUnit can be configured with a maximum
pool size. While the number of concurrently executing tests may exceed the
configured parallelism when tests become blocked, it will not exceed the
maximum pool size.
With the following configuration:
```properties
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.fixed.parallelism=2
```
This example will report between 2-5 tests running concurrently.
```java
class ExampleTest {
private static final AtomicInteger running = new AtomicInteger();
@beforeeach
void increment() {
System.out.println("Running " + running.incrementAndGet());
}
@AfterEach
void decrement() {
running.decrementAndGet();
}
static IntStream numbers() {
return IntStream.range(0, 1000);
}
@ParameterizedTest
@MethodSource("numbers")
void test(int i) throws ExecutionException, InterruptedException {
Runnable sleep = () -> {
try {
Thread.sleep(600);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
};
ForkJoinPool.commonPool().submit(sleep).get();
}
}
```
By also configuring the `max-pool-size` we can ensure the concurrently
executing test does not exceed the configured 2.
```properties
junit.jupiter.execution.parallel.config.fixed.max-pool-size=2
```
Additionally, because the `ForkJoinPool` will by default reject tasks that
would exceed the maximum pool size the
`junit.jupiter.execution.parallel.config.fixed.saturate` property has been
added and will default to `true`. There appears to be no reason to ever set
this `false` but it is there should someone depend on the old behaviour.
These changes were intentionally not made to the `dynamic` strategy to limit
the scope of this pull request. While I can reasonably predict what behaviour
users of the `fixed` strategy might expect, I can not say the same about the
`dynamic` strategy.
Fixes: junit-team#2545
Fixes: junit-team#1858
Fixes: junit-team#3026
JUnit Jupiter (and The JUnit Platform) now support limiting the
maximum number of concurrently executing tests via the
`junit.jupiter.execution.parallel.config.fixed.max-pool-size` property.
With Java 9+ the `ForkJoinPool` used by JUnit can be configured with a maximum
pool size. While the number of concurrently executing tests may exceed the
configured parallelism when tests become blocked, it will not exceed the
maximum pool size.
With the following configuration:
```properties
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.fixed.parallelism=2
```
This example will report between 2-5 tests running concurrently.
```java
class ExampleTest {
private static final AtomicInteger running = new AtomicInteger();
@beforeeach
void increment() {
System.out.println("Running " + running.incrementAndGet());
}
@AfterEach
void decrement() {
running.decrementAndGet();
}
static IntStream numbers() {
return IntStream.range(0, 1000);
}
@ParameterizedTest
@MethodSource("numbers")
void test(int i) throws ExecutionException, InterruptedException {
Runnable sleep = () -> {
try {
Thread.sleep(600);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
};
ForkJoinPool.commonPool().submit(sleep).get();
}
}
```
By also configuring the `max-pool-size` we can ensure the concurrently
executing test does not exceed the configured 2.
```properties
junit.jupiter.execution.parallel.config.fixed.max-pool-size=2
```
Additionally, because the `ForkJoinPool` will by default reject tasks that
would exceed the maximum pool size the
`junit.jupiter.execution.parallel.config.fixed.saturate` property has been
added and will default to `true`. There appears to be no reason to ever set
this `false` but it is there should someone depend on the old behaviour.
These changes were intentionally not made to the `dynamic` strategy to limit
the scope of this pull request. While I can reasonably predict what behaviour
users of the `fixed` strategy might expect, I can not say the same about the
`dynamic` strategy.
Fixes: junit-team#2545
Fixes: junit-team#1858
Fixes: junit-team#3026
JUnit Jupiter (and The JUnit Platform) now support limiting the
maximum number of concurrently executing tests via the
`junit.jupiter.execution.parallel.config.fixed.max-pool-size` property.
With Java 9+ the `ForkJoinPool` used by JUnit can be configured with a maximum
pool size. While the number of concurrently executing tests may exceed the
configured parallelism when tests become blocked, it will not exceed the
maximum pool size.
With the following configuration:
```properties
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.fixed.parallelism=2
```
This example will report between 2-5 tests running concurrently.
```java
class ExampleTest {
private static final AtomicInteger running = new AtomicInteger();
@beforeeach
void increment() {
System.out.println("Running " + running.incrementAndGet());
}
@AfterEach
void decrement() {
running.decrementAndGet();
}
static IntStream numbers() {
return IntStream.range(0, 1000);
}
@ParameterizedTest
@MethodSource("numbers")
void test(int i) throws ExecutionException, InterruptedException {
Runnable sleep = () -> {
try {
Thread.sleep(600);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
};
ForkJoinPool.commonPool().submit(sleep).get();
}
}
```
By also configuring the `max-pool-size` we can ensure the concurrently
executing test does not exceed the configured 2.
```properties
junit.jupiter.execution.parallel.config.fixed.max-pool-size=2
```
Additionally, because the `ForkJoinPool` will by default reject tasks that
would exceed the maximum pool size the
`junit.jupiter.execution.parallel.config.fixed.saturate` property has been
added and will default to `true`. There appears to be no reason to ever set
this `false` but it is there should someone depend on the old behaviour.
These changes were intentionally not made to the `dynamic` strategy to limit
the scope of this pull request. While I can reasonably predict what behaviour
users of the `fixed` strategy might expect, I can not say the same about the
`dynamic` strategy.
Fixes: junit-team#3026
Fixes: junit-team#2545
Fixes: junit-team#1858
JUnit Jupiter (and The JUnit Platform) now support limiting the
maximum number of concurrently executing tests via the
`junit.jupiter.execution.parallel.config.fixed.max-pool-size` property.
With Java 9+ the `ForkJoinPool` used by JUnit can be configured with a maximum
pool size. While the number of concurrently executing tests may exceed the
configured parallelism when tests become blocked, it will not exceed the
maximum pool size.
With the following configuration:
```properties
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.fixed.parallelism=2
```
This example will report between 2-5 tests running concurrently.
```java
class ExampleTest {
private static final AtomicInteger running = new AtomicInteger();
@beforeeach
void increment() {
System.out.println("Running " + running.incrementAndGet());
}
@AfterEach
void decrement() {
running.decrementAndGet();
}
static IntStream numbers() {
return IntStream.range(0, 1000);
}
@ParameterizedTest
@MethodSource("numbers")
void test(int i) throws ExecutionException, InterruptedException {
Runnable sleep = () -> {
try {
Thread.sleep(600);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
};
ForkJoinPool.commonPool().submit(sleep).get();
}
}
```
By also configuring the `max-pool-size` we can ensure the concurrently
executing test does not exceed the configured 2.
```properties
junit.jupiter.execution.parallel.config.fixed.max-pool-size=2
```
Additionally, because the `ForkJoinPool` will by default reject tasks that
would exceed the maximum pool size the
`junit.jupiter.execution.parallel.config.fixed.saturate` property has been
added and will default to `true`. There appears to be no reason to ever set
this `false` but it is there should someone depend on the old behaviour.
These changes were intentionally not made to the `dynamic` strategy to limit
the scope of this pull request. While I can reasonably predict what behaviour
users of the `fixed` strategy might expect, I can not say the same about the
`dynamic` strategy.
Fixes: junit-team#3026
Fixes: junit-team#2545
Fixes: junit-team#1858
JUnit Jupiter (and The JUnit Platform) now support limiting the
maximum number of concurrently executing tests via the
`junit.jupiter.execution.parallel.config.fixed.max-pool-size` property.
With Java 9+ the `ForkJoinPool` used by JUnit can be configured with a maximum
pool size. While the number of concurrently executing tests may exceed the
configured parallelism when tests become blocked, it will not exceed the
maximum pool size.
With the following configuration:
```properties
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.fixed.parallelism=2
```
This example will report between 2-5 tests running concurrently.
```java
class ExampleTest {
private static final AtomicInteger running = new AtomicInteger();
@beforeeach
void increment() {
System.out.println("Running " + running.incrementAndGet());
}
@AfterEach
void decrement() {
running.decrementAndGet();
}
static IntStream numbers() {
return IntStream.range(0, 1000);
}
@ParameterizedTest
@MethodSource("numbers")
void test(int i) throws ExecutionException, InterruptedException {
Runnable sleep = () -> {
try {
Thread.sleep(600);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
};
ForkJoinPool.commonPool().submit(sleep).get();
}
}
```
By also configuring the `max-pool-size` we can ensure the concurrently
executing test does not exceed the configured 2.
```properties
junit.jupiter.execution.parallel.config.fixed.max-pool-size=2
```
Additionally, because the `ForkJoinPool` will by default reject tasks that
would exceed the maximum pool size the
`junit.jupiter.execution.parallel.config.fixed.saturate` property has been
added and will default to `true`. There appears to be no reason to ever set
this `false` but it is there should someone depend on the old behaviour.
These changes were intentionally not made to the `dynamic` strategy to limit
the scope of this pull request. While I can reasonably predict what behaviour
users of the `fixed` strategy might expect, I can not say the same about the
`dynamic` strategy.
Fixes: junit-team#3026
Fixes: junit-team#2545
Fixes: junit-team#1858
JUnit Jupiter (and The JUnit Platform) now support limiting the
maximum number of concurrently executing tests via the
`junit.jupiter.execution.parallel.config.fixed.max-pool-size` property.
With Java 9+ the `ForkJoinPool` used by JUnit can be configured with a maximum
pool size. While the number of concurrently executing tests may exceed the
configured parallelism when tests become blocked, it will not exceed the
maximum pool size.
With the following configuration:
```properties
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.fixed.parallelism=2
```
This example will report between 2-5 tests running concurrently.
```java
class ExampleTest {
private static final AtomicInteger running = new AtomicInteger();
@beforeeach
void increment() {
System.out.println("Running " + running.incrementAndGet());
}
@AfterEach
void decrement() {
running.decrementAndGet();
}
static IntStream numbers() {
return IntStream.range(0, 1000);
}
@ParameterizedTest
@MethodSource("numbers")
void test(int i) throws ExecutionException, InterruptedException {
Runnable sleep = () -> {
try {
Thread.sleep(600);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
};
ForkJoinPool.commonPool().submit(sleep).get();
}
}
```
By also configuring the `max-pool-size` we can ensure the concurrently
executing test does not exceed the configured 2.
```properties
junit.jupiter.execution.parallel.config.fixed.max-pool-size=2
```
Additionally, because the `ForkJoinPool` will by default reject tasks that
would exceed the maximum pool size the
`junit.jupiter.execution.parallel.config.fixed.saturate` property has been
added and will default to `true`. There appears to be no reason to ever set
this `false` but it is there should someone depend on the old behaviour.
These changes were intentionally not made to the `dynamic` strategy to limit
the scope of this pull request. While I can reasonably predict what behaviour
users of the `fixed` strategy might expect, I can not say the same about the
`dynamic` strategy.
Fixes: junit-team#3026
Fixes: junit-team#2545
Fixes: junit-team#1858
|
Because it has not been mentioned yet and, IMO, is related the problem of parallelism, we can even explode the limited parallelism even with one thread. I attached a stacktrace file to my comment, showing a stackoverflow in one of JUnit test thread because 41 tests were started consecutively by the same thread. Our product extensively use at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool.helpJoin(ForkJoinPool.java:1883)
at java.base/java.util.concurrent.ForkJoinTask.awaitDone(ForkJoinTask.java:440)
at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:979)
at com.activeviam.database.sql.internal.jdbc.query.JdbcQueryPreparator.runQuery(JdbcQueryPreparator.java:83)
... (cut)
at com.activeviam.database.tck.OurTestClass.optional_testVectorTableWithCondition(AggregateQueriesWithUnnestedVectorsTck.java:187)
... (cut)
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)JUnit launches one test as a Seeing @christophs78 here, having a thread-pool parallel executor would have help. |
|
@OPeyrusse because your issue has a different root cause (not webdriver) and can not be solved/worked around by fixing the max-poolsize I think it would be good to create a separate ticket. |
JUnit Jupiter (and The JUnit Platform) now support limiting the maximum number of concurrently executing tests via the `junit.jupiter.execution.parallel.config.fixed.max-pool-size` configuration parameter. With Java 9+ the `ForkJoinPool` used by JUnit can be configured with a maximum pool size. While the number of concurrently executing tests may exceed the configured parallelism when tests become blocked, it will not exceed the maximum pool size. Additionally, because the `ForkJoinPool` will by default reject tasks that would exceed the maximum pool size the `junit.jupiter.execution.parallel.config.fixed.saturate` property has been added and will default to `true`. There appears to be no reason to ever set this `false` but it is there should someone depend on the old behavior. These changes were intentionally not made to the `dynamic` strategy to limit the scope of this pull request. While I can reasonably predict what behavior users of the `fixed` strategy might expect, I can not say the same about the `dynamic` strategy. Fixes #3026. Fixes #2545. Fixes #1858.
sbrannen
changed the title
ForkJoinPool
I'm using Junit 5.4.1 and am experiencing a weird issue related to parallelism.
I have a set of Tests that I want to execute in parallel. 4 of these tests are tagged with the tag "main". Which I want to execute using maven. However I only want to execute 2 of them at the same time. Thus I configured jUnit 5 as follows:
junit.jupiter.execution.parallel.enabled=true junit.jupiter.execution.parallel.config.strategy=fixed junit.jupiter.execution.parallel.config.fixed.parallelism=2This config works when I invoke the following mvn command:
mvn testHowever, when i add the parameter to only execute the Tests tagged with main:
mvn test -Dgroups=mainALL of the tests (meaning 4) tagged with "main" are executed in parallel, when only 2 should be executed at the same time. It seems that somehow, the parallelism count is ignored. Unfortunately I can't verify which setting is used by JUnit5 or if this bug is related to mvn surefire or Junit5
Steps to reproduce
Context
The text was updated successfully, but these errors were encountered: