Limit the number of concurrently executing tests via a property

JUnit Jupiter (and The JUnit Platform) now support limiting the
maximum number of concurrently executing tests via the
`junit.jupiter.execution.parallel.config.fixed.max-pool-size` property.

With Java 9+ the `ForkJoinPool` used by JUnit can be configured with a maximum
pool size. While the number of concurrently executing tests may exceed the
configured parallelism when tests become blocked, it will not exceed the
maximum pool size.

With the following configuration:

```properties
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.fixed.parallelism=2
```

This example will report between 2-5 tests running concurrently.

```java
class ExampleTest {

    private static final AtomicInteger running = new AtomicInteger();

    @beforeeach
    void increment() {
        System.out.println("Running " + running.incrementAndGet());
    }

    @AfterEach
    void decrement() {
        running.decrementAndGet();
    }

    static IntStream numbers() {
        return IntStream.range(0, 1000);
    }

    @ParameterizedTest
    @MethodSource("numbers")
    void test(int i) throws ExecutionException, InterruptedException {
        Runnable sleep = () -> {
            try {
                Thread.sleep(600);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        };
        ForkJoinPool.commonPool().submit(sleep).get();
    }
}
```

By also configuring the `max-pool-size` we can ensure the concurrently
executing test does not exceed the configured 2.

```properties
junit.jupiter.execution.parallel.config.fixed.max-pool-size=2
```

Additionally, because the `ForkJoinPool` will by default reject tasks that
would exceed the maximum pool size the
`junit.jupiter.execution.parallel.config.fixed.saturate` property has been
added and will default to `true`. There appears to be no reason to ever set
this `false` but it is there should someone depend on the old behaviour.

These changes were intentionally not made to the `dynamic` strategy to limit
the scope of this pull request. While I can reasonably predict what behaviour
users of the `fixed` strategy might expect, I can not say the same about the
`dynamic` strategy.

Fixes: junit-team#2545
Fixes: junit-team#1858
Fixes: junit-team#3026

documentation/src/docs/asciidoc/user-guide/writing-tests.adoc

@@ -2336,6 +2336,8 @@ configuration parameter to one of the following options.
 `fixed`::
   Uses the mandatory `junit.jupiter.execution.parallel.config.fixed.parallelism`
   configuration parameter as the desired parallelism.
+  The optional `junit.jupiter.execution.parallel.config.fixed.max-pool-size`
+  configuration parameter can be used to limit the maximum number of concurrent tests.
 
 `custom`::
   Allows you to specify a custom `{ParallelExecutionConfigurationStrategy}`
@@ -2346,13 +2348,15 @@ If no configuration strategy is set, JUnit Jupiter uses the `dynamic` configurat
 strategy with a factor of `1`. Consequently, the desired parallelism will be equal to the
 number of available processors/cores.
 
-.Parallelism does not imply maximum number of concurrent threads
-NOTE: JUnit Jupiter does not guarantee that the number of concurrently executing tests
-will not exceed the configured parallelism. For example, when using one of the
-synchronization mechanisms described in the next section, the `ForkJoinPool` that is used
-behind the scenes may spawn additional threads to ensure execution continues with
-sufficient parallelism. Thus, if you require such guarantees in a test class, please use
-your own means of controlling concurrency.
+.Parallelism alone does not imply maximum number of concurrent threads
+NOTE: By default JUnit Jupiter does not guarantee that the number of concurrently
+executing tests will not exceed the configured parallelism. For example, when using one
+of the synchronization mechanisms described in the next section, the `ForkJoinPool` that
+is used behind the scenes may spawn additional threads to ensure execution continues with
+sufficient parallelism.
+If you require such guarantees, with Java 9+, it is possible to limit the maximum number
+of concurrent tests by controlling the maximum pool size of the `fixed` and `custom`
+strategies.
 
 [[writing-tests-parallel-execution-config-properties]]
 ===== Relevant properties
@@ -2404,6 +2408,20 @@ The following table lists relevant properties for configuring parallel execution
 | a positive integer
 | no default value
 
+| ```junit.jupiter.execution.parallel.config.fixed.max-pool-size```
+| Desired maximum pool size of the underlying fork join pool for the ```fixed```
+  configuration strategy
+| a positive integer, must not be smaller then `junit.jupiter.execution.parallel.config.fixed.parallelism`
+| 256 + `junit.jupiter.execution.parallel.config.fixed.parallelism`
+
+| ```junit.jupiter.execution.parallel.config.fixed.saturate```
+| Enable saturation of the underlying fork join pool for the ```fixed``` configuration
+  strategy
+|
+  * `true`
+  * `false`
+| ```true```
+
 | ```junit.jupiter.execution.parallel.config.custom.class```
 | Fully qualified class name of the _ParallelExecutionConfigurationStrategy_ to be
   used for the ```custom``` configuration strategy

junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/Constants.java

@@ -15,7 +15,9 @@
 import static org.apiguardian.api.API.Status.STABLE;
 import static org.junit.platform.engine.support.hierarchical.DefaultParallelExecutionConfigurationStrategy.CONFIG_CUSTOM_CLASS_PROPERTY_NAME;
 import static org.junit.platform.engine.support.hierarchical.DefaultParallelExecutionConfigurationStrategy.CONFIG_DYNAMIC_FACTOR_PROPERTY_NAME;
+import static org.junit.platform.engine.support.hierarchical.DefaultParallelExecutionConfigurationStrategy.CONFIG_FIXED_MAX_POOL_SIZE_PROPERTY_NAME;
 import static org.junit.platform.engine.support.hierarchical.DefaultParallelExecutionConfigurationStrategy.CONFIG_FIXED_PARALLELISM_PROPERTY_NAME;
+import static org.junit.platform.engine.support.hierarchical.DefaultParallelExecutionConfigurationStrategy.CONFIG_FIXED_SATURATE_PROPERTY_NAME;
 import static org.junit.platform.engine.support.hierarchical.DefaultParallelExecutionConfigurationStrategy.CONFIG_STRATEGY_PROPERTY_NAME;
 
 import org.apiguardian.api.API;
@@ -166,6 +168,40 @@ public final class Constants {
 	public static final String PARALLEL_CONFIG_FIXED_PARALLELISM_PROPERTY_NAME = PARALLEL_CONFIG_PREFIX
 			+ CONFIG_FIXED_PARALLELISM_PROPERTY_NAME;
 
+	/**
+	 * Property name used to configure the maximum pool size of the underlying
+	 * fork join pool for the {@code fixed} configuration strategy: {@value}
+	 *
+	 * <p>Value must be an integer and larger or equal to
+	 * {@value #PARALLEL_CONFIG_FIXED_PARALLELISM_PROPERTY_NAME}; defaults to
+	 * {@code 256 + fixed.parallelism}.
+	 *
+	 * <p>Note: This property only takes affect on Java 9+.
+	 *
+	 * @since 5.10
+	 */
+	@API(status = EXPERIMENTAL, since = "5.10")
+	public static final String PARALLEL_CONFIG_FIXED_MAX_POOL_SIZE_PROPERTY_NAME = PARALLEL_CONFIG_PREFIX
+			+ CONFIG_FIXED_MAX_POOL_SIZE_PROPERTY_NAME;
+
+	/**
+	 * Property name used to enable saturation of the underlying fork join pool
+	 * for the {@code ffixed} configuration strategy: {@value}
+	 *
+	 * <p>When set to {@code false} the underlying fork join pool will reject
+	 * additional tasks when all available workers are busy and the maximum
+	 * pool-size would be exceeded.
+
+	 * <p>Value must either {@code true} or {@code false}; defaults to {@code true}.
+	 *
+	 * <p>Note: This property only takes affect on Java 9+.
+	 *
+	 * @since 5.10
+	 */
+	@API(status = EXPERIMENTAL, since = "5.10")
+	public static final String PARALLEL_CONFIG_FIXED_SATURATE_PROPERTY_NAME = PARALLEL_CONFIG_PREFIX
+			+ CONFIG_FIXED_SATURATE_PROPERTY_NAME;
+
 	/**
 	 * Property name used to set the factor to be multiplied with the number of
 	 * available processors/cores to determine the desired parallelism for the

junit-platform-engine/src/main/java/org/junit/platform/engine/support/hierarchical/DefaultParallelExecutionConfiguration.java

@@ -10,6 +10,9 @@
 
 package org.junit.platform.engine.support.hierarchical;
 
+import java.util.concurrent.ForkJoinPool;
+import java.util.function.Predicate;
+
 /**
  * @since 1.3
  */
@@ -20,14 +23,16 @@ class DefaultParallelExecutionConfiguration implements ParallelExecutionConfigur
 	private final int maxPoolSize;
 	private final int corePoolSize;
 	private final int keepAliveSeconds;
+	private final Predicate<? super ForkJoinPool> saturate;
 
 	DefaultParallelExecutionConfiguration(int parallelism, int minimumRunnable, int maxPoolSize, int corePoolSize,
-			int keepAliveSeconds) {
+			int keepAliveSeconds, Predicate<? super ForkJoinPool> saturate) {
 		this.parallelism = parallelism;
 		this.minimumRunnable = minimumRunnable;
 		this.maxPoolSize = maxPoolSize;
 		this.corePoolSize = corePoolSize;
 		this.keepAliveSeconds = keepAliveSeconds;
+		this.saturate = saturate;
 	}
 
 	@Override
@@ -55,4 +60,8 @@ public int getKeepAliveSeconds() {
 		return keepAliveSeconds;
 	}
 
+	@Override
+	public Predicate<? super ForkJoinPool> getSaturatePredicate() {
+		return saturate;
+	}
 }

junit-platform-engine/src/main/java/org/junit/platform/engine/support/hierarchical/DefaultParallelExecutionConfigurationStrategy.java

@@ -42,8 +42,14 @@ public ParallelExecutionConfiguration createConfiguration(ConfigurationParameter
 					() -> new JUnitException(String.format("Configuration parameter '%s' must be set",
 						CONFIG_FIXED_PARALLELISM_PROPERTY_NAME)));
 
-			return new DefaultParallelExecutionConfiguration(parallelism, parallelism, 256 + parallelism, parallelism,
-				KEEP_ALIVE_SECONDS);
+			int maxPoolSize = configurationParameters.get(CONFIG_FIXED_MAX_POOL_SIZE_PROPERTY_NAME,
+				Integer::valueOf).orElse(parallelism + 256);
+
+			boolean saturate = configurationParameters.get(CONFIG_FIXED_SATURATE_PROPERTY_NAME,
+					Boolean::valueOf).orElse(true);
+
+			return new DefaultParallelExecutionConfiguration(parallelism, parallelism, maxPoolSize, parallelism,
+				KEEP_ALIVE_SECONDS, __ -> saturate);
 		}
 	},
 
@@ -66,7 +72,7 @@ public ParallelExecutionConfiguration createConfiguration(ConfigurationParameter
 				factor.multiply(BigDecimal.valueOf(Runtime.getRuntime().availableProcessors())).intValue());
 
 			return new DefaultParallelExecutionConfiguration(parallelism, parallelism, 256 + parallelism, parallelism,
-				KEEP_ALIVE_SECONDS);
+				KEEP_ALIVE_SECONDS, null);
 		}
 	},
 
@@ -114,6 +120,34 @@ public ParallelExecutionConfiguration createConfiguration(ConfigurationParameter
 	 */
 	public static final String CONFIG_FIXED_PARALLELISM_PROPERTY_NAME = "fixed.parallelism";
 
+	/**
+	 * Property name used to configure the maximum pool size of the underlying
+	 * fork join pool for the {@link #FIXED} configuration strategy.
+	 * <p>Value must be an integer and larger or equal to
+	 * {@value #CONFIG_FIXED_PARALLELISM_PROPERTY_NAME}; defaults to
+	 * {@code 256 + fixed.parallelism}.
+	 *
+	 * @since 1.10
+	 * @see #FIXED
+	 */
+	@API(status = EXPERIMENTAL, since = "1.10")
+	public static final String CONFIG_FIXED_MAX_POOL_SIZE_PROPERTY_NAME = "fixed.max-pool-size";
+
+	/**
+	 * Property name used to enable saturation of the underlying fork join pool
+	 * for the {@link #FIXED} configuration strategy.
+	 * <p>When set to {@code false} the underlying fork join pool will reject
+	 * additional tasks when all available workers are busy and the maximum
+	 * pool-size would be exceeded.
+	 * <p>Value must either {@code true} or {@code false}; defaults to {@code true}.
+	 *
+	 * @since 1.10
+	 * @see #FIXED
+	 * @see #CONFIG_FIXED_MAX_POOL_SIZE_PROPERTY_NAME
+	 */
+	@API(status = EXPERIMENTAL, since = "1.10")
+	public static final String CONFIG_FIXED_SATURATE_PROPERTY_NAME = "fixed.saturate";
+
 	/**
 	 * Property name of the factor used to determine the desired parallelism for the
 	 * {@link #DYNAMIC} configuration strategy.

platform-tests/src/test/java/org/junit/platform/engine/support/hierarchical/DefaultParallelExecutionConfigurationStrategyTests.java

@@ -17,8 +17,6 @@
 import static org.mockito.Mockito.when;
 
 import java.util.Optional;
-import java.util.concurrent.ForkJoinPool;
-import java.util.function.Predicate;
 
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -51,7 +49,20 @@ void fixedStrategyCreatesValidConfiguration() {
 		assertThat(configuration.getMinimumRunnable()).isEqualTo(42);
 		assertThat(configuration.getMaxPoolSize()).isEqualTo(256 + 42);
 		assertThat(configuration.getKeepAliveSeconds()).isEqualTo(30);
-		assertThat(configuration.getSaturatePredicate()).isNull();
+		assertThat(configuration.getSaturatePredicate().test(null)).isTrue();
+	}
+
+	@Test
+	void fixedSaturateStrategyCreatesValidConfiguration() {
+		when(configParams.get("fixed.parallelism")).thenReturn(Optional.of("42"));
+		when(configParams.get("fixed.max-pool-size")).thenReturn(Optional.of("42"));
+		when(configParams.get("fixed.saturate")).thenReturn(Optional.of("false"));
+
+		ParallelExecutionConfigurationStrategy strategy = DefaultParallelExecutionConfigurationStrategy.FIXED;
+		var configuration = strategy.createConfiguration(configParams);
+		assertThat(configuration.getParallelism()).isEqualTo(42);
+		assertThat(configuration.getMaxPoolSize()).isEqualTo(42);
+		assertThat(configuration.getSaturatePredicate().test(null)).isFalse();
 	}
 
 	@Test
@@ -183,12 +194,7 @@ void customStrategyThrowsExceptionWhenClassDoesNotExist() {
 	static class CustomParallelExecutionConfigurationStrategy implements ParallelExecutionConfigurationStrategy {
 		@Override
 		public ParallelExecutionConfiguration createConfiguration(ConfigurationParameters configurationParameters) {
-			return new DefaultParallelExecutionConfiguration(1, 2, 3, 4, 5) {
-				@Override
-				public Predicate<? super ForkJoinPool> getSaturatePredicate() {
-					return __ -> true;
-				}
-			};
+			return new DefaultParallelExecutionConfiguration(1, 2, 3, 4, 5, __ -> true);
 		}
 	}