Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend ParallelExecutionConfiguration to specify a predicate for determing if a pool is saturated #2787

Closed
klease opened this issue Dec 5, 2021 · 6 comments · Fixed by #2792

Comments

@klease
Copy link
Contributor

klease commented Dec 5, 2021

I'm working on the Apache Camel project which uses parallel execution on a module with a large number of test cases (over 2400 classes and over 6000 tests.) After upgrading to 5.7.2 or later we were getting RejectedExecutionException which we solved by using a custom configuration with a large value for maxPoolSize.

The cause appears to be that the Tasks for the test classes all call ForkJoinPool.managedBlock and the tests themselves can take up to a few seconds to run. Fairly quickly all threads in the ForkJoinPool are busy and therefore each new call to managedBock causes a new thread to be allocated up to the maxPoolSize. Each of these new threads usually runs only 1 or 2 tests.

I found that by providing a predicate for saturated when creating the ForkJoinPool, and having it return true, it was possible to run all the tests with a much smaller maxPoolSize.

Currently it's not possible to control this parameter using a custom implementation of ParallelExecutionConfiguration.

So I propose to add a method to the interface such as the following:

/**
 * Get the predicate called when there the ForkJoinPool has reached MaxPoolSize
 * and there are no threads available, or null. The predicate should return true if the
 * blocking thread should try again. If the predicate is null or returns false a
 * RejectedExcecutionException is thrown.
 * @return a Predicate accepting a ForkJoinPool as the parameter or null.
 */
Predicate<? super ForkJoinPool> getSaturatePredicate();

The DefaultParallelExecutionConfiguration would return null.

The class ForkJoinPoolHierarchicalTestExecutorService would use configuration.getSaturatePredicate() when creating the ForkJoinPool.

@sbrannen
Copy link
Member

sbrannen commented Dec 6, 2021

How would one supply the Predicate?

@klease
Copy link
Contributor Author

klease commented Dec 6, 2021

By implementing the new method in a custom implementation of ParallelExecutionConfiguration, for example:

@Override
public Predicate<? super ForkJoinPool> getSaturatePredicate() {
    return (ForkJoinPool p) -> true;
}

@sbrannen
Copy link
Member

sbrannen commented Dec 7, 2021

Thanks for the clarification.

Wouldn't getSaturatedPredicate() make more sense (saturated)?

@sbrannen sbrannen changed the title Extend ParallelExecutionConfiguration to specify a predidcate for saturated Extend ParallelExecutionConfiguration to specify a predicate for determing if a pool is saturated Dec 7, 2021
@klease
Copy link
Contributor Author

klease commented Dec 7, 2021

Yes, that does make more sense, even if the parameter name in the java.util.concurrent.ForkJoinPool constructor is named "saturate".

@marcphilipp
Copy link
Member

marcphilipp commented Dec 10, 2021

Team decision: Add ParallelExecutionConfiguration.getSaturatePredicate()

@klease Would you be interested in submitting a PR for this?

@klease
Copy link
Contributor Author

klease commented Dec 10, 2021

@klease Would you be interested in submitting a PR for this?

Yes, of course. It will be my first for junit5 so I'll try to follow the guidelines.

klease added a commit to klease/junit5 that referenced this issue Dec 10, 2021
Add the method getSaturatePredicate to the interface ParallelExecutionConfiguration.
This allows a custom configuration to supply a predicate to handle the case
where the ForkJoinPool has reached the maximum pool size but all threads
are currently blocked.
Add a constructor to DefaultParallelExecutionConfiguration allowing to
speciy the predicate and modify the existing constructor to specify
null for the predicate.
Modify DefaultParallelExecutionConfigurationStrategyTests to check the
saturate predicate value in the default and custom configurations.

Issue: junit-team#2787
@sbrannen sbrannen added this to the 5.9 M1 milestone Jan 17, 2022
@sbrannen sbrannen linked a pull request Jan 17, 2022 that will close this issue
marcphilipp added a commit that referenced this issue Apr 10, 2022
This allows a custom configuration to supply a predicate to handle the case
where the ForkJoinPool has reached the maximum pool size but all threads
are currently blocked.

Resolves #2787.

Co-authored-by: Marc Philipp <mail@marcphilipp.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants