ClassOrderer.Random and MethodOrderer.Random do not use same default seed

[kurellajunior]

The documentation is very misleading. It states, that the seed can be set individually for classes and methods, but neither configuration key is actually used:

• ClassOrderer#RANDOM_SEED_PROPERTY_NAME

  junit5/junit-jupiter-api/src/main/java/org/junit/jupiter/api/ClassOrderer.java

  Line 78 in 2886963

  String DEFAULT_ORDER_PROPERTY_NAME = "junit.jupiter.testclass.order.default";

• MethodOrderer#DEFAULT_ORDER_PROPERTY_NAME

  junit5/junit-jupiter-api/src/main/java/org/junit/jupiter/api/MethodOrderer.java

  Line 75 in 413ccc0

  String DEFAULT_ORDER_PROPERTY_NAME = "junit.jupiter.testmethod.order.default";

instead for both random seeds is used MethodOrderer.Random#RANDOM_SEED_PROPERTY_NAME

junit5/junit-jupiter-api/src/main/java/org/junit/jupiter/api/MethodOrderer.java

Line 297 in 413ccc0

public static final String RANDOM_SEED_PROPERTY_NAME = "junit.jupiter.execution.order.random.seed";

This is also very inconsistent with the output of the tests which clearly suggests to set the respective seed to a specific value.

I do not know what is tha actual desired behaviour, so cannot provide a PR

[marcphilipp]

The default order can be configured separately for classes and methods using the junit.jupiter.testclass.order.default and junit.jupiter.testmethod.order.default configuration parameters, respectively. Both ClassOrderer and MethodOrderer use the same junit.jupiter.execution.order.random.seed configuration parameter for customizing the seed used with their random orderers, though.

The documentation is very misleading. It states, that the seed can be set individually for classes and methods

Where does it state that?

neither configuration key is actually used

By "both" are you referring to the setting of the default class/method orderer or the custom seed?

This is also very inconsistent with the output of the tests which clearly suggests to set the respective seed to a specific value.

Which tests are you referring to?

[kurellajunior]

The default order can be configured separately for classes and methods using the junit.jupiter.testclass.order.default and junit.jupiter.testmethod.order.default configuration parameters, respectively. Both ClassOrderer and MethodOrderer use the same junit.jupiter.execution.order.random.seed configuration parameters for customizing the seed used with their random orderers, though.

that is understood

This is also very inconsistent with the output of the tests which clearly suggests setting the respective seed to a specific value.

Which tests are you referring to?

When I set the ClassOrderer$Random and MethodOrderer$Random as the test class and testmethod order default respectively, the tests log on the CONFIG level the (different) random seeds used. The assumption is they do so to replay a test run with the defined seeds. This is impossible, as both classes use the same configuration key as a custom seed

The documentation is very misleading. It states, that the seed can be set individually for classes and methods

Where does it state that?

Indeed now I see, it states here the opposite:

junit5/junit-jupiter-api/src/main/java/org/junit/jupiter/api/ClassOrderer.java

Lines 220 to 221 in 2886963

	* <p>The same property is used by {@link MethodOrderer.Random} for
	* consistency between the two random orderers.

But actually uses two different seeds when running on fallback. This is then the inconsistency

Neither configuration key is actually used

By "both" are you referring to the setting of the default class/method orderer or the custom seed?

At least I could not find any documentation on how to use ClassOrderer#RANDOM_SEED_PROPERTY_NAME or MethodOrderer#DEFAULT_ORDER_PROPERTY_NAME. Luckily GitHub Copilot knew how to set those in the properties…

[sbrannen]

Thanks for the explanation, @kurellajunior. 👍

But actually uses two different seeds when running on fallback. This is then the inconsistency

That's an actual bug.

I've changed the title of this issue to reflect that, and I've assigned this to the 5.11 M2 milestone.