Limit MATCHING_RESOURCES TestResources to the test that declares them

[snazy]

• Closes: 3.16: @WithTestResource starts all test resources (regression) #44235

[snazy]

@geoand your change LGTM! I've added some missing scope-checks, which otherwise eventually caused callbacks via AbstractTestWithCallbacksExtension not being properly registered, causing class-loader issues with NoSuchMethodExceptions due to "stale" callbacks still being registered.

Applied the change against the 3.16.1 tag and ran our integration tests, manually verifying that only the right test-resources (containers) are running.

[geoand]

Thanks a lot!

[geoand]

@snazy something seems to have broke with the latest change

[snazy]

@geoand yea, just making MATCHING_RESOURCES work like RESTRICTED_TO_CLASS isn't enough (actually, too much).

[snazy]

Added two more fixes to this change.

1. consider the resource-args when ordering tests, and also for restarting
2. ensure that the JUnit callbacks are cleaned up, when having test classes having a mixed set of @QuarkusTest and @QuarkusIntegrationTest.

Re 2) I realized that the NoSuchMethodException was actually caused by JUnit callbacks that were registered for a @QuarkusTest (we call it an integration test in Nessie, because it uses test containers), but threw when running a @QuarkusIntegrationTest, because the callbacks were not cleared.

[geoand]

Thanks!

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 54f8235.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

[jedla97]

@geoand @snazy looking at this at this and think that this bug fix should be documented in release notes as it can possibly break someone tests.
Our one TS adopted this back in September and I probably use incorectly MATCHING_RESOURCES scope (Probably I interpret the java docs incorectly and it was working fine). There can be some early adopter of this which done same error as me (also in guides we still use @QuarkusTestResource so there isn't much example or info about this).

If I understand this change correctly. If you want use the MATCHING_RESOURCES you need to anotate every test class with it, to all the test classes with same resource are grouped together and use that resource without restart. Before this PR it behave same/similar to GLOBAL, that there was no need of anotation of every test classes. So it was enough to have one testResource class to start the resource to be used for all the test.

Simply:

• RESTRICTED_TO_CLASS start for every test class
• MATCHING_RESOURCES start for specific group of test classes
• GLOBAL start for every test class

[geoand]

Before this PR it behave same/similar to GLOBAL, that there was no need of anotation of every test classes

Correct, but this was a bug honestly

[geoand]

That said, improvements to the docs and / or Javadoc are always welcome :)

[holly-cummins]

@snazy, have you got a reproducer I could run for the callbacks issue? I'm doing a rebase as part of https://github.com/orgs/quarkusio/projects/30. After my changes, I think the callback clearing is either going to need to be implemented differently (classloader issues), or may not be needed at all. I'm not entirely sure which it is, and want to make sure I don't mess it up!