Filtering by tags in JUnit 5 prevents Spock specs from being executed

[jakub-bochenski]

Description

This comes from Junit's EngineDiscoveryOrchestrator

I think this clearly shows there is a bug. Tags should not affect other engines than Jupiter.

How to reproduce

Add some Spock 1.x Specs and have a gradle test task configured as follows:

    test {
            useJUnitPlatform {
                includeTags("SomeTag")
                includeEngines("junit-vintage")
            }
    }

Observe specs are not being executed.
If you comment out the line with "includeTags" the specs are executed.

I've checked in the debugger that a RunnerTestDescriptor with a Sputnik runner is being discovered (the Specs are not being skipped/ignored by Spock).

The same happens with Spock 2.x

    test {
            useJUnitPlatform {
                includeTags("SomeTag")
                excludeEngines("junit-vintage")
            }
    }

Also reported as junit-team/junit5#2563

Additional Environment information

• Build Tool/IDE: gradle 6.8.3
• Spock 1.3-groovy-2.5, 2.0-M2-groovy-2.5
• JUnit / Platform: 5.6.3 / 1.6.3

[leonard84]

Tags are a JUnit Platform feature that Spock 2.0 doesn't support at the moment.
For Spock 1.x that support would need to come from Vintage/JUnit4.

@marcphilipp it might worthwhile to allow engines to opt-in to such features and ignore them otherwise.

[jakub-bochenski]

@marcphilipp it might worthwhile to allow engines to opt-in to such features and ignore them otherwise.

Yes, that's my point. Right now if you want to use tags you can't use any other engines together with Jupiter

[marcphilipp]

Tags are not a Jupiter but a Platform feature. Hence, if you include a certain tag, all engines will be filtered accordingly. The Vintage engine converts JUnit 4's @Category annotations into tags so it does support tags. If you want to run a set of Jupiter tests with a certain tag and all Spock tests, you can define a separate test task for one of them. Alternatively, if you want to include all untagged tests and a certain tag you can use includeTags("none() | SomeTag").

[jakub-bochenski]

@marcphilipp if tags are a platform feature why is the annotation org.junit.jupiter.api.Tag ?

[Vampire]

That's the way to set a tag in Jupiter. Other engines can support different ways like for example category annotation or configuration file or naming convention or cyclomatic complexity of a method or data about previous runs and so on. The engine translates whatever means it sees fit to a platform tag on which you then can filter.

[leonard84]

So, if you are still on Spock 1.x you can use JUnit 4's @Category annotation to define tags for the time being.

[jakub-bochenski]

So, if you are still on Spock 1.x you can use JUnit 4's @category annotation to define tags for the time being.

Yeah, that would be helpful (too bad they can't work as meta-annotations).

My point is: JUnit manual does a very bad job of explaining this whole thing, it only mentions org.junit.jupiter.api.Tag in this context. (Well I guess nothing Spock team can do about it...)

Any chance of Spock 2 getting tag support now, that it became apparent that it's less feature-complete than 1.x? :D

[leonard84]

We are thinking about how to add it, but Spock always had its own way of including/excluding Specs and Features, and I'd argue it is more powerful.

[jakub-bochenski]

Powerful or not it doesn't work well with JUnit 5, as explained above it makes it necessary to have a separate execution.

Also I've always found it awkward to have to edit a file to specify the options. There's no easy way to specify it directly from a gradle build (you could try https://stackoverflow.com/a/66371811/1237617).

[marcphilipp]

JUnit manual does a very bad job of explaining this whole thing, it only mentions org.junit.jupiter.api.Tag in this context.

@jakub-bochenski Could you please open an issue in the junit5 repo for this?

[leonard84]

As the file is a groovy script, you can react to system properties / environment variables or other things to control what gets included/excluded.
But yes, if you want to mix it with Jupiter, you currently can't really filter those two in parallel.

[jacattau]

Just dropping in to say I discovered this very same issue today. Agree with @jakub-bochenski that having Spock 2.0 support the Tag annotation would be extremely helpful.

[blindpirate]

Can we maybe add a global tag like Spock to all test descriptors discovered by Spock? In this way, we can use

includeTags(Spock)

and

runner {
   include "Something to filter in spock"
}

See https://junit.org/junit5/docs/5.0.0-M3/api/org/junit/platform/engine/TestDescriptor.html#getTags--

[leonard84]

The main reason that Spock doesn't support this yet, is that there already exists an ITaggable and Tag in Spock, which have different semantics. This makes adding a new different kind of tag awkward in the API, simply renaming the existing one would be a breaking change. An alternative would be to simply overload the meaning of Tag and treat a special key, e.g., junit-platform-test-tag as special, but this also seems rather ugly and leaks implementation details to the API.

[vanta]

Any update on this issue? Our internal test runner was based on @Category, we updated Spock to 2.0, added spock-4, junit-vintage and it's not working. We migrated @Category to JUnit's @Tag and still not working. It's a blocker for us. We need JDK17, and it needs Groovy 3, and it needs Spock 2.0 and here we are...

[Zardoz89]

Any update on this issue? Our internal test runner was based on @Category, we updated Spock to 2.0, added spock-4, junit-vintage and it's not working. We migrated @Category to JUnit's @Tag and still not working. It's a blocker for us. We need JDK17, and it needs Groovy 3, and it needs Spock 2.0 and here we are...

Nearly the same situation for me.
There isn't any example of how setup a simple Spock 2 + Maven setup where surefire runs normal tests, and failsafe run integration tests makes with @tag or whatever way Spock 2 haves to mark a integration test.

EDIT: At least, I find someone that make something similar and put an example : https://github.com/Martin-BG/spock-jupiter-integration

However, this requires to rename integration tests with the suffix "IT"

[leonard84]

However, this requires to rename integration tests with the suffix "IT"

That is the standard way for surefire (*Test) and failsafe (*IT), using @Category for that seems like an anti-pattern.

[kriegaex]

I agree with @leonard84. I also use Maven whenever working with Spock, and I use convention over configuration, sticking to the Surefire and Failsafe default names, i.e. unit tests end have names like *Test while integration tests are named *IT. So I never needed tags or categories for that. If you insist on naming your unit tests something like *Spec and maybe the ITs *IntegrationSpec or whatever, you can adjust the includes and excludes of both plugins, maybe in a pluginManagement section which you can share among modules of the same project or even among multiple projects (as a BoM).

We migrated @Category to JUnit's @Tag and still not working.

Maybe you missed what @leonard84 said at the beginning of this thread:

Tags are a JUnit Platform feature that Spock 2.0 doesn't support at the moment.

I.e., the concept of tags are a platform feature, but the @Tag annotation is a JUnit Jupiter thing, i.e. the specific way in which the Jupiter engine supports the platform feature. See also the JUnit 5 manual.

Having said that, I understand that it might be useful for the Spock engine (which is a sibling, not a child of the Jupiter engine on top of the JUnit 5 platform) to also offer a means to map some user-configurable information to tags. At the moment, it simply does not (yet).

---

@leonard84: Actually, the main thing that is missing at the moment in spock.config.IncludeExcludeCriteria is some way to not just specify class/annotation names in the Spock configuration, but also match annotation values. E.g. something like

include {
  annotation org.junit.jupiter.api.Tag, { value.contains("fast") && !value.contains("slow") }
}

exclude {
  annotation org.junit.jupiter.api.Tag, { value.equals("UI") }
}

I.e., an optional closure parameter with filter criteria. For user convenience, the annotation instance could be bound already, I imagine something like Spock evaluating with annotationInstance { filterResult = filterClosure() } or so (pseudo code), then the user could just access all annotation parameters and return something truthy when evaluating them inside the closure. Users would be free to access e.g. system properties defined in the Surefire/Failsafe configuration from within the closure and evaluate them as they see fit. If they want to support something fancy like logical operations within the annotation value, a simple parser like the one in the JUnit 5 platform would be easy to implement. A specific Spock extension might even take care of parsing or the Spock config could directly access the platform's original parser, whatever.

[leonard84]

Actually, the main thing that is missing at the moment in spock.config.IncludeExcludeCriteria is some way to not just specify class/annotation names in the Spock configuration, but also match annotation values.

This wont solve the problem mentioned in this issue, as the platform is doing the filtering in this case not Spock.

[kriegaex]

I was not talking about delegating to JUnit platform, not knowing the implications of the way spock generates test names for unrolled methods, but about extending the include exclude extension in such a way that we have an analog to it and can parametrise it from outside, say from a build tool like Maven.

BTW, does the Spock engine "see" the platform config coming from e.g. the Maven JUnit 5 provider? Could it read the tags (categories) configuration in principle? Does the platform pass it through to the Spock engine? If so, where in the Spock source would I have to look? Maybe one of these days I want to play with some ideas, just for fun.

[leonard84]

I was not talking about delegating to JUnit platform

Me neither, but this issues is about the case when you use

    test {
            useJUnitPlatform {
                includeTags("SomeTag")
            }
    }

In this case the plattform is doing the filtering and removes all Spock tests, because Spock returns an empty list of tags here https://github.com/junit-team/junit5/blob/718b6937415dd825383641f9ddb00ce12f4640fb/junit-platform-engine/src/main/java/org/junit/platform/engine/TestDescriptor.java#L78
So, it wouldn't help if Spock were doing some filtering on its own, unless it also returns the tags via this method.

[leonard84]

Fixed by #1467

[Ruchira-R]

Hiya, how do I now make test suites that works with Maven? I've been using Spock 2.x Groovy 3.x, and Categories don't work with Maven. I've been wanting to run a Test Suite through Maven, and all of the documentation only show support for Gradle.
@leonard84, could you help?
Does the TestTag work with Maven?

[leonard84]

That is on the JUnit Platform level and you can read about it here: https://junit.org/junit5/docs/current/user-guide/#running-tests-build-maven-filter-tags
Be advised, that this feature hasn't been released yet and will come with Spock 2.2 M2

[Ruchira-R]

Is there a timeline on when that will be released?

[leonard84]

If you don't have to interoperate with other test engines, I suggest to use Spock's built-in include/exclude functionality that has been available for a long time.