ArchUnit is run regardless of selected tests scope with JUnit5 in IntelliJ IDEA

[esiha]

Some context to help understand

I have a Kotlin project with the following package structure:

com.example.project
|- domain
|- driving
`- driven

And some ArchUnit tests that ensure that the hexagonal architecture constraints are met in com.example.project.HexagonalArchitectureTest.kt.

I'm using:

• Kotlin 1.4.31
• ArchUnit 0.17.0
• JUnit 5 5.7.1
• IntelliJ IDEA Ultimate 2020.3

The problem I'm facing

Whenever I run a subset of my tests in IntelliJ IDEA, say by hitting Ctrl+Shift+F10 on the domain package, the test console shows a bunch of ArchUnit-related logs before actually running the tests I've selected, starting with:

13:51:05.266 [main] INFO com.tngtech.archunit.core.PluginLoader - Detected Java version 11.0.10
13:51:05.279 [main] DEBUG com.tngtech.archunit.core.PluginLoader - Current Java version is compatible to JAVA_9 => Loading Plugin com.tngtech.archunit.core.importer.ModuleImportPlugin
13:51:05.407 [main] DEBUG com.tngtech.archunit.ArchConfiguration - No configuration found in classpath at archunit.properties => Using default configuration
... (logs telling which classes are being analysed) ...
... (then the tests I wanted to run actually run)...

This is a problem to me because this situation artificially increases the duration of my feedback loop when working in a Test-Driven Development setting: the ArchUnit analysis is usually much longer to run than the tests I actually meant to run.

What I've been able to work out so far regarding this issue

• Using fields or functions to declare the @ArchTests has no effect on the issue;
• Changing the packages attribute of @AnalyzeClasses has no effect on the issue;
• Using archunit-junit4 instead of archunit-junit5 works as a workaround;
• Violating the ArchUnit rules on purpose does not affect the overall tests results shown in IntelliJ IDEA.

[hankem]

I'm not an expert on JUnit internals (at all 😉), but I wonder whether this could be related to #504?

[codecholeric]

Hmm, my problem is, that I can't really reproduce this 🤔
I didn't try it with Kotlin, as that seems unrelated (but maybe I'm wrong), but when I e.g. add a new package with some JUnit Jupiter test parallel to https://github.com/TNG/ArchUnit-Examples/tree/main/example-junit5/src/test/java/com/tngtech/archunit/exampletest/junit5 and execute the tests of this package from IntelliJ I don't see any logs about class import and the test runs quickly 🤔

Could you share a little example project that reproduces the problem? Or can you reproduce your problem in ArchUnit-Examples? Thanks!

[esiha]

Sorry for the late answer, I'll put together an example project tomorrow.

[esiha]

I have stripped down the project I work on and in that process I've managed to find some interesting things:

1. The issue arises only when running tests on a package (for example on the package com.example.extensions), not on any given test class;
2. The issue seems to be linked in some way to Spring Boot. When I remove the spring-boot-starter-web dependency, the issue disappears.

Here's the stripped-down project: https://github.com/esiha/archunit-bug-546

[tedyoung]

I've run into this same issue on several different projects with 0.23.1 (they're all Java). Any update on a fix or a better workaround? I've had to resort to temporarily removing ArchUnit from the classpath while I'm doing my fast TDD test cycling.

[lmartelli]

I also have this problem : when I filter tests on a directory or some tags, ArchUnit is processing all the classes even if the tests are not run :-(

[codecholeric]

I looked into this issue again, and I think I understand where the problem/confusion comes from. I never could reproduce it, because I was looking for ArchUnit resolving the classes of @AnalyzeClasses, even though the test class is ignored. But I don't think that is ever the case. However, the ArchUnit TestEngine also uses the ClassFileImporter internally and I think this is what causes the problem here.
The root is that a TestEngine needs to discover all executable classes in a package. Since scanning the classpath for this is not trivial the ArchUnitTestEngine just reuses the ClassFileImporter internally to find all candidate for ArchUnit tests. Since it imports the whole graph and transitively resolves a lot of further stuff, it is naturally slower than just going over the classes and looking into their annotations.

To conclude, I don't think that the ArchUnitTestEngine imports any classes for @AnalyzeClasses tests that are filtered out during the discovery process. It does however import all the classes within the selected package to look for test candidates, even if it turns out later, that there are none.

If possible I would like to keep the implementation as it is, since it is robust and doesn't require further dependencies. However, I think there is a massive possible performance boost by turning of the transitive resolution process for further dependencies for this specific internal class scanning. This way during the discovery process ArchUnit would really only import the classes within the package, which should be fast, and refrain from importing all sorts of JDK and library classes.

I published a SNAPSHOT where I implemented this, could you try if this is fast enough for your usual test flow? This will still use ArchUnit to scan the classes within the given package to test, but should optimize the importer.

<repositories>
    <repository>
        <id>sonatype-snapshots</id>
        <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>com.tngtech.archunit</groupId>
        <artifactId>archunit-junit5</artifactId>
        <version>0.24.0-discovery-SNAPSHOT</version>
        <scope>test</scope>
    </dependency>
</dependencies>

In the end each TestEngine must implement a way somehow to discover all test candidates, given just a package like com.foo.bar. And this is actually not trivial, because such a package can in theory reside anywhere on the classpath. One way to avoid this is to simply specify a list of test classes to run, instead of a package. Then no classpath scanning would be necessary.

[esiha]

I've tested your snapshot with the sample repository I've mentioned earlier. I confirm the scan is still present but much faster as it is now constrained only to the classes present in the project.

To me, it's an acceptable workaround I'd be happy to use on a daily basis.

[codecholeric]

Cool, then I'm gonna create a PR an merge it 👍

[codecholeric]

-> #881