Quarkus code coverage with failsafe

[danielpetisme]

Not sure if it's a bug or a question so feel free to route me.

I followd the guide explaining how to compute a code coverage with Quarkus
https://quarkus.io/guides/tests-with-coverage

Normally, surefire is used for unit test only where failsafe is the proper plugin for integration testing. So I tried to update the test with failsafe and... it does not work

Here you have a reproducer
https://github.com/danielpetisme/test-jacoco-failsafe
SImply run ./mvnw clean verify and you'll the jacoco file produced for the unit tests but skipped for the integration tests.

I discuss this point with @cescoffier.

While surefire only relies on target/classes to run the tests the jacoco agent can rely on the coverage instrumentation where failsafe use only the jar which 1- is not using the instrumented class 2- you don't want to pollute your final jar with jacoco instrumentation.

Finally I fallback to surefire to run Integration tests but I think it worth a mention in a documentation somewhere.

WDYT?

[stuartwdouglas]

Do you mean the native image integration tests? I don't think it will ever be possible to get code coverage for native image tests.

[danielpetisme]

Hello, No I mean Java integration tests. Jacoco and Quarkus byte instrumentation looks not compatible + I'm not sure if failsafe is using the final jar or `target/classes` to run. Try to run the reproducer it'll clearer ;-) Le jeu. 12 mars 2020 à 02:13, Stuart Douglas <notifications@github.com> a écrit :

…

Do you mean the native image integration tests? I don't think it will ever be possible to get code coverage for native image tests. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7536 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEQ5HCBG7FRXFTZB5DG76TRHAZNHANCNFSM4LANINWQ> .

[stuartwdouglas]

I think the solution here is to use the agent instead of the instrument task. https://github.com/quarkusio/quarkus/blob/master/build-parent/pom.xml#L677 gives an example for surefire, but I assume it is similar for failsafe.

[pilhuhn]

@danielpetisme Do you have an error message for "not compatible" ?

[danielpetisme]

Hello sorry for this delayed answer.... I have to do a reproducer, I'll share it ASAP Le lun. 22 juin 2020 à 15:51, Heiko W. Rupp <notifications@github.com> a écrit :

…

@danielpetisme <https://github.com/danielpetisme> Do you have an error message for "not compatible" ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7536 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEQ5HE6JNRRVPRUOCOXZODRX5OU5ANCNFSM4LANINWQ> .

[bvarner]

Bump on this issue:

I've been trying to get Jacoco to report coverage from failsafe tests, and I cannot get it to do it.
I've set everything up to use the agent, but when the jar is executed by io.quarkus.test.common.JarLauncher the agent is never included.

Looking into the JarLauncher, you can see the following, which corresponds to my logging output from the failsafe plugin:

public void start() throws IOException {

        System.setProperty("test.url", TestHTTPResourceManager.getUri());

        List<String> args = new ArrayList<>();
        args.add("java");
        args.add("-Dquarkus.http.port=" + httpPort);
        args.add("-Dquarkus.http.ssl-port=" + httpsPort);
        // this won't be correct when using the random port but it's really only used by us for the rest client tests
        // in the main module, since those tests hit the application itself
        args.add("-Dtest.url=" + TestHTTPResourceManager.getUri());
        Path logFile = PropertyTestUtil.getLogFilePath();
        args.add("-Dquarkus.log.file.path=" + logFile.toAbsolutePath().toString());
        args.add("-Dquarkus.log.file.enable=true");
        if (profile != null) {
            args.add("-Dquarkus.profile=" + profile);
        }
        for (Map.Entry<String, String> e : systemProps.entrySet()) {
            args.add("-D" + e.getKey() + "=" + e.getValue());
        }
        args.add("-jar");
        args.add(jarPath.toAbsolutePath().toString());

        System.out.println("Executing " + args);

        Files.deleteIfExists(logFile);
        quarkusProcess = LauncherUtil.launchProcess(args);
        ListeningAddress result = waitForCapturedListeningData(quarkusProcess, logFile, jarWaitTime);
        updateConfigForPort(result.getPort());
        isSsl = result.isSsl();
    }

As you can see, there is no way to inject an argline argument.