Feature Request: Provide a skip GraalVM test annotation?

[dzou]

Large projects will have numerous unit tests and often use mocking libraries (like Mockito, EasyMock) which may have tricky reflection.

It would be nice to have an annotation like @SkipNativeTest to annotate individual unit tests that should be skipped in native image testing. This would allow projects to gradually adopt native-image testing (not have all tests working at the start) and also give some flexibility for tests that may not be possible to get working due to mocking libraries.

[dzou]

Just as a follow-up, we've been trying to get some larger projects doing GraalVM JUnit testing. We have some bigger projects where I found 216/499 tests created mocks using Mockito. And these don't seem to work because of the dynamic class creation discussed here: #76 (comment)

I'm thinking maybe we would like to have some automatic way of skipping these tests instead of a manual annotation. I'll do some research to see if this might be possible from a feature; perhaps modifying the list of tests being run produced by the initial mvn test.

[dzou]

@sbrannen -- Hey I was wondering what your recommendations are for developers trying to use this plugin for larger projects that may have tests using mocking libraries etc. For context, we are trying to apply the plugin for ~95 different projects in our company. I wonder if there is a way to either get the mocking tests to work or skip them? I am under the impressions that mocking libraries don't work based off internal usage of ClassLoader.defineClass.

@sdeleuze -- I recall Spring was one of the first projects to adopt native image unit testing. How did your team get around the mocking libraries issue?

[sbrannen]

Hi @dzou,

Thanks for raising the issue and opening up a public discussion!

We have been aware of such needs for tests for over a year now. This feature branch for the Spring Framework served as the proof of concept that ended up influencing the Gradle and Maven plugins in this repository.

That POC has not been kept up to date, so some of the tags used there may no longer apply. In any case, you can see the JUnit Jupiter ExecutionConditions and tags that I was using back then here.

For general use cases where the test method or test class in question could be compiled into a native image but not actually executed within a native image, I used the @DisabledInGraalVmNativeImage annotation.

For use cases that would not even compile into a native image, I used tags to prevent their inclusion in the native image -- for example, @UsesMockito which would apply to your use case.

You can see how the "unsupported" tags were used in my POC here.

As for how we want to go forward, there is a lot of room for discussion there.

We might choose to introduce annotations like @DisabledInGraalVmNativeImage and various @Uses* annotations.

We might find some way to automate the exclusion of things that just won't work in a native image.

So let's use this issue to brainstorm.

[dzou]

Thanks for getting back to me and sharing this info Sam! I see, the annotations are helpful.

I think for us, we would definitely be interested in exploring the automated method to try skipping these tests or just getting it to work. We are trying to get a large number of projects for our company using this plugin (it is really great!), and it would be a much more persuasive sell if this can be done automatically rather than asking project owners to annotate tests manually.

I can help with researching what is possible. I have some experience working with GraalVM Features, though I don't see anything in the interface that could be used to help us do this. The closest thing it offers are Reachability handlers, which is close to what we want, but they seem to lack the additional context that we need (such as the call path that led to the method). Do you know if there is any other method of registering hooks into the compiler that are stronger than what Features offer?

I also wonder if it is possible to extend the functionality native-image-agent somehow. Seeing it is possible to have caller-based filters for the tracing agent, it seems very close to being able to output a list of methods that access illegal native image operations.

The last thing is: instead of skipping, just trying to just get the tests using mocking libraries to work. Maybe it's possible with oracle/graal#2442. I'll give this a try in the next few days.

[sdeleuze]

Now tracked by junit-team/junit-framework#2830.

[sbrannen]

FYI: @EnabledInNativeImage and @DisabledInNativeImage conditions will be available out of the box beginning with JUnit 5.9.1.

• Introduce @EnabledInNativeImage & @DisabledInNativeImage conditions for GraalVM junit-team/junit-framework#2830