Parameterized tests start failing after upgrading to 1.11.2.0 and 1.6.0 #358
Comments
|
Thanks for reporting – what an esoteric stack trace! I will try to reproduce this myself, but please allow me a few follow-up questions first:
I do know that there is apparently some fixes related to parallel testing in JUnit 5.11.2, so I wonder if there is an unexpected side effect for the Android environment introduced by those. If you want, you could also try the intermittent plugin version 1.11.1.0, which also ships with instrumentation 1.6.0. Maybe if that also works, we have narrowed down the problem surface. Thanks in advance! |
|
I appreciate how much thought you already put into this! I didn't want to make you create the reproducer, so I managed to make it myself. It did take a while, and I'd like to open a discussion about why debugging this was so difficult. Here is the reproducer. Run As you will see:
The much larger issue here, in my opinion, is how difficult this was to debug. Why was the error swallowed? Why is not shown even in the debug log? Why does gradle report that "tests failed" when actually the test framework itself failed? I don't want to risk blaming this library, as I understand this may be an AGP issue. But I have to say, I recall many instances since I started working with android testing where I experience an error thrown from the test framework but not showing anywhere and misleading me about the source of the problem. Is there anything I can be doing better on my end to get to the source of these types of issues faster? |
|
I will add that I am observing this in the debug log only during the case where But then I'm unable to find the stack trace, class, or message from this exception in the log. The gradle debug log is large - could I be missing it? |
|
Great work getting that reproducer put together, I appreciate that a lot. The continued mentioning of UTP doesn't bode well since we don't have any real way to control its internals, however this should work as an initial point of investigating! No promises of a timeframe here, but I'll mark this for now and will return as soon as I can. 🙇♂️ |
Everything works as expected using versions 1.11.0 and 1.5.0.
Then after upgrading to 1.11.2 and 1.6.0, @ParameterizedTest causes an exception to be thrown from some internal context. This exception is very difficult to see as it doesn't show up for me in the test report or the regular build output. I only see it with debug output. The build says one of my parameterized tests failed, which is misleading since the exception came from some internal issue in the test framework.
I want to also say I don't really understand what's going on here, so if the issue isn't clear please let me know what further information I can provide. Creating a reproducer might take a while, but I can try that as a last resort.
For now, downgrading back to 1.11.0 and 1.5.0 prevented the issue from happening.
The text was updated successfully, but these errors were encountered: