Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Request] Hide the JUnit Callstack from the stacktrace #1281

Closed
Tracked by #977
leogott opened this issue Aug 16, 2021 · 8 comments · Fixed by #1402
Closed
Tracked by #977

[Request] Hide the JUnit Callstack from the stacktrace #1281

leogott opened this issue Aug 16, 2021 · 8 comments · Fixed by #1402
Labels
enhancement
Milestone
0.35.0

Comments

@leogott
Copy link

leogott commented Aug 16, 2021

There are (few, but there are) cases in which I'd like to take a look at the callstack for a failed (as opposed errored) test. Mainly when using assertThrows. However I would like to have it shortened by default so that in the simplest case every line starting with at org.junit is replaced with the ... X more lines just like the end is shortened right now.

I'd be satisfied with a setting like "shorten lines matching this regex from the stack trace".

For example when assertThrows fails, because instead of ExpectedException I got DifferentException, I'd love the output to look like

org.opentest4j.AssertionFailedError: Unexpected exception type thrown ==> expected: <org.example.ExpectedException> but was: <org.example.DifferentException>
    at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:65)
    ... 70 more lines
Caused by: org.example.DifferentException: Message
    at x
    at y
    at z
@jdneo
Copy link
Member

jdneo commented Aug 19, 2021 •
edited
Loading

@leogott Could you give an example of what's the original stack trace looks like?

In the example you gave, why we just hide the first part of the stack trace but leave the second one as it is?

@leogott
Copy link
Author

leogott commented Aug 21, 2021

For a small example I wrote (junit5 assert throws expects A but receives B caused by C), I get this unabbreviated result.

Every line beginning with at org.junit.** (for example) is part of the testing implementation and I don't care about the junit internals, because I'm confident my own code is the likely cause of the failed test.

I don't care about the callstack of the AssertionFailedError (beyond that it was in line 25 of AppTest.java) , because it was thrown by the assertThrows method.

I do however care about the Cause (the exception of wrong type), and where it originated.

So in this case I'm getting the following information:

%TESTC  1 v2
%TSTTREE2,org.example.AppTest,true,1,false,1,AppTest,,[engine:junit-jupiter]/[class:org.example.AppTest]
%TSTTREE3,someFeature_stackTrace(org.example.AppTest),false,1,false,2,someFeature_stackTrace(),,[engine:junit-jupiter]/[class:org.example.AppTest]/[method:someFeature_stackTrace()]
%TESTS  3,someFeature_stackTrace(org.example.AppTest)
%FAILED 3,someFeature_stackTrace(org.example.AppTest)
%TRACES 
org.opentest4j.AssertionFailedError: Unexpected exception type thrown ==> expected: <java.io.IOException> but was: <java.lang.IllegalArgumentException>
        at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:65)
        at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:37)
        at org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:3007)
        at org.example.AppTest.someFeature_stackTrace(AppTest.java:25)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
        at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
        at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
        at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
        at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
        at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
        at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
        at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
        at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
        at java.util.ArrayList.forEach(ArrayList.java:1259)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
        at java.util.ArrayList.forEach(ArrayList.java:1259)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
        at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
        at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:84)
        at org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:98)
        at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:40)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:529)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:756)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:452)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)
Caused by: java.lang.IllegalArgumentException: Argument must not be null
        at org.example.App.someFeatureImTesting(App.java:20)
        at org.example.AppTest.lambda$0(AppTest.java:26)
        at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:55)
        ... 70 more
Caused by: java.lang.NullPointerException
        at org.example.App.someFeatureImTesting(App.java:17)
        ... 72 more
%TRACEE 
%TESTE  3,someFeature_stackTrace(org.example.AppTest)
%RUNTIME175


> Test run finished at 21.8.2021, 01:23:53 <

But this is what I care about:

TEST: someFeature_stackTrace(org.example.AppTest)
RESULT: failed
TRACE:
org.opentest4j.AssertionFailedError: Unexpected exception type thrown ==> expected: <java.io.IOException> but was: <java.lang.IllegalArgumentException>
        ... 3 more
        at org.example.AppTest.someFeature_stackTrace(AppTest.java:25)
        ... X more
Caused by: java.lang.IllegalArgumentException: Argument must not be null
        at org.example.App.someFeatureImTesting(App.java:20)
        at org.example.AppTest.lambda$0(AppTest.java:26)
        ... 71 more
Caused by: java.lang.NullPointerException
        at org.example.App.someFeatureImTesting(App.java:17)
        ... 72 more

Would it be possible to at least cut the callstack for the AssertionFailed Exception that fails the Test?

@jdneo
Copy link
Member

jdneo commented Aug 21, 2021

I see, so that is to only reserve the stack trace which belongs to the source?

@leogott
Copy link
Author

leogott commented Aug 21, 2021

Oh you're correct. I didn't look at it that way, but it absolutely is just the parts of the callstack within the source at test.

Yes, I'd like to see a filtered view or something that cuts away the callstack that doesn't belong to the source.

@jdneo
Copy link
Member

jdneo commented Nov 25, 2021

Get a very similar feedback from a customer interview. He hope the stacktrace can be hidden by clicking some buttons like show more, show less

@jdneo jdneo mentioned this issue Nov 25, 2021
Closed
This was referenced Apr 7, 2022
Closed
Merged
@jdneo
Copy link
Member

jdneo commented Apr 7, 2022

Hi @leogott,

I pushed a change to filter out those 'internal' traces. Currently we don't have a setting to control it. But we can definitely add a setting for that if users require.

You can try the new behavior in the next pre-release of the extension (should be available tomorrow).

image

Let me know if you have any feedback, thanks!

@jdneo jdneo added this to the 0.35.0 milestone Apr 7, 2022
@jdneo
Copy link
Member

jdneo commented Apr 22, 2022

To verify this issue

  • Run a JUnit test case which will fail.
  • Check the stacktrace, it should not contain following packages/classes:
    • org.eclipse.jdt.internal.junit.runner.*
    • org.eclipse.jdt.internal.junit4.runner.*
    • org.eclipse.jdt.internal.junit5.runner.*
    • org.eclipse.jdt.internal.junit.ui.*
    • junit.framework.TestCase
    • junit.framework.TestResult
    • junit.framework.TestResult$1
    • junit.framework.TestSuite
    • junit.framework.Assert
    • org.junit.*
    • java.lang.reflect.Method.invoke
    • sun.reflect.*
    • jdk.internal.reflect.*

// cc @testforstephen

@jdneo jdneo mentioned this issue Apr 22, 2022
Closed
16 tasks
@testforstephen
Copy link
Contributor

The latest implementation works well. One minor suggestion is to hide the highlighting two call stacks as well, since they doesn't provide any valid info for diagnose.
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
0.35.0
Development

Successfully merging a pull request may close this issue.

feat: Simplify the stacktrace microsoft/vscode-java-test
3 participants
@jdneo @testforstephen @leogott