ConsoleLauncher should output diff of expected and actual String values

[carolosf]

As a developer if I am running JUnit tests using the Console Launcher on the command line if two large strings are found to be different and I get "expected:" "but was:" if I want to see what was different I need to copy the strings and manually diff them.

It would be easier if there was --details flag or another flag (I think another flag is a better option) that would show me a diff between the two strings for actionable results. In most cases using an IDE is suitable but in workflows such as Bazel and CI it would be useful if a diff would be displayed.

Apologies if this is already supported didn't find anything obvious in the docs.

[marcphilipp]

Potential library to implement this: https://github.com/java-diff-utils/java-diff-utils

@carolosf We've briefly discussed this in our team call today and think this would be feasible since AssertionFailedError from opentest4j carries actual and expected in separate fields.

We'll have to take a closer look first to decide whether --details diff would be the right way of enabling this. Maybe it could even be enabled by default.

[carolosf]

@marcphilipp

If it becomes the default behaviour it might break IDEs and plug-ins - which is why I suggested a totally separate flag.

I didn't spend much time on it but I noticed previously that the Bazel plugin for IntelliJ does a regex for "expected but was" and splits the results.

Thanks for discussing though - if implemented it would significantly reduce my debugging time!

[marcphilipp]

@carolosf IDEs and build tools usually do not run tests via ConsoleLauncher but use the Launcher API directly so this shouldn't be a problem.

[marcphilipp]

Team decision: Change ConsoleLauncher output for test failures to include diff if actual and expected of AssertionFailedError are of type CharSequence.

[marcphilipp]

@carolosf Would you be interested to work on a PR for this?

[bechte]

Hey @marcphilipp & @carolosf - if ok for you, I'd like to provide a PR for this. Any veto?

[carolosf]

@carolosf Would you be interested to work on a PR for this?

Would love to but I don't have any capacity to work on this in the near future right now.

[carolosf]

Hey @marcphilipp & @carolosf - if ok for you, I'd like to provide a PR for this. Any veto?

I'm not working on this.

@marcphilipp Anything to add?

[marcphilipp]

@bechte Sure, go ahead

[bechte]

To introduce a new dependency to a library, I assume that I am extending the list in gradle/libs.versions.toml, e.g.:

java-diff-utils = { module = "io.github.java-diff-utils:java-diff-utils", version = "4.12" }

And with that plus running a build, I am able to add the following to the platform console project:

dependencies {
	/*...*/
	implementation(libs.java.diff.utils)
	/*...*/
}

Questions to the team, FYI: @marcphilipp, @sbrannen

• Is that the current approach we are taking on dependencies?
• Furthermore, would you like to shadow this dependency?

[bechte]

I have added some questions within the PR #3397.

[XJ114514]

Hi, I am currently a college student that wants to contribute to Junit 5 project.
Can I do a PR for this issue? Please assign to Xj114514(green avatar.)
As I had just made a commit that successfully located the ConsoleLauncher output and print out expected and actual value as the Team decision stated.

I tested the code by building a console standalone jar to run one test case I wrote.
I spent the entire day to search for it as I knows nothing about Gradle, Junit 5 , and next I will learn how to use the diff library you provide to finish this issue.

ps: Can someone look at my code to see if I did anything wrong? it only change a tiny part of original code.

[XJ114514]

About the auto test case, I can not think a good way to test it, Could you please give me some idea how to implement the execution below?
A fail execute with AssertionFailedError AND the expected and the actual value are both charSequence.

[XJ114514]

Conclusion of the problems that I had encountered:

1. 4 failed Test cases about adding the external module io.github.javadiffutils as dependency

   • All the test cases seem to result in Module Not Found, while the gradle can successfully build and fetch the jar normally.

     • If I remove the io.github.javadiffutils from the module-info.java (documenting purpose), the project can build and the standalone will have the extra diff functionality when comparing two different strings.
   • list of failed test cases
     • JarDescribeModuleTests [ [7] module=junit-platform-console ]
     • ModularUserGuideTests [ runTestsFromUserGuideWithinModularBoundaries(Path) ]
     • StandaloneTests              [ printVersionViaModule() ]
     • ToolProviderTests           [ findAndRunJUnitOnTheModulePath() ]
2. No clues in how to write Auto Test cases
   • The DemoHierarchicalTestEngine dummyTestEngine does not support making a test comparing two strings.

I will try my best to see if I can solve them. Because of the enormous amount of codes without any annotation, this could be all I can do in making solution this issue.