fix: flakiness in org.json.junit.JSONObjectTest#valueToString #2
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem:
The string in the assertion can change, because the Map which is used to store the data in the JSONObject returns the data in non-deterministic order (the order of child elements in JSON strings do not matter, and JSON strings are equal regardless of the ordering of the elements on the same hierarchy level)
The flaky test was found by using the NonDex tool.
The flakiness was discovered on the following lines
JSON-java/src/test/java/org/json/junit/JSONObjectTest.java
Lines 2028 to 2029 in 01727fd
as well as
JSON-java/src/test/java/org/json/junit/JSONObjectTest.java
Lines 2039 to 2040 in 01727fd
Solution:
Changed the assert statement from a JUnit Assertion to a JSON Assertion, so the strings are not compared charwise but are compared the way as specified for JSON strings (not taking care of the order of the elements on the same level).
The JSONAssertion library was chosen because it provides general assertions to compare JSON strings without the need of writing complex JUnit Assertions, what would need a lot of work to convert the strings in a different data structure, a lot of boilerplate code as well as the chance of adding more errors than fixing.
Result:
The test is deterministic and not flaky. This improves the quality of the test and reduces the time to search for the bug during future development.
Reproduce: