Fixed Flaky Tests Caused by JSON permutations #734
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Flaky Tests found using NonDex by running the commands -
mvn -pl . edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.json.junit.XMLTest#testIndentComplicatedJsonObject
mvn -pl . edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.json.junit.XMLTest#testIndentSimpleJsonArray
mvn -pl . edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.json.junit.XMLTest#testIndentSimpleJsonObject
The logged failure was-
[[ERROR] Failures:
[ERROR] XMLTest.testIndentSimpleJsonObject:1193 expected:<...>
<[married>true
sonoo
56000</salary]>
</Te...> but was:<...>
<[name>sonoo
56000
true</married]>
</Te...>
The issue is the same for all three tests, so here I only show the failure message for the third test (to reduce the length of the error message).
Investigation
The tests fail with a comparison error while comparing an expected JSON String and the result from the value returned from XML.toString(). The toString function of XML makes no guarantees as to the iteration order of the attributes in the object. This makes the test outcome non-deterministic, and the test fails whenever the function returns a mismatch in order of the elements in the JSON String. To fix this, the expected and actual keys should be checked in a more deterministic way so that the assertions do not fail.
Fix
Expected and Actual values can be converted into JSONObject and the similar function can be used to compare these objects. As this function compares the values inside the JSONObjects without needing order, the test becomes deterministic and ensures that the flakiness from the test is removed.
The PR does not introduce a breaking change.