Skip to content

Commit

Permalink
Fixed Flaky Tests Caused by JSON permutations
Browse files Browse the repository at this point in the history
###Description
Flaky Tests found using NonDex by running the commands -
mvn -pl . edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.json.junit.XMLTest#testIndentComplicatedJsonObject

mvn -pl . edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.json.junit.XMLTest#testIndentSimpleJsonArray

mvn -pl . edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.json.junit.XMLTest#testIndentSimpleJsonObject

The logged failure was-

[ERROR] Failures:
[ERROR] XMLTest.testIndentSimpleJsonObject:1193 expected:<...>
<employee>
<[married>true</married>
<name>sonoo</name>
<salary>56000</salary]>
</employee>
</Te...> but was:<...>
<employee>
<[name>sonoo</name>
<salary>56000</salary>
<married>true</married]>
</employee>
</Te...>

The issue is the same for all three tests, so here I only show the failure message for the third test (to reduce the length of the error message).

### Investigation

The tests fail with a comparison error while comparing an expected JSON String and the result from the value returned from XML.toString(). The toString function of XML makes no guarantees as to the iteration order of the attributes in the object. This makes the test outcome non-deterministic, and the test fails whenever the function returns a mismatch in order of the elements in the JSON String. To fix this, the expected and actual keys should be checked in a more deterministic way so that the assertions do not fail.

### Fix

Expected and Actual values can be converted into JSONObject and the similar function can be used to compare these objects. As this function compares the values inside the JSONObjects without needing order, the test becomes deterministic and ensures that the flakiness from the test is removed.

The PR does not introduce a breaking change.
  • Loading branch information
superMaaax committed Mar 22, 2023
1 parent 8353b9c commit 48fb526
Showing 1 changed file with 9 additions and 3 deletions.
12 changes: 9 additions & 3 deletions src/test/java/org/json/junit/XMLTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -1111,6 +1111,7 @@ public void testIndentComplicatedJsonObject(){
"}" ;
JSONObject jsonObject = new JSONObject(str);
String actualIndentedXmlString = XML.toString(jsonObject, 1);
JSONObject actualJsonObject = XML.toJSONObject(actualIndentedXmlString);
String expected = "<success>true</success>\n" +
"<response>\n" +
" <dateTimeISO>2022-10-05T00:00:00+03:00</dateTimeISO>\n" +
Expand Down Expand Up @@ -1170,7 +1171,8 @@ public void testIndentComplicatedJsonObject(){
" <timestamp>1664917200</timestamp>\n" +
"</response>\n" +
"<error>null</error>\n";
assertEquals(actualIndentedXmlString, expected);
JSONObject expectedJsonObject = XML.toJSONObject(expected);
assertTrue(expectedJsonObject.similar(actualJsonObject));


}
Expand All @@ -1183,14 +1185,16 @@ public void testIndentSimpleJsonObject(){
" }}";
JSONObject jsonObject = new JSONObject(str);
String actual = XML.toString(jsonObject, "Test", 2);
JSONObject actualJsonObject = XML.toJSONObject(actual);
String expected = "<Test>\n" +
" <employee>\n" +
" <name>sonoo</name>\n" +
" <salary>56000</salary>\n" +
" <married>true</married>\n" +
" </employee>\n" +
"</Test>\n";
assertEquals(actual, expected);
JSONObject expectedJsonObject = XML.toJSONObject(expected);
assertTrue(expectedJsonObject.similar(actualJsonObject));
}

@Test
Expand All @@ -1201,6 +1205,7 @@ public void testIndentSimpleJsonArray(){
"] ";
JSONArray jsonObject = new JSONArray(str);
String actual = XML.toString(jsonObject, 2);
JSONObject actualJsonObject = XML.toJSONObject(actual);
String expected = "<array>\n" +
" <name>Ram</name>\n" +
" <email>Ram@gmail.com</email>\n" +
Expand All @@ -1209,7 +1214,8 @@ public void testIndentSimpleJsonArray(){
" <name>Bob</name>\n" +
" <email>bob32@gmail.com</email>\n" +
"</array>\n";
assertEquals(actual, expected);
JSONObject expectedJsonObject = XML.toJSONObject(expected);
assertTrue(expectedJsonObject.similar(actualJsonObject));


}
Expand Down

0 comments on commit 48fb526

Please sign in to comment.