serializeJson() continues appending to String when allocation fails

[softplus]

I'm using deserializeJson / serializeJson to receive, store, send data and noticed weird behavior for "longish" inputs. My guess is something is being incorrectly buffered and overwriting itself, leading to invalid serialized JSON output.

This is on Arduino 1.8.13, using ArduinoJson 6.18.0, compiling for ATmega32U4 on Mac OSX 11.3.1

Code:

#include <ArduinoJson.h>

void setup() {
  Serial.begin(115200); delay(2000);

  DynamicJsonDocument doc(1000);

  //String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}],\"led\":[{\"p\":4,\"t\":1,\"c\":{\"p\":1,\"v\":2,\"i\":255,\"d\":10},\"s\":[2]}]}"; // fails
  String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}],\"led\":[{\"p\":4,\"t\":1,\"c\":{\"p\":1,\"v\":2,\"i\":255,\"d\":10},\"s\":20}]}"; // fails
  //String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}],\"led\":[{\"p\":4,\"t\":1,\"c\":{\"p\":1,\"v\":2,\"i\":255,\"d\":10},\"s\":2}]}"; // ok
  //String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}],\"led\":[{\"p\":4,\"t\":1,\"c\":{\"p\":1,\"v\":2,\"i\":255,\"d\":10}]}"; //ok
  //String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}]}"; // ok

  Serial.println("Deserializing: ");
  Serial.print("input ="); Serial.println(s);
  Serial.print("len="); Serial.println(s.length()); /* Serial.println(strlen(s)); */ 
  DeserializationError error = deserializeJson(doc, s);
  Serial.print("error="); Serial.println(error.c_str());
  int input_length = s.length();

  // Serialize again
  String result;
  Serial.println("Serializing: ");
  int bytes = serializeJson(doc, result);
  Serial.print("bytes="); Serial.println(bytes);
  Serial.print("result="); Serial.println(result);
  Serial.print("len="); Serial.println(result.length());

  int output_length = result.length();
  if (input_length == output_length) { Serial.println("PASS"); } else { Serial.println("FAIL"); }
}

void loop() { }

Uncommenting / commenting a line with "fails" fails, one with "ok" works. Longer ones all fail. The difference seems to be in string length, since changing an int parameter from "2" to "20" triggers the same behavior (data size should be the same). serializeJson returns the correct number of bytes, but the output string is shorter, and sometimes invalid JSON.

Sample output:

10:05:48.591 -> Deserializing: 
10:05:48.591 -> input ={"cmd":"set","ver":2,"key":[{"p":2,"t":1,"k":["textsingleoh boy","textdouble"]},{"p":3,"t":2,"k":["2extsingle","2extdouble"]}],"led":[{"p":4,"t":1,"c":{"p":1,"v":2,"i":255,"d":10},"s":20}]}
10:05:48.591 -> len=189
10:05:48.591 -> error=Ok
10:05:48.591 -> Serializing: 
10:05:48.591 -> bytes=189
10:05:48.591 -> result={"cmd":"set","ver":2,"key":[{"p":2,"t":1,"k":["textsingleoh boy","textdouble"]},{"p":3,"t":2,"k":["2extsingle","2extdouble"]}],"led":[{"p":4,"t":1,"c":{"p"}]}
10:05:48.591 -> len=158
10:05:48.591 -> FAIL
...
10:06:23.389 -> Deserializing: 
10:06:23.389 -> input ={"cmd":"set","ver":2,"key":[{"p":2,"t":1,"k":["textsingleoh boy","textdouble"]},{"p":3,"t":2,"k":["2extsingle","2extdouble"]}],"led":[{"p":4,"t":1,"c":{"p":1,"v":2,"i":255,"d":10},"s":2}]}
10:06:23.389 -> len=188
10:06:23.389 -> error=Ok
10:06:23.389 -> Serializing: 
10:06:23.389 -> bytes=188
10:06:23.389 -> result={"cmd":"set","ver":2,"key":[{"p":2,"t":1,"k":["textsingleoh boy","textdouble"]},{"p":3,"t":2,"k":["2extsingle","2extdouble"]}],"led":[{"p":4,"t":1,"c":{"p":1,"v":2,"i":255,"d":10},"s":2}]}
10:06:23.389 -> len=188
10:06:23.389 -> PASS

[bblanchon]

Hi,

The moment I saw DynamicJsonDocument doc(1000); and ATmega32U4 together, I knew it would be an out-of-memory error.
In this case, the result string fails to allocate a buffer large enough to store the remaining output.

You can fix this bug by reducing the capacity of the JsonDocument, which will leave more room for the String to grow.
You can also preallocate a buffer of the right size by calling result.reserve().
However, I recommend that you avoid using the heap on AVR because there is just too little RAM on these boards; instead, I recommend using a StaticJsonDocument and a char[].

Nevertheless, I'm still classifying this issue as a bug because serializeJson() should have stopped writing to the string as soon as it failed, i.e., just after "p".

Thank you very much for reporting this issue.

Best regards,
Benoit

[bblanchon]

Unfortunately, I cannot fix this issue on Teensy because of PaulStoffregen/cores#561.

[bblanchon]

The fix is included in ArduinoJson 6.18.1