[deserialize] Unicode escape "\u0000" is silently ignored

[bupjae]

ArduinoJson version: v6.18.3
Target platform: x86-64
Toolchain: gcc (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0

Code snippet:

#include <iostream>
#include <string>

#include "ArduinoJson-v6.18.3.hpp"

using namespace std;
using namespace ArduinoJson;

int main() {
  StaticJsonDocument<1024> doc;
  cout << deserializeJson(doc, R"({"data": "wx\u0000yz"})") << endl;
  string value = doc["data"];
  cout << value << endl;
  cout << value.length() << endl;
}

Excepted result (NUL character is replaced to '_' for visibility):

Ok
wx_yz
5

Actual result:

Ok
wxyz
4

I understand that NUL character inside string is very hard to handle for C/C++ code. However, I think this behavior should be documented at least.

[bblanchon]

Hi @bupjae,

You're right; there was something wrong: instead of including the null in the string, it dropped it entirely.
I fixed this problem in the main branch.

Now, if you extract a const char*, you'll see that it includes all characters; however, it cannot return an std::string with the correct size because it stores strings as pointers (and naturally expects them to be null-terminated).
I could fix this, but I'm afraid it would increase the size of the library for nothing.
So, unless you have a solid use case, I think this change can wait v7.

I agree to add a word about this in the documentation, but the question is: where?
What do you think, @bupjae? Where would you expect to find such information?

Best regards,
Benoit

[bupjae]

It seems that unless JsonDocument holds length for each involved strings, NUL character inside document can cause various problems across all places. For example, after applying patch, output of serializeJson(doc, cout) is {"data":"wx"}. If such strings go to key of JsonObject, this issue will become more complicated.

I agree that the priority is low as this is very artificial example and may not have solid use case. Until then, we should be documented this issue; for example, "Strings containing NUL character (ASCII code 0, or \u0000) is not (fully) supported".

Suggested places to document this issue:

• In troubleshooter after https://arduinojson.org/v6/troubleshooter/#runtime/serialization/incomplete/overflowed-0 , additional question "Does your JSON has strings containing NUL character?"
• In troubleshooter after https://arduinojson.org/v6/troubleshooter/#runtime/deserialization/ok , additional item "JsonDocument is incomplete" and then "Does your JSON has strings containing NUL character?"
• Create item "Known issues" inside https://arduinojson.org/v6/doc/
• https://arduinojson.org/v6/api/config/decode_unicode/

[bblanchon]

I documented the issue in the four suggested locations.
Thank you very much for reporting this issue, @bupjae.

[bblanchon]

The fix was release in ArduinoJson 6.19.0.