Possible to speed up json::parse?

[Noitidart]

Is it possible for me to give a hint to json::parse so it is faster?

I am receiving a JSON string from stdin. It contains a 1024kb string. Is there any tricks to make parse faster? Right now it takes 4.5 seconds.

Thanks

[nlohmann]

Can you provide the concrete example?

[Noitidart]

Oh wow SUPER fast reply thank you sir!

I currently do it like this:

auto payload = json::parse(payload_str);

payload_str is a std::string sent from javascript (NativeMessaging for browser extensions). This is the javascript:

 { method:'test', arg:new Uint8Array(new ArrayBuffer(1024*1024)).toString(), cbid:5 }

arg is a string of a byte array. So it is just 0,0,0,0,0,0,0,0,0,0,0,0,... (fake data, real data its a number between 0-255)

[nlohmann]

Thanks, I shall have a look.

[gregmarr]

So payload_str is something like this:

{ method:'test', arg: "0,0,0...0,0,0", cbid:5 }

What kind of hint were you thinking to send that would help parse that 1MB string faster?

[nlohmann]

I generated a file that fits your description, see issue398.json.zip. Then I ran a benchmark (using https://nonius.io) parsing this file. Here is the output:

benchmarking parse issue398.json
collecting 1000 samples, 1 iterations each, in estimated 37.623 s
mean: 38.9618 ms, lb 38.8074 ms, ub 39.1166 ms, ci 0.95
std dev: 2.48386 ms, lb 2.38355 ms, ub 2.61042 ms, ci 0.95
found 3 outliers among 1000 samples (0.3%)
variance is severely inflated by outliers

The mean parsing time is 38.96 milliseconds. I have no idea why parsing takes 4.5 seconds in your case.

How does your input file look like? How does your parsing code look like? What compiler flags are you using?

[Noitidart]

Thanks very very much for willing to discuss this with me, as it looks like it's a personal issue on my end!

I am new to wirting C++. I am using Visual Studio 2015.

Here is image of my code - http://i.imgur.com/OCmdZOC.png

(side note: i dont know why it underlines json::parse, but it doesnt prevent compiling - on hover it shows this message - http://i.imgur.com/OhSYpnr.png )

My compile flags I'm not sure, but I looked up how to see compile configuration on the net and it led me to this dialog box - http://i.imgur.com/FXv82EJ.png

Sincerest thanks!

[Noitidart]

@gregmarr yep that's correct it looks like that. I'm not sure, what hint to give I was just trying my luck to ask haha.

[whackashoe]

@Noitidart can you try compiling as release rather than debug

[Noitidart]

Thanks @whackashoe testing right now! :)

[gregmarr]

Debug Visual Studio STL is SIGNIFICANTLY slower than release due to all the extra checking that happens.

[Noitidart]

Oh wow! You were right it is now 49ms if I do it with "Release". My sincerest thanks @nlohmann, @whackashoe, and @gregmarr!!!

[whackashoe]

Glad it worked! Have a good one.

[Noitidart]

Hi folks, you guys were really helpful thank you! I have been working with json.hpp for about a month now and have some questions. Is there a place I can post this? I searched the issues and didn't find similar. I don't want to clog up the issues forum, I'm not sure if it's ok with you all. Is there somewhere you can recommend I can ask questions please?

[nlohmann]

@Noitidart Just open a new issue and ask. :)

[Noitidart]

You are too too fast haha thank you very much for being so friendly all of you to new comers!

[balannarcis96]

Is there a workaround for this? , i'm loading ~100mb of JSON and it takes about 2-3 minutes to parse in debug mode.

[nlohmann]

How fast is it in release mode?

[balannarcis96]

In release mode is as expected, very fast, but my main concern is debug mode, as the json data my program parses is big and can't be reduced very easily.