Sequential reading of JSON arrays

[gjgiezeman]

I have an application that logs events as an array of JSON objects. The objects are written one by one to some network stream. I would like to process them when they arrive, not after the application has closed and the array is complete.

So, I would like to have (or be able to write) a class with an interface like this:

class SeqJsonReader {
  public:
    void append(std::string const &str);
    void append(char const *str);
    void append(char const *begin, char const *end);
    bool has_value() const;
    nlohmann::json next_value();
    bool at_end() const; // true if closing ']' of array is processed.
};

Usage could be like this:

bool reader_loop(vector<string> const &msgs)
{
    SeqJsonReader reader;
    auto cur = msgs.begin();
    while (!reader.at_end() && cur!=msgs.end()) {
        try {
            reader.append(*cur++);
            while (reader.has_value()) {
                nlohmann::json val = reader.next_value();
                // do something with val
            }
        } catch (...) {
            return false;
        }
    }
    return true;
}

int main()
{
    vector<string> msgs;
    msgs.push_back(R"([{"product":"apple","amount":"512","unit":"gram"})");
    msgs.push_back(R"(,{"product":"pear")");
    msgs.push_back(R"(},42,{"product":"orange","amount":"402","unit":"gram"}])");
    reader_loop(msgs);
}

In practice the strings (msgs) would come from a recv call of a socket or something like that.

Am I right that currently this is hard to implement? It seems that I can't parse partial JSON input.

[gregmarr]

Are you able to change the format at all? Could you write as individual objects instead of an array? Essentially JSON Lines http://jsonlines.org/
I'm pretty sure that the latest version can handle this using some of the istream methods.

[pingsoli]

@gregmarr I'm really sorry to bother you, I have a question about json array of string, it doesn't have a key, how can I detect the value in the array ? Following is my solution

json j = { 1, 2, 3, 4, 5 };
for (auto &e : j)
{
  if (e == 2)
  {
    // json array contains 2
    std::cout << "Found 2" << std::endl;
    // do something 
  }
}

Does their has any function like this ?

json j = { 1,  2, 3, 4, 5 };
if (j.contains(2)) 
{
  // do something 
}

[nlohmann]

@pingsoli You can use std::find (http://en.cppreference.com/w/cpp/algorithm/find), just like you would for a std::vector.

[pingsoli]

@nlohmann I appreciate your reply, thank you very much. Your reply is really fast, thanks for your replay again. Have a good day. : )

[gjgiezeman]

Gregg, thank you for your suggestion. I'll see if I can convince upstream to change the format.

[nlohmann]

You could try to use a callback function which you can pass to the parse function, see https://nlohmann.github.io/json/classnlohmann_1_1basic__json_aecae491e175f8767c550ae3c59e180e3.html#aecae491e175f8767c550ae3c59e180e3:

#include <iostream>
#include "json.hpp"

using json = nlohmann::json;

int main(int argc, char* argv[]) {
    json x = {1,2,3,4,5,6,7,8,9};
    std::string s = x.dump();

    json::parser_callback_t cb = [](int depth, json::parse_event_t event, json & parsed)
    {
        if (event == json::parse_event_t::value)
        {
            std::cout << parsed << std::endl;
            return false;
        }
        else
        {
            return true;
        }
    };

    
    json y = json::parse(s, cb);
    
    std::cout << y << std::endl;
}

The example shows how to process values individually (here: print to std::cout) and not store them (hence returning false). As a result, the last line returns null, because no value has been created with the parse call. The signature and the approach is different from the one you proposed, but I think you can build on it.

[nlohmann]

Can I close this issue?

[ataber]

Is there any way to do the callback-based parsing while reading from an ifstream incrementally?

[nlohmann]

I am afraid this is not possible in the moment. There is a private API, but it is not exposed publicly so far.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.