Add support for streaming JSON encoder/decoder

[pjsg]

Fixes #1666 (partially).

• This PR is for the dev branch rather than for master.
• This PR is compliant with the other contributing guidelines as well (if not, please describe why).
• I have thoroughly tested my contribution.
• The code changes are reflected in the documentation at docs/en/*.

This makes use of the jsonsl package (which is under a very liberal license). It wraps it in a layer to interface to lua for decoding json strings. There is also an incremental encoder.

The decoder can call out to lua code during the decoding to give the lua full control over which parts of the data structure should be retained (or handled inline).

It does provide an API which is compatible with cjson.

This is the first step to resolve the issue #1666. The next is to deal with the http client....

[djphoenix]

Overflows iROM... again. Hmmm...

[pjsg]

I think that we are getting close to where we can't increase the irom any more.... We have enough modules that they will not be able to be built at the same time!

[djphoenix]

Safe size for iROM is 0x100000 (1M - max mapped size) - 0x8000 (iRAM size) - 0x14000 (dRAM size) = 0xE4000.
If we will use a bootloader (e.g. rBoot), we may increase it to 0x100000 - 0x2000 (bootloader + config size) = 0xFE000, because iROM starts right after loader, and iRAM/dRAM may be placed over 1M bound.

[pjsg]

Also should resolve #1316 #1580

[jmattsson]

What is the intended use of setpath? The first argument appears to always be an empty table. Can I do something interesting with that table? Can I return something from the setpath call to indicate that I don't want this piece decoded/kept?

[pjsg]

I've added documentation to describe it better. I'm definitely open to making this a checkpath method instead that returns true if you want to keep the element, and false if not. Default would be true if the checkpath method was not supplied.

[jmattsson]

I haven't really thought through what the likely usecases are here, but I suspect a checkpath would be more readily understood. Or it could be that we just need some more docs or examples?

[jmattsson]

So this is the same value as from sjson.decoder:result() then? Do we really want both of these then? It seems redundant at first glance.

[pjsg]

It was a way to signal that decode is complete (if that is of interest). You want the result method so that you can always get the final result.

[marcelstoer]

Safe size for iROM is .... 0xE4000.

Then why don't we use that value - at least during PR builds?

[pjsg]

I changed the setpath to checkpath to allow you to suppress an entire object. Also, I fixed the maximum length of a string to be limited by memory rather than artificially constrained.

[marcelstoer]

Do you really want to maintain two JSON modules? If it's a drop-in replacement for cjson don't we want to dump, or at least deprecate, that one?

[pjsg]

I'd like to replace CJSON entirely. I didn't want to call this new code cjson, but maybe that is the easiest thing.....

[marcelstoer]

This value is also good enough to build #1231, I just tested. So, if we merge this one first we can finally merge that one as well.

[marcelstoer]

I didn't want to call this new code cjson, but maybe that is the easiest thing.....

Even though JSON support is quite popular (among top-10 non-default modules) I wouldn't do that and introduce a breaking change instead. Anyone building a new firmware will notice that CJSON is gone and will check the documentation. Of course the release notes will also state this but who reads release notes anyway.

[marcelstoer]

Oh, and ideally you keep cjson.md to avoid dead links even when the CJSON code is gone. The page would only need to show a brief note that the module was replaced by SJSON.

[pjsg]

This removes cjson (but leaves the documentation that now just points to sjson). Examples updated.

[jmattsson]

Sorry for the delay, I had it in my head this had already been merged. Many thanks @pjsg, I won't miss the cjson mess :)

[hanfengcan]

hi, I'm write a article about this module。 I have some difficulty understanding metatable when I use sjson.decoder()。

http://nodemcu.readthedocs.io only one example。Can anyone give more than one example?

and this is my articles http://www.jianshu.com/nb/7000517 ,Chinese Only

thanks

[pjsg]

The idea is to allow you to decode large JSON responses without needing to get the entire response into memory at once. One of the motivations that I had was to be able to store a nodemcu lua project in GIT and have the nodemcu auto-provision itself and auto-update itself. I haven't actually done this yet, but it is not that difficult. The steps are:

• Connect to (for example) https://api.github.com/repos/nodemcu/nodemcu-firmware/git/trees/master
• Parse the large returned JSON and extract the URLs for each of the blobs that are required and put the URL and path into a list.
• Now iterate over the list and fetch the blobs and write them to the file system

[Actually, now I read it, this is basically the example]

There are a bunch of JSON endpoints which return more data than the client actually wants. This allows that filtering. I'll dig up some more examples.

What aspect are you interested in?

[hanfengcan]

such as a weather @pjsg . Usually, weather api return json, and I need some infomation like temperature

[hanfengcan]

here is a free weater api
https://api.caiyunapp.com/v2/TAkhjf8d1nlSlspN/121.6544,25.1552/forecast.json
and return json.
But, the return data is larger(24K byte).

http model can't deal it, How can I do?

[pjsg]

I wrote this example piece of code that just gets the weather in London. It actually doesn't query the live API, but uses a sample data blob so it should print out "Drizzle"

The __newindex function is just selecting the single attribute main from the JSON and saving its value. Everything else is discarded. The HTTP fetch function doesn't build a single buffer with all the data in it, it just incrementally calls the decoder:write() method.

Does this help?

function getdata(host, path, callback, meta)
    local conn=net.createConnection(net.TCP, 0) 
    conn:on("connection", function(conn) 
        conn:send("GET " .. path .. " HTTP/1.0\r\nHost: " .. host .. "\r\n"
            .. "Accept: */*\r\n\r\n")            
    end)
    local decoder = nil
    
    conn:on("receive", function(conn, pl)    
        if not decoder then
            local location = string.find(pl, "\r\n\r\n")        
            if location ~= nil then
                pl = string.sub(pl, location + 4)
                decoder = sjson.decoder({ metatable=meta })
            else
                return
            end
        end    
        decoder:write(pl)
    end)
    conn:on("disconnection", function(conn)
       callback(decoder:result())
    end)
    conn:connect(80, host)
end

local main = nil
getdata("samples.openweathermap.org",
        "/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1", 
        function(r) print(main) end, 
        {checkpath=function() return false end,
         __newindex=function(t,k,v) if k == "main" then main = v end end})

[hanfengcan]

yes, thinks @pjsg

[dnc40085]

@pjsg Recently, I was looking at the Travis CI build log and saw a couple of warnings ELF binary has undefined symbol calloc and ELF binary has undefined symbol free which I traced back to app/sjson/jsonsl.c, I'm not sure if you noticed this when testing this PR.

[jmattsson]

I can confirm that I too am seeing these undefined symbol warnings. I'm surprised that they're only warnings, I'm sure they used to be errors.

[spotrevoc]

The docs explain that the __newindex method can be used to filter what goes into the new table by checking the key value and skipping the 'rawset' function.

When the checkpath method is called, the metatable has already be associated with the new table. Thus the checkpath method can replace it if desired. For example, if you are decoding { "foo": { "bar": [1,2,3,4], "cat": [5] } } and, for some reason, you did not want to capture the value of the "bar" key, then there are various ways to do this:

In the __newindex metamethod, just check for the value of the key and skip the rawset if the key is "bar". This only works if you want to skip all the "bar" keys.]
Can you please provide a working example of this for a nested table? I'm pulling the forecast data from api.weather.gov/gridpoints/STO/60,69/forecast and I only want to store everything under the "periods" path. I have figured out a way to break up the returned string into a string array and feed it back to the decoder via the decode:write() function.

This is the code i have so far
decode = sjson.decoder({metatable={__newindex=function(t,k,v) if k == "periods" then rawset(t,k,v) end end}})

But when I try to recall or iterate through decode:result() it turns out to be nil. What am I missing here?

[HHHartmann]

Please do not post on closed items.
And for any sort of developer support questions like "how do I ..." or a "I can't get this to work ...". See our support page for alternatives.