Wildcard filter: remove empty objects

[jshep321]

Hi,
I wrote a sample .ino to demonstrate the challenge from https://github.com/bblanchon/ArduinoJson/issues/1309 with descriptions in the serial printout. File is here: https://gist.github.com/jshep321/2b9ffb745edfa9dc121257656d811a07

Note that for the json format I would prefer an array for the "data" level, but the Google cloud database converts arrays to object automatically, so I'm a bit out of luck without restructuring the data or starting elsewhere!

Output from the program:

-----------initial json input string: -----------
{"USER1": { "data" : { "1" : "item 1 description", "2" : "item 2 description"}, "associated user data" : "value"}, "USER2": { "data" : { "10" : "item 10 description", "11" : "item 11 description"} ,"associated user data" : "value"}}
-----------fullUserDB after prettify (no filter yet): -----------
{
  "USER1": {
    "data": {
      "1": "item 1 description",
      "2": "item 2 description"
    },
    "associated user data": "value"
  },
  "USER2": {
    "data": {
      "10": "item 10 description",
      "11": "item 11 description"
    },
    "associated user data": "value"
  }
}
make a filter -- 
ideally I want to keep the target data of "10" and associated higher levels only 
--> { USER2, data : { 10 : item 10 description }, associated user data: value } <--
Filter: 
filter["*"]["data"]["10"] = true
-----------filtered userDB after pretty+filter: -----------
{
  "USER1": {
    "data": {}
  },
  "USER2": {
    "data": {
      "10": "item 10 description"
    }
  }
}
Result: Keeping unwanted USER1 and truncated wanted associated user data

Help appreciated or else I might have to do wonky stuff to search and filter this json object.

Thanks!

[bblanchon]

Hi @jshep321,

Thank you very much for this explanation; I can clearly see what you want.
In #1309, I said that this is the expected behavior, but I'm not so sure anymore.

On the one hand, what's the point of keeping the empty object if there is no desired value in it?
But on the other hand, we're changing the layout of the document: we had two users, and now just one...
Also, how do we deal with objects in arrays? If we remove them, then the indexes will change, which is likely to cause bugs.
Maybe the remove-empty-object feature should be used only if there was a wildcard in the path... but the implementation (and the tests!) becomes quite complicated.

@jshep321, why is this such a big deal to get the empty objects?
Sure, it wastes a little space, but you should still have a lot of room in your ESP32.

Best regards,
Benoit

[jshep321]

Good points @bblanchon... it is complex for sure.

IMHO it's perfectly ok to change the structure of the (non-wanted parts of the) document after filtering. The core idea of a filter is to remove unwanted stuff and keep wanted stuff. Filtering out all undesired stuff saves resource -- RAM and CPU cycles. Otherwise we will have to re-process the filtered output, which makes the filtering step not really useful in this case. Since I don't see an easy way to remove it with the ArduinoJson library, I would have to introduce either a new mechanism/library or make my own.

The more I think about it, I realize that I'm probably half-wrong in my idea of the filter; 1) correct: removing USER1's structure is desirable but 2) keeping the "associated user data" is likely incorrect (since I am not explicitly filtering on it).

Regarding the ESP32: I'm starting with it because it's got more RAM/CPU -- I plan to scale down to smaller MCUs. Also, my user DB is much larger than 2 users. So the case of keeping "USER1" and it's empty structure in this example will be expanded both in depth of each user and number of users.

Perhaps a filter bool option to prune unfiltered objects in the tree would be the best path? That way you would allow current users (especially arrays) to not worry about unexpected behavior, but allow use cases like mine keep the filter as a single processing step?

[jshep321]

@bblanchon FYI, here is an overview of the Google firebase (non-array) challenge explained in a much clearer way than I explained above.
https://www.tutorialspoint.com/firebase/firebase_arrays.htm

In order to search for the index of name "amanda" in that example, I would try a filter of
filter["players"]["*"]["name"]["amanda"] = true

but this would not work and I don't think there is a way in arduinojson to return the key "1" to then use to retrieve the wanted data.

[bblanchon]

"amanda" is not a key but a value, right?
So this is not just a filter but an actual search query, as you would do on a database with a WHERE clause.
While it's not a bad idea, it's clearly more involved than the current filtering feature...

[jshep321]

Yes, you caught me... I am trying to use your filter to do actual searching (on both keys and values). :) I can see a need to do this in a resource efficient way and your filter mechanism is so very close!

[bblanchon]

I agree this would be a useful feature, but I don't think I'll be able to implement it any time soon.
I suggest that we keep this issue to track the "remove empty objects" feature, and if you want, we can open another feature request for the query functionality.