Functions json.get.path and json.set.path

[Merudo]

Is your feature request related to a problem? Please describe.
It can be slow and cumbersome to deal with nested jsons.

For example, to get a "Level" field from a json, I might need

[att = json.get(json, "PrimaryAttributes")]
[st = json.get(att, "ST")]
[stLevel = json.get(st, "Level")]

The situation is worse when I need to set a property deep inside a nested json, in which case I need to update each level of the json.

Describe the solution you'd like
Functions json.get.path() and json.set.path() to automate the process.

For example, in my example I would only need

[stLevel = json.get.path(json, "PrimaryAttributes", "ST", "LEVEL")]

Similarly for set.path, to change the Level to 5 (say),

[stLevel = json.set.path(json, 5, "PrimaryAttributes", "ST", "LEVEL")]

[Azhrei]

This is a great proposal, but I'd like to see good error messages for when the user messes it up! Suppose the JSON looks like:

{ name: { first: "Bob", last: "Smith" },
  "class": [ "cleric", "fighter", "cleric", "cleric", "fighter" ],
  hp:  [ 5, 9, 6, 4, 8 ],
  ...
}

To get to the number of HP for the third level of cleric it would be:

JSON.get.path(json, "hp", 3)

Because the first level is an object and the second level is an array.

Giving a number instead of a string can work for an object, but not for an array, so the error message on arrays is pretty easy: "cannot access array element using a string in JSON.get.path() in parameter 3", for example.

Otherwise, I think this is great. It would make accessing JSON data much simpler (at least, until such time as JSON becomes a first class data type).

[JamzTheMan]

I approve. I know a lot of JSON libs have a json.path built in. I haven't checked to see if the lib(s) we use have it or not.

They may or may not have standard error responses for errors.

[Merudo]

I don't think the package net.sf.json has json.path, unfortunately.

Regarding error messages, are there cases where we would prefer to return "" instead of an error?

[aliasmask]

This is the code I use in MapTool to json.set.path(), json.append.path() and json.get.path(). I was hoping if this was done in java there would be an easier method or at the very least be quicker in java.

@@ @json.set.path
<!-- json.set.path(jsonName,path,key,value)
   jsonName - json name to have data added to
   path - array of path to json data
   key - key name to insert
   value - value to insert
   json - original json will be updated using its name for reference.
   
   This will update the original json and insert data in to its structure given the path.
-->
[H: sub_jsonName = arg(0)]
[H: sub_path = arg(1)]
[H: sub_key = arg(2)]
[H: sub_value = arg(3)]

[H: sub_root = eval(sub_jsonName)]
[H: sub_tree = sub_root]
[H: sub_array = json.append("",sub_root)]

<!-- deconstruct items in path for update -->
[H, foreach(sub_limb,sub_path), code: {
   [H: sub_branch = json.get(sub_tree,sub_limb)]
   [H: sub_array = json.append(sub_array,sub_branch)]
   [H: sub_tree = sub_branch]
}]

<!-- add value to last branch -->
[H: sub_tree = json.set(sub_tree,sub_key,sub_value)]

<!-- reconstruct the object -->
[H: sub_pathRev = json.reverse(sub_path)]
[H: sub_array = json.reverse(sub_array)]
[H, foreach(sub_limb,sub_pathRev), code: {
   [H: sub_tree = json.set(json.get(sub_array,roll.count+1),sub_limb,sub_tree)]
}]

<!-- update original object -->
[H: set(sub_jsonName,sub_tree)]

!!
@@ @json.append.path
<!-- json.append.path(jsonName,path,value)
   jsonName - json name to have data added to
   path - array of path to json data
   value - value to insert
   json - original json will be updated using its name for reference.
   
   This will update the original json and insert data in to its structure given the path.
-->
[H: sub_jsonName = arg(0)]
[H: sub_path = arg(1)]
[H: sub_value = arg(2)]

[H: sub_root = eval(sub_jsonName)]
[H: sub_tree = sub_root]
[H: sub_array = json.append("",sub_root)]

<!-- deconstruct items in path for update -->
[H, foreach(sub_limb,sub_path), code: {
   [H: sub_branch = json.get(sub_tree,sub_limb)]
   [H: sub_array = json.append(sub_array,sub_branch)]
   [H: sub_tree = sub_branch]
}]

<!-- add value to last branch -->
[H: sub_tree = json.append(sub_tree,sub_value)]

<!-- reconstruct the object -->
[H: sub_pathRev = json.reverse(sub_path)]
[H: sub_array = json.reverse(sub_array)]
[H, foreach(sub_limb,sub_pathRev), code: {
   [H: sub_tree = json.set(json.get(sub_array,roll.count+1),sub_limb,sub_tree)]
}]

<!-- update original object -->
[H: set(sub_jsonName,sub_tree)]

!!
@@ @json.get.path
<!-- json.get.path(jsonName,path): data
   jsonName - text name of json
   path - json array of path to get
   data - value at the path
-->

[H: sub_jsonName = arg(0)]
[H: sub_path = arg(1)]

[H: sub_tree = eval(sub_jsonName)]

[H, foreach(sub_limb,sub_path): sub_tree = json.get(sub_tree,sub_limb)]

[H: macro.return = sub_tree]
!!

[Merudo]

EDIT: Outdated, check json.path.read instead.

PR #651 adds json.get.path.

Example of use:

[h:troll = json.set("{}", "name", "Troll", "HP", 75, "Attacks", json.append("Claw", "Bite"))]
[h:orc = json.set("{}", "name", "Orc", "HP", 13, "Attacks", json.append("Sword", "Punch"))]
[h:monsters = json.set("{}", "Troll", troll, "Orc", orc)]

[h: path = json.append("Orc", "Attacks", 1))]
[r: json.get.path( monsters, path)]
[r: json.get.path( monsters, "Orc", "Attacks", 1)]

Which both return "Punch".

In case of an error due to invalid array index:

[r: json.get.path( monsters, "Orc", "Attacks", -1)]

returns

Invalid index "-1" at [Orc, Attacks] for array (size of 2) in function "json.get.path"

In case of an error due to invalid field:

[r: json.get.path( monsters, "Orc", "Spells", 0)]

returns

Invalid value for key "Spells" at [Orc] in function "json.get.path".

And, in the case of an error due to the object not being a JSON:

[r: json.get.path( monsters, "Orc", "Attacks", 0, "Damage")]

returns

Invalid path "Damage" at [Orc, Attacks, 0] in function "json.get.path". Object is not a JSON Array or a JSON Object.

[aliasmask]

FYI, typical behavior for a json.get when the field doesn't exist (except for arrays) is it returns "". For example, [H: json.get(monsters,"Goblin")] would return "". There's no good way to test for something not there without full stop on the execution. So, for [r: json.get.path(monsters,"Orc","Spells")] I would prefer a "" return, but if the path has an invalid index then there should be an error. Also, if the path goes beyond an existing path by more than 1 there should be an error. So, all your examples above would be valid, I just want to make sure getting "Spells" would not toss an error.

This way you can get a path that may not always be there and be able to test the value without an error code.

[JamzTheMan]

Why are we not using the standard dot notations? Eg json.get.path(obj, ".Orc.Spells")?

[aliasmask]

I would think that some keys are not dot notation friendly.

[JamzTheMan]

There shouldn't be. Also, there is sort of a standard. (See other libs and CLI like jq) Doing it this way is like reinventing regular expressions.

If I already know how to do JSON path notation in other apps, why do it special in MT?

See https://restfulapi.net/json-jsonpath/ for examples.

[JamzTheMan]

FWIW I would rather include the jayway lib and change the signature so it's just json.get.path(json obj, string path)

It would also be more powerful and less cumbersome. You can walk the tree, mix arrays and objects etc. AFAIK this is the standard lib as it were.

Plus, documentation is easier, you can just link to external sites like https://www.baeldung.com/guide-to-jayway-jsonpath :)

[Azhrei]

I agree that standards are great. Everybody should have one.

But I’m with AM on this one. Imagine you have a three-level object and want to iterate over all elements of the middle level but keep the first and last level constant. Much simpler if they’re separate parameters. And besides, string concatenation is faster in Java — Forcing MTscript to do it is a loss. 😐

And there would be no way to specify a literal dot within a property using a single string path (without reverting to backslashes and then treating it special anyway because I’ll bet the library API doesn’t account for that).

[JamzTheMan]

It's bad practice to use . in json key but if you do, in JQ you quote the key out.

I'm not sure without seeing your json and what you are trying get but "$..author" would get all the authors from store.book (3 levels deep)

Try looking at the examples/docs then present a case that can not be done using the lib but could be done via this implementation.

In fact, in this implementation it looks like I have to know the key names for all three levels?

Also, does this handle objects and arrays? Can I do weapons[].damage or weapons[2].damage?

Better yet, can I get all weapon damages that have a child type == piercing like:

weapons.[?(@.type == "piercing")].damage

[Merudo]

I think writing a wrapper for jayway would be immensely useful and would be relatively easy. I was thinking the call could look like

jsonpath(json, path, tagValue)

As proposed, the functions json.get.path and json.set.path are replacements for the UDFs json.pget and json.pset that a lot of folks already use.

EDIT: Another problem with jayway is that it might not be compatible with the json library we are using.

[JamzTheMan]

I suppose I could possibly get behind that (although it does proliferate macro functions even further, ug, so many functions lol)

What's the tagValue, for sets? (and prefer to keep the . notation since we already have every other json function like that)

json.path(json, path, [tagValue])

Where if tagValue is passed sets the value for the path? Both return the result set?

Looks like if it's a single value it returns a string, otherwise it returns either an json array or object. (not sure what you want to do for this implementation?)

Also, I suppose this should now go to it's own issue for YAF.

[Merudo]

I don't think jayway is compatible with the json library we are using ( json-lib aka net.sf.json), so that would be an issue.

Should we switch to a different json implementation? The one we are using has not been updated for almost a decade. Maybe Gson or Jackson would work well?

[JamzTheMan]

I think gson is but in the Gradle? It has some nice features.

By not compatible, how does it have to interact with the current libs? Take a string as json and output a JSON as a string?

But otherwise I have no issue updating our json libs, we have several libs that need to be updated. Sadly we don't have coverage to test everything after such changes though.

[Merudo]

I guess we could use both gson and the old json-lib for now. So we could use gson for the jsonPath stuff and json-lib for the rest.

That might end up being confusing though.

[Merudo]

New PR #657 offers json.path.read, which I believe to be a better alternative to my proposed json.get.path.

Now if we have

[h:troll = json.set("{}", "name", "Troll", "HP", 75, "Attacks", json.append("Claw", "Bite"))]
[h:orc = json.set("{}", "name", "Orc", "HP", 13, "Attacks", json.append("Sword", "Punch"))]
[h:monsters = json.set("{}", "Troll", troll, "Orc", orc)]

we can easily get a nested field, for example

[json.path.read(monsters, "Orc.Attacks.[1]")]

to returns Punch.

Additionally, if we want an array containing the attacks of each monster, we could type

[json.path.read(monsters, ".Attacks")]

which returns [["Claw","Bite"],["Sword","Punch"]]

Inline filters are also supported, so that if we want the name of the monsters with > 30 HPs, we can type

[json.path.read(monsters, ".[?(@.HP > 30)].name")]

which returns ["Troll"].

Errors

Jayway returns error messages in English which I use for the exceptions. So, if I type

[json.path.read(monsters, "Orc.Spells.[0]")]

I get the error

Error with function "json.path.read": Missing property in path $['Orc']['Spells']

Main issue is that the jayway error messages are not localized.

More information

Information on how to specify the path parameter can be found here.

[Merudo]

The PR now adds three functions: json.path.add, json.path.put and json.path.set.

To add an element to an array at a path, use json.path.add. For example, to add an attack to our Orc:

[json.path.add(monsters, "Orc.Attacks", "Bow")]

To replace a value in an array, for example to replace "Sword" by "Axe", we need to use json.path.set instead:

[json.path.set(monsters, "Orc.Attacks.[0]", "Axe")]

And, to add or replace a value in an object, we need to use json.path.put,

[json.path.put(monsters, "Orc", "AC", 12)]

Together, the three function call would return

{"Troll":{"name":"Troll","HP":75,"Attacks":["Claw","Bite"]},"Orc":{"name":"Orc","HP":13,"Attacks":["Axe","Punch","Bow"],"AC":12}}

EDIT: Also added a json.path.delete, to delete a jsonObject field or a jsonArray element. Example:

[json.path.delete(monsters, "Troll.Attacks.[0]")]

and

[json.path.delete(monsters, "Troll.Attacks[?(@ == 'Claw')]")]

can each delete the "Claw" attack of the Troll.

[Merudo]

Should put and set be combined in a single function?

[Merudo]

[r: test = json.path.put("{}", "$", "new-key", "new-val")]

@JamzTheMan change the sign $ to @:

[r: test = json.path.put("{}", "@", "new-key", "new-val")]

[JamzTheMan]

Ok, ya, did that but now how do you add an object to an object?

So this fails because it's escaping and putting extra quotes in it looks like:

[r: summary = json.path.put("{}", "@", "Summary", "A mechanical race")]
<br>
[r: races = json.path.put("{}", "@", "Android", summary)]

{"Summary":"A mechanical race"}
{"Android":"{\"Summary\":\"A mechanical race\"}"}

[r: test = json.path.read(races, "Android.Summary")]
Errors with

Error with function "json.path.read": Expected to find an object with property ['Summary'] in path $['Android'] but found 'java.lang.String'. This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.

Where as this works ok:

[r: test = json.path.read('{"Android": {"Summary": "A mechanical race"} }', "Android.Summary")]

A mechanical race

I can't seem to create this easy object: {"Android": {"Summary": "A mechanical race"} } using .put. Ideally creating it in one put statement would be great but even in 2 steps it fails. I'm trying to google to see if we can even do it in one step but haven't found anything yet...

[Merudo]

I probably just need to add a conversion to json if the object in put is a json string.

[JamzTheMan]

@Merudo Thanks! That seemed to do the trick. At least for the escaping of quotes.

Do you know if it's possible to create a nested object is possible, like:
[r: races = json.path.put("{}", "@", "Android.Summary", "a mechanical race")]

[Phergus]

@JamzTheMan If you were expecting `{"Android":{"Summary":"a mechanical race"}} then the answer is no.

[Phergus]

Wiki updated by Merudo for json.path.add, .delete, .put and .set functions.

[Phergus]

@Merudo
For this data:

{
  "$id": "10639",
  "editionNumber": "5e",
  "language": "de-de",
  "siNnerMetaData":   {
    "$id": "10640",
    "id": "009f0da8",
    "visibility":     {
      "$id": "10641",
      "id": null,
      "isPublic": false,
      "isGroupVisible": false,
      "userRights": []
    }
  }
}

Expected this to work:
[r: vis = json.path.read(data,"$.siNnerMetaData.visibility")]

But get this:
java.lang.IllegalArgumentException: Illegal group reference error executing expression vis = json.path.read(test,"$.siNnerMetaData.visibility")

The path works on the Jayway Json Path Evalutor

And it works if I take out the "$.".

Have to escape the dollar sign.
[r: vis = json.path.read(data,"\$.siNnerMetaData.visibility")]

That should probably be in the wiki since it points to the Jayway docs which states:

$ - The root element to query. This starts all path expressions.

[Merudo]

Here is the code to create @Phergus's json if anyone wants to try it out:

[h: visibility = json.set("{}", "id", "null", "isPublic", "false", "isGroupVisibile", "false", "userRights", "[]")]
[h: siNnerMetaData = json.set("{}", "\$id", "10640","id", "009f0da8","visibility", visibility)]
[h: data = json.set("{}", "\$id", "10639", "editionNumber", "5e", "language", "de-de", "siNnerMetaData", siNnerMetaData)]

[Phergus]

I just cheat and paste it into a token notes field and then read it from there. Cuz I'm just testing stuff.

[h: data = getNotes("Chummer")]
[r: vis = json.path.read(data,"siNnerMetaData.visibility")]

[rkathey]

Is there a need for the $ in the id field? Does it serve a purpose in the json.path calls?

[Phergus]

That's just how the data was when I got it from the Chummer guys. This is actually a tiny snippet of a much large character data collection.

[Phergus]

@Merudo Is "$." automatically being prepended to the path internally?

[Merudo]

@Merudo Is "$." automatically being prepended to the path internally?

No, it is not, but it could be.

Is there any reason not to start the path with $?

[Phergus]

It acts like it is. The online evaluator will take this $..visibility and return the same result as MapTool does with [r: vis = json.path.read(data,".visibility")].

So far any example I find online that starts with "$.", I just take those off and use the remainder in MapTool for the json.path.read() command.

[Phergus]

Just testing the new functions with examples of normal JSON paths functionality from the JsonPath docs to verify our implementaton is working as expected.

[rkathey]

I'm testing as well to see how versatile it is. I did find one thing that diverges from the jaway docs.

[h: j='{ "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99 }, { "category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99 }, { "category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99 } ], "bicycle": { "color": "red", "price": 19.95 } }, "expensive": 10 }'] [r:q=".title"] [r:json.path.read(j,q)]<br> [r:q=".author"] [r:json.path.read(j,q)]<br> [r:q=".bicycle"] [r:json.path.read(j,q)]<br> [r:q=".color"] [r:json.path.read(j,q)]<br> [r:q=".store.book[*].author"] [r:json.path.read(j,q)]<br> [r:q=".*.book[2]"] [r:json.path.read(j,q)]<br> [r:q="..book[2]"] [r:json.path.read(j,q)]<br>

The last one fails. It works if you add the wild card like the statement above.
See https://github.com/json-path/JsonPath/blob/master/README.md for additional queries.

[Phergus]

Yeah. json.path.read doesn't like the deep scan syntax (".."). If you just leave off the first period it will work. See my comments two posts prior.

[Phergus]

Basic testing of new functions completed.

• json.path.add
• json.path.set
• json.path.put
• json.path.read
• json.path.delete

Closing.

[Phergus]

@Merudo What happened to the idea of either using the DEFAULT_PATH_LEAF_TO_NULL configuration or having a 3rd parameter for json.path.read to set the desired behavior?

See comment above.

Never mind. I see that was PR #871 that got deferred.