Root Reference

[springcomp]

Root Reference

JEP	17
Author	Maxime Labelle
Status	draft
Created	25-02-2022

Abstract

This document proposes grammar modifications to JMESPath
to support referring to the original input JSON document
inside an expression.

Motivation

As a JMESPath expression is being evaluated, the current scope changes.
Given a simple sub expression such as foo.bar, first the foo
expression is evaluated with the starting input JSON document, and the
result of that expression is then used as the current scope when the
bar element is evaluated.

Once we’ve drilled down to a specific scope, there is no way, in the
context of the currently evaluated expression, to refer to any
elements outside of that element.

A common request when querying JSON objects is the ability to refer to
the input JSON document.

For example, suppose we had this data:

{
  "first_choice": "WA",
  "states": [
     {"name": "WA", "cities": ["Seattle", "Bellevue", "Olympia"]},
     {"name": "CA", "cities": ["Los Angeles", "San Francisco"]},
     {"name": "NY", "cities": ["New York City", "Albany"]}
 ]
}

Let’s say we wanted to get the list of cities of the state corresponding
to our first_choice key. We’ll make the assumption that the state
names are unique in the states list. This is currently not possible
with JMESPath. In this example we can hard code the state "WA":

states[?name==`"WA"`].cities

but it is not possible to base this on a value of first_choice, which comes from the parent element.
This JEP proposes a solution that makes this possible in JMESPath.

Specification

The grammar will support a new token $ that refers to the root of the original input JSON document.

The $ token is inspired by the JSONPath specification which has a token with the same name.

The $ token is also inspired by the XPath specification, where the / token designates the root of the original XML document.

This JEP introduces the following productions:

root-node     = "$"

The expression production will also be updated like so:

expression    =/ root-node

Motivating Example

With these changes defined, the expression in the “Motivation” section can be be written as:

states[?name==$.first_choice].cities[]

Which evalutes to ["Seattle", "Bellevue", "Olympia"].

Rationale

This JEP standardizes a common request when querying JSON document as seen in existing library implementations.

Some alternatives to this JEP are considered. Most notably, the Lexical Scoping proposal introduces a let() function that is a more general an flexible way to achieve the desired result. For instance, with the let() function, the expression in the “Motivation” section can be written as:

let({root: @}, &states[?name==root.first_choice].cities[])

Tradeoff must be considered while evaluating this JEP if Lexical Scoping were accepted.

[innovate-invent]

I'd like to co-opt this proposal for the purpose of the map and reduce projections.The $ symbol would be more context dependant.

It would refer to the root node outside of a projection, but refer to the key or index of the current value being iterated over within a mapping projection. Additionally, within the reduce projection being proposed, $ would refer to the initial/last/accumulated value returned during the projection.

The following would effectively replace the keys() function:

search({"a":1,"b":2}, *.$) -> ["a", "b"]

This would be functionally identical to the proposed items() function:

search({"a":1,"b":2}, *.[$,@]) -> [["a", 1], ["b", 2]]

Nested projections would redefine $ at each level:

search({"a":["x","y","z"],"b":["c": 2]}, *.[$, *.$]) -> [["a", [0, 1, 2]], ["b", [0]]]

[springcomp]

I understand the two last examples, but not the first.
In addition, does the motivating example still works with your alternate proposal ?

[innovate-invent]

for the first example * projects the remaining expression over the values of the input object. Within that projection the key for the current value is represented by $ and the value is represented by @ (which is then discarded). This means that *.$ returns an array of the keys vs *.@ which would return an array of the values.

The example in the original proposal above will continue to work given that $ is not being used within a projection.
Otherwise, if a reference to the root is required within a projection, then the let() function would be used to provide that reference.

[innovate-invent]

I just thought of another use for the $ symbol. It could refer to the defining context within an expression-type. This could be useful for functions like sort_by() or more importantly reusable expressions.

[innovate-invent]

this might also be more appropriately described as the ancestor-node, with a consistent paradigm that it refers to an ancestor relevant to the current context. In the general context it is the root node, for a mapping the key is the ancestor of the value, reduce has the elements as child components of the accumulated values, expression-type is a child of the defining context.

[innovate-invent]

OK! So after some discussion with @springcomp in Gitter, he wanted a more technical rationalization of how my proposed use of $ is consistent in paradigm across the general context, mappings, reduce, and expression-types.

The intent is to provide a token that is a useful reference to a 'greater' context within the different evaluation modes (general, projection, deferred (expression-type) ). It is always a 'greater' or ancestor context rather than pointing to something adjacent or a forward reference. This is so that users, when reading a path, encounter a $ they know to read left until the first projection/function encountered to know the context/mode defining it. This of course is a very non-technical way of describing it so I prepared some drawings to hopefully illustrate how this behavior is consistent from a technical standpoint.

Not sure if this is clear, but all grey boxes are nodes regardless of shape

The mapping transform is hopefully the most obvious.
There are two graphs in play here, the input data structure, and the transformed graph.
A object is a structure with keys as parent nodes to the value nodes, similarly an array is keys of the indices as parent nodes to the value nodes. (root in the diagram is the root of the object/array)
An expression results in a new node in the transformed graph.
The keys in the transformed graph are created by the transform (T), while the value nodes are the result of evaluating the expression (E).
hash-wildcard-expression or list-wildcard-expression being the transform (T) in this context, while the subexpression is the expression E

Reduce has no concept of keys to map to and so the 'parent' is defined a bit differently:

The root reference is straight forward, and similar to reduce() points at the most relevant ancestor:

You might want to argue that $ should point to the previous expression result node, but that is already pointed at by @.

I haven't finished the expression-type diagram as it is a bit more complicated as it exists in a different value space, but I wanted to get these posted before the end of today. Please let me know if there is anything unclear, I rushed this a bit.

[springcomp]

@hell-racer I noticed you had an interest in this JEP.

As the JEP-011 Lexical Scoping proposal is being included in the next milestone, I’m thinking out loud whether we could merge this proposal with JEP-011? As a separate proposal, this feels redundant.

What do you think?