Add JsonPath support to JsonDocument/JsonElement

[gary-holland]

Hi,

I'd like to request JsonPath support for querying the JsonDocument/JsonElement classes. JsonPath provides similar capability to XPath (and even Sql) in that it allows queries to be performed against Json documents. This currently represents a major gap for us shifting from Newtonsoft to system.text.json, as we provide JsonPath values as an input parameter to a data load process which can't be worked around via code.

The JsonPath syntax is described here.

The equivalent functionality in the Newtonsoft library is:

var jsonPath = "$.my.path";
var json = JToken.Parse(jsonString);
var token = json.SelectToken(jsonPath);

The following proposed syntax would work well in the JsonDocument structure:

var jsonPath = "$.my.path";
var jsonDoc = JsonDocument.Parse(json);
var element = jsonDoc.SelectElement(jsonPath); //returns JsonElement
var elements = jsonDoc.SelectElements(jsonPath); //returns JsonElement.ArrayEnumerator

Thanks.

[dazinator]

Just searched for a similar thing myself. The appropriate newtonsoft documentation for this: https://www.newtonsoft.com/json/help/html/QueryJsonSelectTokenJsonPath.htm

[dkmiller]

Seconding this. It would be extremely useful.

[NickMSW]

Would also like to push this up the stack. We want to use system.text.json but the lack of JsonPath is blocking us.

[azambrano]

In the meantime. I'm doing some experiment following the same strategy of the Json.net for support JsonPath but using System.Text.Json.JsonDocument https://github.com/azambrano/JsonDocumentPath

[NinoFloris]

@layomia did this and writedom move out of scope for 5.0?

[zmhh]

If this is implemented it would be nice to have the option for case insensitivity.

[Laiteux]

We want this

[onionhammer]

This would be very useful

[blushingpenguin]

I've ported the Newtonsoft.Json implementation to work with JsonDocument, along with the tests.

nuget: https://www.nuget.org/packages/BlushingPenguin.JsonPath/
source: https://github.com/blushingpenguin/BlushingPenguin.JsonPath/

[Laiteux]

I've ported the Newtonsoft.Json implementation to work with JsonDocument, along with the tests.

nuget: https://www.nuget.org/packages/BlushingPenguin.JsonPath/
source: https://github.com/blushingpenguin/BlushingPenguin.JsonPath/

Hello, what's the difference with this? https://github.com/azambrano/JsonDocumentPath

[blushingpenguin]

It's pretty similar (it's also a port of Newtonsoft.Json's implementation), but is missing some of the functionality of the original and lacks a nuget package. (AFAIK I've ported all of the original functionality).

I could have forked that one, but my version is actually a port of https://github.com/blushingpenguin/MongoDB.Bson.Path (which is a version that works on the MongoDB.BsonDocument family).

[gregsdennis]

More support for JSON Path

[GF-Huang]

Does it merged into .NET 5?

[gregsdennis]

@ADustyOldMuffin, what's with the downvote without an explanation?

[ZukkyBaig]

Is this still being looked at? Without JSON path it is a hindrance.

[layomia]

This feature is proposed for .NET 6, but not committed. To be clear, we acknowledge that this is an important feature for many users, however, work on features with higher priority may prevent this from coming in .NET 6.

[gregsdennis]

.Net and any other implementors should be aware that there is currently an effort to standardize JSON Path. It would be a good idea to follow that progress and perhaps join the effort. More people pushing it forward could help it go faster.

[jaldinger]

I agree this is highly necessary. I also believe it should follow the proposed standard as closely as possible.

[steveharter]

Note that there is now the System.Text.Json.Nodes.Node APIs that supports JsonPath via GetPath() and also support case-insensitivity for property names as was requested above.

If you already have an instance of a JsonElement, there is interop with JsonDocument`JsonElement` via static factory methods on the JsonNode-derived classes.

Adding support for JsonPath to JsonDocument`JsonElement` will be decrease performance since the design today is based on a low-level "metadata database" design which is optimized to reduce memory usage and deferred value creation, and not directly extensible to add a "parent" and "property name" semantics required for JsonPath.

[gregsdennis]

@steveharter can you provide links, please? On which versions of .net is this available?

Also what support for JSON Path is there (given that there is no standard)? Or is it just JSON-Path-like?

[WeihanLi]

Great idea, and maybe better if implemented for JsonNode

[IanKemp]

Assuming this isn't going to make .NET 8 either?

[jeffhandley]

Assuming this isn't going to make .NET 8 either?

You're correct, @IanKemp. This will be evaluated again during our .NET 9 planning.

[frankhaugen]

Assuming this isn't going to make .NET 8 either?

You're correct, @IanKemp. This will be evaluated again during our .NET 9 planning.

When is it scheduled on the "docket" for planning? (When will we have a decision)

[gregsdennis]

The JSON Path specification has been released!

https://www.rfc-editor.org/rfc/rfc9535.html

Again, JsonPath.Net fully supports the specification. The library has been bumped to v1.0.0 with the release of the spec.

[peteraritchie]

Related: https://josef.codes/some-basic-query-support-for-system-text-json-jsonpath-inspired/

[frankhaugen]

The JSON Path specification has been released!

https://www.rfc-editor.org/rfc/rfc9535.html

Again, JsonPath.Net fully supports the specification. The library has been bumped to v1.0.0 with the release of the spec.

That is awesome and for my personal stuff this is great, but professionally, I might be limited by corporate policy to use 1st party (Microsoft), or 2nd party (.net foundation membered), or "verified" 3rd party, (Newtonsoft), libraries.

I say corporate, but mostly those policies come from auditing agencies for things like SOC2 and ISO2700 -certifications. So for many its not an option to use your library as you don't provide support, personal/individual ownership of the library, and so its plausibly dangerous to use your library seen from the perspective of the corporate laywers and C-level management. Explaining that your lib is deserving of exception when others exist that might be less good but by other standards are "safer" on paper, is a lot of work.

That's why many of us are begging MS to add functionality like this, as we either have to write it ourselves, or use some 3 year out-of-date stuff, that tripple memory use, (like newtonsoft), to get some functionality that frankly should have been there from the start in the runtime when they started on JSON

[danielaparker]

That's why many of us are begging MS to add functionality like this, as we either have to write it ourselves, or use some 3 year out-of-date stuff, that tripple memory use, (like newtonsoft), to get some functionality that frankly should have been there from the start in the runtime when they started on JSON

Is JSONPath really necessary for querying JSONElement instances? Can't LINQ serve that purpose?

[gregsdennis]

2nd party (.net foundation membered)

I've been looking for reasons to submit my json-everything project to .net foundation. This is a good one.

Still, I think it's the role of the developer to argued that well-established 3rd party libs are fine, even if it's on a case-by-case basis.

[gregsdennis]

as you don't provide support

What makes you think this?

I certainly do provide support. Issues don't stay open long, and I usually respond within 12 hours (depending on whether I'm sleeping). I also have a dedicated Slack workspace that's open for all.

I don't offer a paid support "tier" because I treat every issue this way. I'm employed by Postman specifically to work on JSON Schema (the spec and community) and this suite of libraries.

But if you have something else in mind that will result in lining my pockets, I'm open to ideas.

[gregsdennis]

functionality that frankly should have been there from the start in the runtime when they started on JSON

I expect that by posting here you understand that software is iterative.

JSON Path/Pointer/Schema/Patch/etc. are extensions to JSON. The primary functionality is data modeling and serialization, which is exactly what has been provided, and it's why this issue has been pushed back.

Basics first.

[LeaFrock]

So for many its not an option to use your library as you don't provide support, personal/individual ownership of the library, and so its plausibly dangerous to use your library seen from the perspective of the corporate laywers and C-level management.

The open-source packages from a person/enterprise are not different when talking about 'dangerous'. Otherwise what's the meaning of open-source? Considering the risk of EOT, even MS abandons a lot of projects too. And what's the next then? Keep begging MS to give you an exception? The corporate policy should give a standard to 'verify' 3rd libraries, and that's what MS hope to promote within .NET ecosystem too.

many of us are begging MS to add functionality like this

Though I hope the runtime supports JsonPath, I also support 3rd community libraries while the runtime team has the right of saying NO. It's been long time since the .NET Framework time which developers begging MS to release ’everything‘. I really hope the .NET community grows up to achieve a balance which .NET developers rely on MS only to a limited extent. So, encourage, embrace, and engage community open-source projects.

I stand with @gregsdennis. Appreciate a lot for your work!

P.S. a similar case occurs in this discussion CSV support in .NET Core.

[ay-azara]

Instead of a complex query mechanism could we get a JSON flatten function so we can deserialize to a Dictionary<string, string> and perform the "lookup" on the key?

{
    "foo": {
        "bar": "baz"
    }
}

{
    "foo.bar":  "baz"
}

// Something like
var dict = json.Deserialize<Dictionary<string, string>>(JsonSerializer.Flatten(json))
var val = dict['foo.bar']

[peteraritchie]

var val = dict['foo.bar']

That's getting really close to being a JSONPath expression, there's just an implied $. at the start of foo.bar. With the notation you're proposing, how would you support collections and arrays? i.e. one way to support that is to use JSONPath notation. :)

[gregsdennis]

If you want a single value, you don't want JSON Path. You want JSON Pointer: /foo/bar. I have an implementation of that, too.

[ay-azara]

var val = dict['foo.bar']

That's getting really close to being a JSONPath expression, there's just an implied $. at the start of foo.bar. With the notation you're proposing, how would you support collections and arrays? i.e. one way to support that is to use JSONPath notation. :)

It would be the index of the array/collection object. foo.bar.0.baz

Just to clarify, when I said "instead" it was meant as a short term compromise, not "you all should give up on JSON Path support". From my perspective, the issue has been open since 2019 so I can only assume that implementing JSON Path is a bigger ask than a flattening function. I'm willing to settle for a less rigorous but still workable solution in the short term rather than a perfect solution that may never get implemented. It's fine if you don't want to settle, my use case is not as stringent as yours probably is.

And I think Greg's json-everything is wonderful but for some orgs it's easier to use a library that has already been cleared for use.

Based on the flurry of downvotes I see it's not something worth pushing further.

[gregsdennis]

@ay-azara to get you by, here's an extension that builds an index keyed by JSON Pointers:

public static Dictionary<string, JsonNode?> Index(this JsonNode? node)
{
    var index = new Dictionary<string, JsonNode?>();
    var search = new Queue<(string Pointer, JsonNode? Value)>();
    search.Enqueue((string.Empty, node));

    while (search.Any())
    {
        var current = search.Dequeue();
        index[current.Pointer] = current.Value;
        switch (current.Value)
        {
            case JsonObject obj:
                index[current.Pointer] = obj;
                foreach (var kvp in obj)
                {
                    search.Enqueue(($"{current.Pointer}/{Encode(kvp.Key)}", kvp.Value));
                }
                break;
            case JsonArray arr:
                index[current.Pointer] = arr;
                for (var i = 0; i < arr.Count; i++)
                {
                    var value = arr[i];
                    search.Enqueue(($"{current.Pointer}/{i}", value));
                }

                break;
        }
    }

    return index;
}

private static string Encode(string value)
{
    if (value.All(c => c is not ('~' or '/'))) return value;

    var builder = new StringBuilder();
    foreach (var ch in value)
    {
        switch (ch)
        {
            case '~':
                builder.Append("~0");
                break;
            case '/':
                builder.Append("~1");
                break;
            default:
                builder.Append(ch);
                break;
        }
    }

    return builder.ToString();
}

Again, pointers are ideal since each entry only identifies a single location.

[ay-azara]

@gregsdennis Thanks man, appreciate you taking the time :)

[isaevdan]

Any updates when this will be available? The issue is from 2019
BTW Newtonsoft.Json has bugs in their implementation

[gregsdennis]

@isaevdan as mentioned in a comment above, you can use JsonPath.Net if you need something now.

[isaevdan]

@isaevdan as mentioned in a comment above, you can use JsonPath.Net if you need something now.

Yep, thanks and appreciate you work
Just in progress in migrating to JsonPath.NET, but still weird it's not part of library :)

[isaevdan]

Created JsonExtensions - wrapper with couple of same methods as from NewtonsoftJson but with provided libraries, may be a good starting point for others who are going to do migration

public static class JsonExtensions
{
    public static JsonNode SelectToken(this JsonNode node, string path,
        PathEvaluationOptions options = null)
    {
        var jsonPath = JsonPath.Parse(path);
        var matches = jsonPath.Evaluate(node, options).Matches;
        if (matches.Count > 1)
            throw new JsonException("Path returned multiple tokens.");
        return matches.FirstOrDefault()?.Value;
    }

    public static List<JsonNode> SelectTokens(this JsonNode node, string path,
        PathEvaluationOptions options = null)
    {
        var jsonPath = JsonPath.Parse(path);
        return
            jsonPath.Evaluate(node, options)
                .Matches
                .Select(m => m.Value)
                .ToList();
    }

    public static IList<JsonNode> Children(this JsonNode node)
    {
        return node switch
        {
            JsonArray array => array.ToList(),
            JsonObject obj => obj.Select(e => e.Value).ToList(),
            _ => []
        };
    }

    public static JsonNode? ParseJsonNode(string text)
    {
        // Handle empty or null input text
        if (string.IsNullOrWhiteSpace(text))
        {
            return null;
        }

        try
        {
            return JsonNode.Parse(text);
        }
        catch (JsonException)
        {
            return null;
        }
    }
}