Results do not match other implementations

[cburgmer]

The following queries provide results that do not match those of other implementations of JSONPath
(compare https://cburgmer.github.io/json-path-comparison/):

• $[1:3]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["second", "third"]
  

  Actual output:

  [
    "second",
    "third",
    "forth"
  ]
  

• $[0:5]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["first", "second", "third", "forth", "fifth"]
  

  Actual output:
  NOT_FOUND

• $[1:10]
  Input:

  [
    "first",
    "second",
    "third"
  ]
  

  Expected output:

  ["second", "third"]
  

  Actual output:
  NOT_FOUND

• $[-4:-4]
  Input:

  [
    2,
    "a",
    4,
    5,
    100,
    "nice"
  ]
  

  Expected output:

  []
  

  Actual output:

  [
    4
  ]
  

• $[-4:-3]
  Input:

  [
    2,
    "a",
    4,
    5,
    100,
    "nice"
  ]
  

  Expected output:

  [4]
  

  Actual output:

  [
    4,
    5
  ]
  

• $[-4:2]
  Input:

  [
    2,
    "a",
    4,
    5,
    100,
    "nice"
  ]
  

  Expected output:

  []
  

  Actual output:

  [
    4
  ]
  

• $[-4:3]
  Input:

  [
    2,
    "a",
    4,
    5,
    100,
    "nice"
  ]
  

  Expected output:

  [4]
  

  Actual output:

  [
    4,
    5
  ]
  

• $[:2]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["first", "second"]
  

  Actual output:

  [
    "first",
    "second",
    "third"
  ]
  

• $[-4:]
  Input:

  [
    "first",
    "second",
    "third"
  ]
  

  Expected output:

  ["first", "second", "third"]
  

  Actual output:
  NOT_FOUND

• $[0:3:1]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["first", "second", "third"]
  

  Actual output:

  [
    "first",
    "second",
    "third",
    "forth"
  ]
  

• $[0:4:2]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["first", "third"]
  

  Actual output:

  [
    "first",
    "third",
    "fifth"
  ]
  

• $[]
  Input:

  {
    "": 42,
    "''": 123,
    "\"\"": 222
  }
  

  Expected output:

  NOT_SUPPORTED
  

  Actual output:

  [
    {
      "": 42,
      "\"\"": 222,
      "''": 123
    }
  ]
  

• $..[1].key
  Input:

  {
    "k": [
      {
        "key": "some value"
      },
      {
        "key": 42
      }
    ],
    "kk": [
      [
        {
          "key": 100
        },
        {
          "key": 200
        },
        {
          "key": 300
        }
      ],
      [
        {
          "key": 400
        },
        {
          "key": 500
        },
        {
          "key": 600
        }
      ]
    ],
    "key": [
      0,
      1
    ]
  }
  

  Expected output (in any order as no consensus on ordering exists):

  [200, 42, 500]
  

  Actual output:

  [
    42
  ]
  

• $..key
  Input:

  {
    "object": {
      "key": "value",
      "array": [
        {
          "key": "something"
        },
        {
          "key": {
            "key": "russian dolls"
          }
        }
      ]
    },
    "key": "top"
  }
  

  Expected output (in any order as no consensus on ordering exists):

  ["russian dolls", "something", "top", "value", {"key": "russian dolls"}]
  

  Actual output:

  [
    "something",
    "top",
    "value",
    {
      "key": "russian dolls"
    }
  ]
  

• $.store..price
  Input:

  {
    "store": {
      "book": [
        {
          "category": "reference",
          "author": "Nigel Rees",
          "title": "Sayings of the Century",
          "price": 8.95
        },
        {
          "category": "fiction",
          "author": "Evelyn Waugh",
          "title": "Sword of Honour",
          "price": 12.99
        },
        {
          "category": "fiction",
          "author": "Herman Melville",
          "title": "Moby Dick",
          "isbn": "0-553-21311-3",
          "price": 8.99
        },
        {
          "category": "fiction",
          "author": "J. R. R. Tolkien",
          "title": "The Lord of the Rings",
          "isbn": "0-395-19395-8",
          "price": 22.99
        }
      ],
      "bicycle": {
        "color": "red",
        "price": 19.95
      }
    }
  }
  

  Expected output (in any order as no consensus on ordering exists):

  [12.99, 19.95, 22.99, 8.95, 8.99]
  

  Actual output:

  [
    19.95
  ]
  

• $[?(1==1)]
  Input:

  [
    1,
    3,
    "nice",
    true,
    null,
    false,
    {},
    [],
    -1,
    0,
    ""
  ]
  

  Error:

  unexpected character ']' at 1:40
  

For reference, the output was generated by the program in https://github.com/cburgmer/json-path-comparison/tree/master/implementations/Golang_github.com-ohler55-ojg.

[ohler55]

Thanks. I'll update Oj to more closely follow the consensus.

As a side note, on Safari the header labels on your table all blended together making it unreadable for the golang entries.

[cburgmer]

You might also be interested in the JSONPath draft development (https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-jsonpath) and the slack channel to drive a common interpretation of JSONPath (https://join.slack.com/t/jsonpath-standard/shared_invite/zt-es623y9z-~JOiqJTylBXSttT5n_xaPQ).

[ohler55]

Thanks

[ohler55]

I wasn't successful running the commands outlined in the README.md but I'm sure I can get something figured out with the selector and document files. Is there any way to get a listing of the consensus results without running all the tests?

[cburgmer]

I wasn't successful running the commands outlined in the README.md

Currently the Docker build seems broken, I'll push a fix. I mostly run natively on OSX, so the Docker build is a bit behind, sadly.

Is there any way to get a listing of the consensus results without running all the tests?

Yes, and using this is definitely encouraged: https://github.com/cburgmer/json-path-comparison/blob/master/regression_suite/regression_suite.yaml

This gets updated manually every few weeks when the versions of dependencies are bumped.

[ohler55]

I am using OSX as well so that would help.

I think I can drive a test with just the consensus file so that is great. Thanks.

[ohler55]

The consensus branch has the fixes in it. Is it possible to recheck with that branch?

[cburgmer]

This one is still failing locally:

• $[?(1==1)]
  Input:

  [
    1,
    3,
    "nice",
    true,
    null,
    false,
    {},
    [],
    -1,
    0,
    ""
  ]
  

  Error:

  unexpected character ']' at 1:40
  

There are further selectors I have omitted from the initial issue, as they are not wrong by the consensus, just unsupported. I'm posting them now for completeness:

• $[::]
  Input:

  [
    "first",
    "second"
  ]
  

  Expected output:

  ["first", "second"]
  

  Actual output:
  NOT_SUPPORTED

• $[010:024:010]
  Input:

  [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7,
    8,
    9,
    10,
    11,
    12,
    13,
    14,
    15,
    16,
    17,
    18,
    19,
    20,
    21,
    22,
    23,
    24,
    25
  ]
  

  Expected output:

  [10, 20]
  

  Actual output:
  NOT_SUPPORTED

• $[1:3:]
  Input:

  [
    "first",
    "second",
    "third",
    "forth",
    "fifth"
  ]
  

  Expected output:

  ["second", "third"]
  

  Actual output:
  NOT_SUPPORTED

• $.key-dash
  Input:

  {
    "key-dash": "value"
  }
  

  Expected output:

  ["value"]
  

  Actual output:
  NOT_SUPPORTED

[ohler55]

Odd that $[?(1==1)] fails for you. I even have a unit test for that.

I'll look at the others as well. Thanks for all the test cases.

[ohler55]

Support for $.key-dash introduces other issues. Consider $[?(@.key > @key-1)]. Is that a comparison to a name of "key-1" or the value of key minus 1?

[cburgmer]

As for $[?(@.key > @key-1)] we discussed this a while ago in cburgmer/json-path-comparison#42.

[ohler55]

It doesn't look like there was a final agreement. Personally I prefer to support simple operations in scripts over allowing - in keys especially wince block notation still allows paths that get to keys that include a -. I'll go with not following the consensus in favour of supporting operations in scripts for OjG.

That leaves the $[?(1==1)] and why it is failing in your environment and not mine.

[ohler55]

I am still not able to run the tests by following the README.md instructions but I did notice that the Golang targets use go v1.13 which is 2 minor versions out of date and more than a year old. The current version is v1.15.

[cburgmer]

It seems the error depends on the JSON payload to surface:

$ echo '[1, 3, "nice", true, null, false, {}, [], -1, 0, ""]' | ./implementations/Golang_github.com-ohler55-ojg/run.sh '$[?(1==1)]'
unexpected character ']' at 1:40

However

$ echo '[1, 3, "nice", true, null, false, {}, -1, 0, ""]' | ./implementations/Golang_github.com-ohler55-ojg/run.sh '$[?(1==1)]'
[1,3,"nice",true,null,false,{},-1,0,""]

[ohler55]

Thanks. That should help. I'll run that way to check as well.

[ohler55]

I think the latest in the consensus branch should pass all but the key with a dash in it but I don't plan on changing that one.

[ohler55]

Anyway, let me know if it looks good and I'll make a release.

[cburgmer]

Looks good to me!

I suggest you look at the other queries as well, even those lacking consensus, as there are a few cases you might also want to support. Take for example: https://cburgmer.github.io/json-path-comparison/#array_slice_with_open_start_and_negative_step.

Once you release I can re-run and publish the comparison.

[ohler55]

I think I have the slices covered. I had the end being inclusive and the consensus was for exclusive. That covered a majority of the differences. I'll release and take another look.

BTW, the headers are readable now.

[ohler55]

Version v1.4.0 has been released.

[ohler55]

I'll submit a merge request for the update along with a few other changes to main.go and go.mod. You can choose to ignore of merge. I won't take offence in either case.

[ohler55]

Based on the results I think this issue can be closed. You agree?

Unrelated to closing the issue, have you considered scoring each implementation and displaying that somewhere? Maybe 1 point for consensus, .5 for a majority, .25 for minority, and zero for errors or not matching the consensus? I just picked those values on the spur of the moment.

The OjG README.md now has a link to your site.

[cburgmer]

Yes, happy to close.

Scoring

Yes and no. I believe it's complex :)
Currently the comparison table is nearly overflowing my monitor, and such a big table is difficult to read either way. I did contemplate allowing a denser view by default via hiding some of the less "compliant" implementations, but that opens up other questions:

• Would we only prioritise one or two implementations per language, or should we hide the whole Clojure column, just because it's implementation doesn't support much yet?
• Just counting successful test cases does not shed a good light on how many features are actually well supported. Especially as many queries test a handful of features together. We might be penalising implementations that don't support a certain feature that just happens to have very deep test coverage, and thus a high fail count.

The OjG README.md now has a link to your site.

Awesome. I hope we can help our users make better decisions when adopting a certain implementation.

[cburgmer]

Scoring

But feel free to raise an issue over at the comparison project so we can discuss with others whether there's a good alternative.

Also, my answer might have not caught you initial point. What is your goal for the scoring?

[ohler55]

The goal was a kind of overview or summary of the very long list. I'll create an issue and see how the conversation goes.