You should include the module is-my-json-valid by mafintosh, It's the fastest #72

ebdrup · 2015-01-18T10:38:37Z

is-my-json-valid is the fastest JSON-schema validator around.
It blows the competition out of the water in speed.
There is a blog post about it here:
http://cosmicrealms.com/blog/2014/08/29/benchmark-of-node-dot-js-json-validation-modules-part-3/

ebdrup · 2015-01-18T10:39:44Z

I mean in your benchmarks

automatthew · 2015-01-18T18:03:06Z

If the validator doesn't support the full spec, it is not a JSON Schema validator. I can write a very fast validator that doesn't support all of JSON Schema:

is_valid = (schema, document) ->
  true

That's a reductio ad absurdum, obviously. But the point should be clear. If a validator doesn't reject documents that are invalid according to the JSON Schema standard, there can be no valid (pun intended) benchmarking of it in comparison to actual JSON Schema implementations.

automatthew · 2015-01-18T18:05:06Z

I do want to review is-my-json-valid, though. And we should add to the benchmarks a check that the tested libraries can actually detect invalid documents.

ebdrup · 2015-01-18T18:33:34Z

Actually I think it does the spec.

—
Sent from Mailbox

On Sun, Jan 18, 2015 at 7:05 PM, Matthew King notifications@github.com
wrote:

I do want to review is-my-json-valid, though. And we should add to the benchmarks a check that the tested libraries can actually detect invalid documents.

Reply to this email directly or view it on GitHub:
#72 (comment)

mafintosh · 2015-01-18T21:55:25Z

is-my-json-valid should support all of schema v4 (except for refRemote, ref, which this validator doesn't seem to support either)

ebdrup · 2015-01-19T06:06:26Z

It will be interesting to see how you handle this. You have a chance to really shine for the open source community.

automatthew · 2015-01-20T20:43:37Z

is-my-json-valid should support all of schema v4 (except for refRemote, ref, which this validator doesn't seem to support either)

Good deal. No, JSCK doesn't do refRemote yet.

automatthew · 2015-01-20T21:05:01Z

@mafintosh, I'm seeing some indications that is-my-json-valid does not, in fact, support all of JSON Schema draft 4. Feel free to email me if you want to discuss this in private.

mafintosh · 2015-01-20T21:11:01Z

@automatthew we can do it in public. which parts are missing?

automatthew · 2015-01-20T21:21:36Z

I don't see where, for instance, "multipleOf" is implemented.

Following that line of inquiry, I changed the value of that keyword in your test fixture to something that should cause failures for both the valid and invalid documents. All tests still pass.

gilesbowkett · 2015-01-21T22:05:11Z

sorry, we're going to pass on this. we can give you a shout-out in the readme or something.

is-my-json-valid is very fast but it doesn't seem to properly support the spec.

lots more detail here: https://www.pandastrike.com/posts/20150121-the-fast-and-the-spurious

automatthew · 2015-01-21T23:45:12Z

It will be interesting to see how you handle this. You have a chance to really shine for the open source community.

I found it very interesting.

mafintosh · 2015-01-22T00:18:04Z

Didn't know there was a json-schema test-suite. I'll make is-my-json-valid pass that. The orderly thing isn't used for anything internally (or has anything to do with the performance). When I researched json-schema initially i just liked the syntax but since I don't really use it anymore I'll probably remove it at some point.

automatthew · 2015-01-22T00:27:25Z

You may also find these documents helpful:

http://tools.ietf.org/html/draft-zyp-json-schema-04

http://tools.ietf.org/html/draft-fge-json-schema-validation-00

gilesbowkett · 2015-01-22T01:28:49Z

I took out most of the stuff about Orderly. Wasn't quite sure how that worked anyway. The good news is every time I tried it, I got [object Object] from toString(), so index.js was probably falling through to the catch every time. So everything I said about Orderly may have been irrelevant. The bad news is I got this result while trying to get is-my-json-valid to tell me that an invalid doc was valid, and that I got that to happen pretty quickly.

mafintosh · 2015-01-22T09:25:07Z

@gilesbowkett @automatthew is-my-json-valid 2.0.0 now passes the entire JSONSchema draft4 test suite (with exception of remote refs and multibyte unicode string with min/max length)

Running your benchmark yields the following result

## Benchmarks for Draft 4

Schema: 'Event - Valid document'.  A simple schema, exercising very few attributes
Sample size: 64
Validations per sample: 1024

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 228.955    max: 254.094    min: 143.277

  tv4: validations/millisecond
  median: 125.429    max: 133.629    min: 82.071

  jayschema: validations/millisecond
  median: 1.933    max: 2.08    min: 1.554

  is-my-json-valid: validations/millisecond
  median: 1528.358    max: 2348.624    min: 965.127

  z-schema: validations/millisecond
  median: 97.789    max: 100.53    min: 63.175

Relative speeds:
is-my-json-valid : 1.000
JSCK : 6.675
tv4 : 12.185
z-schema : 15.629
jayschema : 790.847


Schema: 'Configuration'.  A moderately complex schema with some nesting and value constraints
Sample size: 64
Validations per sample: 256

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 107.113    max: 127.745    min: 52.502

  tv4: validations/millisecond
  median: 36.626    max: 46.486    min: 12.066

  jayschema: validations/millisecond
  median: 1.017    max: 1.127    min: 0.703

  is-my-json-valid: validations/millisecond
  median: 563.877    max: 992.248    min: 503.937

  z-schema: validations/millisecond
  median: 43.866    max: 45.674    min: 27.465

Relative speeds:
is-my-json-valid : 1.000
JSCK : 5.264
z-schema : 12.855
tv4 : 15.395
jayschema : 554.556


Schema: 'Transaction'.
Sample size: 64
Validations per sample: 64

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 15.885    max: 16.763    min: 14.408

  tv4: validations/millisecond
  median: 3.541    max: 3.632    min: 3.004

  jayschema: validations/millisecond
  median: 0.046    max: 0.05    min: 0.041

  is-my-json-valid: validations/millisecond
  median: 69.945    max: 87.075    min: 38.186

  z-schema: validations/millisecond
  median: 4.068    max: 4.455    min: 2.156

Relative speeds:
is-my-json-valid : 1.000
JSCK : 4.403
z-schema : 17.193
tv4 : 19.756
jayschema : 1519.454

is-my-json-valid is still consistently ~5x faster than JSCK

ebdrup · 2015-01-22T11:04:47Z

It will be interesting to see how you handle this. You have a chance to really shine for the open source community.

I found it very interesting.

It still is :-) But I'm really glad that you helped mafintosh make is-my-json-valid pass the tests.

gilesbowkett · 2015-01-23T20:26:32Z

reopened because #74, and well done @mafintosh.

just FYI, the pandastrike.com blog post about this will probably get updated accordingly.

mafintosh · 2015-01-24T16:38:29Z

@gilesbowkett cool! looking forward to reading it

gilesbowkett · 2015-01-26T17:49:37Z

here you go: https://www.pandastrike.com/posts/20150123-congrats-imjv

Still got to add an update to the pandastrike.com post about IMJV, but my personal blog post was updated with a link.

gilesbowkett · 2015-01-29T00:13:17Z

resolved in 5dba843

ebdrup closed this as completed Jan 18, 2015

ebdrup reopened this Jan 18, 2015

gilesbowkett closed this as completed Jan 21, 2015

mafintosh mentioned this issue Jan 22, 2015

Fixes #72 add is-my-json-valid to benchmarks #73

Closed

gilesbowkett reopened this Jan 23, 2015

mctep mentioned this issue Jan 25, 2015

Include mc-schema in bechmarks, It's faster #76

Closed

gilesbowkett closed this as completed Jan 29, 2015

gilesbowkett mentioned this issue Feb 1, 2015

Add link to open source benchmark for JavaScript implementations json-schema/json-schema#153

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

You should include the module is-my-json-valid by mafintosh, It's the fastest #72

You should include the module is-my-json-valid by mafintosh, It's the fastest #72

ebdrup commented Jan 18, 2015

ebdrup commented Jan 18, 2015

automatthew commented Jan 18, 2015

automatthew commented Jan 18, 2015

ebdrup commented Jan 18, 2015

I do want to review is-my-json-valid, though. And we should add to the benchmarks a check that the tested libraries can actually detect invalid documents.

mafintosh commented Jan 18, 2015

ebdrup commented Jan 19, 2015

automatthew commented Jan 20, 2015

automatthew commented Jan 20, 2015

mafintosh commented Jan 20, 2015

automatthew commented Jan 20, 2015

gilesbowkett commented Jan 21, 2015

automatthew commented Jan 21, 2015

mafintosh commented Jan 22, 2015

automatthew commented Jan 22, 2015

gilesbowkett commented Jan 22, 2015

mafintosh commented Jan 22, 2015

ebdrup commented Jan 22, 2015

gilesbowkett commented Jan 23, 2015

mafintosh commented Jan 24, 2015

gilesbowkett commented Jan 26, 2015

gilesbowkett commented Jan 29, 2015

You should include the module is-my-json-valid by mafintosh, It's the fastest #72

You should include the module is-my-json-valid by mafintosh, It's the fastest #72

Comments

ebdrup commented Jan 18, 2015

ebdrup commented Jan 18, 2015

automatthew commented Jan 18, 2015

automatthew commented Jan 18, 2015

ebdrup commented Jan 18, 2015

I do want to review is-my-json-valid, though. And we should add to the benchmarks a check that the tested libraries can actually detect invalid documents.

mafintosh commented Jan 18, 2015

ebdrup commented Jan 19, 2015

automatthew commented Jan 20, 2015

automatthew commented Jan 20, 2015

mafintosh commented Jan 20, 2015

automatthew commented Jan 20, 2015

gilesbowkett commented Jan 21, 2015

automatthew commented Jan 21, 2015

mafintosh commented Jan 22, 2015

automatthew commented Jan 22, 2015

gilesbowkett commented Jan 22, 2015

mafintosh commented Jan 22, 2015

ebdrup commented Jan 22, 2015

gilesbowkett commented Jan 23, 2015

mafintosh commented Jan 24, 2015

gilesbowkett commented Jan 26, 2015

gilesbowkett commented Jan 29, 2015