Building "official" JSON Schema foundational tooling

[jviotti]

At the time of this writing, there are is a significant amount of third-party JSON Schema tooling that deviates from the specification despite the best intentions of their authors given the complexity of JSON Schema. When foundational pieces of JSON Schema code start deviating, higher-level software built on top of this shaky foundation starts deviating as well. This results in a significant amount of confusion around certain features of JSON Schema, adds significant support overhead to the JSON Schema team and even results in entire products that mis-use JSON Schema.

To break this cycle, I propose that the JSON Schema organization maintains foundational tooling around JSON Schema themselves. This would allow JSON Schema to provide robust foundational blocks that others depend on and steer the world towards a correct use of JSON Schema.

I'd like this thread to serve two purposes: identify the minimum set of foundational blocks that must be built to support higher-level applications and identify the guidelines for a JSON Schema module to attain "official" status.

Foundational Blocks

These modules must be low-level and generic utilities that other module or product authors can plug together to make use of JSON Schema on their respective use cases.

Some ideas:

• A JSON Schema validator that produces output according to the specification
• A JSON Schema bundler
• A JSON Schema "walk" module that visits every subschema (as a foundational block to implement linters, etc)
• A JSON Schema testing framework
• A JSON Schema linter
• A JSON Schema tool that transforms schema definitions across drafts for backwards compatibility reasons (in a lossy manner)
• A JSON Pointer and Relative JSON Pointer implementation

Attaining Official Status

I think we should clearly define what are the characteristics that a JSON Schema tool must have to be considered "official". These are some of the questions I can think of:

• Should it support the last draft, or the N last drafts?
• Should it be published under an official jsonschema namespace? (in the case of NPM)
• Should it be written in JavaScript? What about other programming languages?
• Should it run on the browser?
• Should it be making use of the official test suite in some way or another?
• Should it have an appointed maintainer?
• What is the expected level of documentation?
• What software license must be adopted?

What do you think?

[gregsdennis]

We've had internal discussions around this before, and even quite recently, especially since entertaining the idea of joining a publication group such as IETF or JSF, and decided that we want to focus on writing the specification and maintaining the test suite which ensures conformance, leaving implementations and other tooling to the community.

One of the problems that side out of providing such tooling ourselves is language/framework. How do we, the maintainers of a specification, decide which language is best to write a reference implementation? Having such an implementation could sway decisions on features because the chosen language does or does not support a proposal (easily). We tend toward interoperability, so it's disadvantageous for us to choose one over others.

Many of the other questions you mention were also considerations.

[Relequestual]

tl;dr: I don't want this Discussion to be about having an "official validator" or not, because I'm confident we don't need or want one. I want us to think about OTHER tooling we could build.

Nearly the whole value proposition for JSON Schema is that it's interoperable. If you don't need interoperability, you're almost certainly better off going with something else that isn't limited by the tradeoffs of something designed for interoperability (ajv vs joi). - @jdesrosiers

100% agree on this! I raise this often.
"If you need to do more than JSON Schema can do, and interoperability isn't a concern, then you might have the wrong tool for the job."

---

This post started as an idea from a conversation with @Relequestual , who mentioned json-schema-ref-parser in particular and AJV having certain deviating features. I think Ben might have more examples to add to this discussion. - @jviotti

I don't think we as an org need to create and publish a JSON Schema Validator. We invest in the test suite. We try to make picking an implementation. We can do more to help this with a badging system. I'm working on a proposal for this.

What I DO think we can consider is other tooling, such as a linter.

JSON Schema ref parser that isn't?

json-schema-ref-parser is IMHO badly named. Personally I want to superseed it with a simple bundler (such as the one @jdesrosiers recently published), but the problem is tooling which doesn't want to handle ANY references is still going to be around.

The problem is, people will continue to write tooling which "works for them" and simple schemas, but that doesn't support references in any way. We should avoid using those or help them support internal references. I think most people underestimate how a basic and simple implementation of using references can work. (This would be an example of a thing we could build as a reference implementation to demonstrate.)

I know @philsturgeon would like us to rally and try to upgrade json-schema-ref-parser to be JSON Schema aware, but I don't know if that effort is worthwhile when a new bundler was written overnight that follows the (now defined) bundling process. Granted, it wasn't defined previously. I don't know how much effort it would take, nor if it would result in breaking expectations for current users. (The new bundler looks to do a single thing, while json-schema-ref-parser does several.)

(The work @philsturgeon does for open source is admirable and I have a lot of time for him. I do however struggle to get excited about a tool which historically seems to cause lots of problems. Especially when there's a new simpler tool which will be easier to maintain.)

---

The problems we have with ajv today have more to do with what they don't implement than any customizations.

There is the issue of the current default settings which throw errors when processing schemas AJV deems to be "not best practice", yet are totally valid under the spec. Sure, you can change these options, but only when you're using it directly. If it's a deeply nested dependency, you're out of luck, and that just sucks.

[jdesrosiers]

Oh yeah, I can't believe I forgot about ajv's "linting" features. That is divergent and has certainly been causing problems.

As for json-schema-ref-parser, I don't think that's a matter of us rallying. My understanding was that the maintainer is simply not interested in including JSON Schema awareness in the library. The problem isn't that no one has made the effort to do it, it's that the maintainer doesn't want it. I could be wrong about that, but that was my understanding.

[Relequestual]

The maintainer has stepped back a little and it's now housed under https://github.com/APIDevTools which is mostly driven by Phil, as I understand.

In recent private DMs, it sounded like Phil wanted to see a scope change to make it JSON Schema aware. Hopefully @philsturgeon will pick up on this discussion and clarify =]

[philsturgeon]

@jdesrosiers that's definitely not the case. The threads you've seen were talking about the original intent of the code, not about what is wanted or not. Instead of trying to wrangle json-schema-ref-parser into something that properly supported $id/$schema, the maintainer started creating https://github.com/APIDevTools/json-schema-reader instead. This would have everything folks liked about json-schema-ref-parser, but be properly JSON Schema modern compatible.

Mid way through this new build, the maintainers family went through some difficulties which I wont get into, but they were forced to refocus their time and abandon the work on both tools. I got @stoplightio to pick up some of the maintenance work on json-schema-ref-parser, but that did not include time to finish the rewrite json-schema-reader.

So, that library is still the north star for solving the problem, and it would definitely benefit a thorough investigation to see if it's any good.

If it's good but needs work, let's find people to get the work done.

If it's no good, put together a library which does what is aims to do and write a migration guide. I'll help folks transition.

As I said to Ben before we don't need to fight it or explain to people why its crap, we need to replace it with something viable and help people move over, and maybe that's json-schema-reader and maybe its something you folks knock up.

[jdesrosiers]

@philsturgeon Thanks for the correction and additional context.

[Relequestual]

I think some of these items would be in scope, and some would be out of scope.
My feeling considerations will be different if we had full time staff to develop and maintain tools. It's highly likely at least one full time developer will be joining next year, and I'm hoping for more.

Let's approach this discussion with an air of "what if?" or "what would it take?" thinking.
Let's imagine.

[jviotti]

Indeed. I realize I should have mentioned that. My proposal is based on the assumption that JSON Schema will have N number of full-time developers soon.

[Relequestual]

FYI, here's a strawman badges proposal: #118