JSON schema

[dontcallmedom]

See http://lists.w3.org/Archives/Public/site-comments/2018Jan/0005.html and https://datatracker.ietf.org/doc/draft-handrews-json-schema/

[tidoust]

A few notes following discussions with Ben Hutton:

• All efforts are being tracked on the JSON Schema Web site
• Upcoming v07 draft of JSON Schema should be ready in a few months from now.
• Many tools still follow draft v04. Goal for standardization is not to rubber-stamp v07, but to find the right stable set of features for JSON Schema.
• JSON Schema's main strength and weakness is that it is used by many many companies and developers around the world, often for "small" schemas. So it's both pervasive and hard to single out a core set of companies that could drive the standardization effort.
• As such, if JSON Schema is to move to standardization, next step would be to assemble a standardization community around it. Idea could be to look at products that embed JSON Schema one way or the other, following the chain of dependencies on npm and others (e.g. the list of projects that depend on the json-schema library or the list of projects that depend on ajv)

[tidoust]

Another point to keep in mind for later, again if JSON Schema is to move to standardization: we would need to sort out copyrights and license for the document. The initial versions of JSON Schema were written by people who are no longer around. Also, I don't see any reference to licensing terms on json-schema.org.

[awwright]

@tidoust As far as licensing goes, as an IETF submission, all parts of the drafts are covered by RFC 5378 (as you're probably familiar, I'm not sure if that's sufficient though). Additionally, all of my work (which is around half of each of the documents) is public domain.

[handrews]

@tidoust I should double-check with my employer, but when I started working on the spec they had no concerns over it, and were familiar with IETF IP rules (other folks in the company have contributed to RFCs). Anything that I would personally have rights to I'm placing in the public domain. I'm pretty sure that 90% of the current draft has been reworked to some degree or another by either @awwright or myself, if that makes any difference. Of the past authors, only Gary Court has avoided replying to any effort to reach him. We've interacted with Krys and Geraint within the last few months, and I think Francis is active in other projects on GitHub.

[Relequestual]

@tidoust Apologies for still having not replied to your follow up email. Joining the W3C from a company perspective for me is not going to happen. I think we're focusing on getting draft-8 done now, but we'll set aside some time to discuss this soon!

[epicfaace]

@Relequestual Has the discussion on whether to move JSON Schema to W3C happened yet? I'm curious to know what the conclusion was.

[iherman]

The JSON Schema homepage has the following remark:

The JSON Schema project intends to shepherd all four draft series to RFC status. Currently, we are continuing to improve our self-published Internet-Drafts. The next step will be to get the drafts adopted by an IETF Working Group. We are actively investigating how to accomplish this.

In other words, it looks like the community has decided to go with the IETF and not with W3C. If this is confirmed, we should probably close this issue.

@tidoust ?

[handrews]

@iherman this has not been resolved one way or the other.

That statement is on the JSON Schema web site because unless/until a concrete new plan is present, there is no reason to take down the prior plan. We're still publishing IETF drafts (hopefully again by the end of this month, in fact).

However, the same issues that brought up the question of IETF vs W3C vs ??? still exist.

[iherman]

Thanks @handrews, good to know!

[epicfaace]

What are the general benefits/drawbacks/considerations for deciding to host a spec with IETF vs W3C?

[akuckartz]

What is the relation/compatibility with JSON-LD ?

[handrews]

@epicfaace it's discussed in detail at json-schema-org/json-schema-spec#526

In particular, as you can see at https://mailarchive.ietf.org/arch/msg/json/w--4QtcF9remEI3FfGM6ghBFkEQ that the folks on the IETF JSON mailing list were very uninterested in the project. There are a couple of competing projects, none of which are widely used or implemented, which the IETF seems to prefer. The fact that many popular projects and products use JSON Schema, and have for many years, and in some cases are actively collaborating with us on the direction of the spec, seems to be completely irrelevant to the IETF folks.

For us at the JSON Schema project, it is important to continue to support our large existing user community. Since the IETF is only interested in starting over, at least for a standards-track RFC, it seems unlikely that we'll go that route. We could publish an informational RFC, though, which might be the fastest track to a "standard" (informational RFCs are, of course, not actually official standards, but they are at least published which is sometimes mostly what people want).

Speaking only for myself, I would expect having JSON Schema adopted by a standards group would result in further changes and refinements- that does not bother me. But I don't see the point in us throwing out what we have entirely and starting over, but still calling it JSON Schema.

[handrews]

@akuckartz I see the technologies as complementary.

JSON Schema essentially lets you do two things: assert conditions about a JSON instance document, and annotate that document with further information. Assertions and annotations can be combined such that annotations are applied conditionally depending on which assertions are found to be valid vs invalid.

You can also use those assertions and annotations without an instance to generate things related to the set of possible instances, such as a web form that can be used to construct an instance, or an OO class that can work with an instance in a particular programming language.

It does not specifically work with semantics (although the format keyword sort-of does, but format is a bit of a mess- it gets slightly better in the next draft, but fundamentally if you want to work with semantics you should go with JSON-LD).

I believe that JSON-LD could be mixed with JSON Schema using JSON Schema's annotation functions. Essentially, JSON Schema's format keyword attempts to annotate fields/objects/arrays in a JSON document with their semantic meaning, but is a very limited way of doing that.

It's been ages since I even looked at JSON-LD so I can't give a good example off the top of my head.

[BigBlueHat]

It's been ages since I even looked at JSON-LD so I can't give a good example off the top of my head.

I'd be happy to help explore making JSON Schema and JSON-LD work (even) better together. The Web of Things WG's Thing Description spec builds upon a sub-set of the JSON Schema vocabulary and integrates JSON-LD alongside, so folks in that community might also be interested in helping there.

[BigBlueHat]

Speaking only for myself, I would expect having JSON Schema adopted by a standards group would result in further changes and refinements- that does not bother me. But I don't see the point in us throwing out what we have entirely and starting over, but still calling it JSON Schema.

JSON Schema's wide adoption (and adoption/use within various W3C specifications) makes it a great candidate for a home at the W3C. The place (afaik) to start would be via a Community Group.

I'd be happy to work with y'all to make that happen--and I've reached out to @Relequestual recently to discuss that (and then was pointed here by @iherman 😄).

I'm subscribed to the json-schema-org/json-schema-spec#526 issue and would be happy to help assist in getting a Community Group started--which should be the right place to sort out IP concerns, setup calls/processes, and get things moving forward under the W3C banner.

There's a lot more tofu making beyond that, but the CG model should be the right place to take the first steps! Or so I hope. 😃

[Relequestual]

My feeling is that we put this discussion on hold till at least half way through 2020.
We have our work cut out getting the test suite done for draft 2019-09.

[iherman]

@Relequestual understood.

[BigBlueHat]

@Relequestual do reach out along the way, as I'd be happy to help you all get integrated here!

Also "done" specs don't ever stay "done" when they go through any standards process...so starting earlier than later has massive benefits.

I'll be around in both communities, so please ping me as you need/want help in this area.

❤️,
🎩

[handrews]

@BigBlueHat Part of the reason for waiting is that I've been the one driving the major features like vocabulary support, and those are hard enough within our own little group to get to a point where we can publish and seek feedback.

Given the choice between delaying to involve a formal group vs reaching feature-complete as quickly as possible, when we have major users waiting on the changes who have their own timelines that are more industry-driven than standards-body-driven, my strong preference is to produce a more or less functional, feature-complete spec.

I do not personally have the patience to deal with a formal standards body, while @Relequestual is both willing and better suited to it 😄 I'll probably be actively around for one more significant draft incorporating feedback on the slightly unfinished features, after which I'll step back and the formal standardization process can do whatever they do. I'll be available for consultation as needed, but I will not be looking for any control over where it ends up after that. For me, "done" is "it meets the needs of the existing community who are already trying to use it."

I'll be enthusiastically rooting for the standards process, regardless of whether the end result looks like what goes in or not. But I want to resolve my own work and do a clean hand-off rather than a tug-of-war between visions that will take a near-complete system and re-introduce uncertainty. Which seems inevitable with more stakeholders. I recognize that this is a bit of a selfish position, but I've put a lot into the last several drafts and this is the only way I could finish that without total burnout. I want to hand off something that makes sense as a system, because that seems like the best starting point for the next phase.

And I feel justified in having held off by the imminent inclusion of the latest draft into OpenAPI 3.1, which will resolve an extremely problematic years-long schism (in which there are no villains- everyone has made the best available choices). I'm more concerned with that practical problem than any formal stamp (others definitely have other priorities!).

Anyway, if you're concerned over any delay in engaging the formal standards process, my point here is that I'm the roadblock more than @Relequestual or anyone else is. But I am working to get out of the way. Feel free to reach out to me on the json schema slack if you want to discuss my personal concerns in more detail, otherwise @Relequestual is the person you want to talk with 😁

[pchampin]

I had a conversation today with @egekorkan from Siemens and from the WoT WG (NB: the WoT WG works quite a lot with JSON-Schema, see for example that blog post). He mentioned that JSON-Schema's governance has quite changed since we had the conversation above, and that it might be worth reactivating this debate again.

Note that currently, citing JSON-Schema normatively is challenging, because despite being largely used an implemented, JSON-Schema is not specified by a recognized entity. The WoT WG has encountered this problem, and was led to duplicate part of the JSON-Schema spec in its own REC for that reason -- which is a source of confusion for readers...

[iherman]

FYI, the VC WG has the same problems concerning the normative reference to JSON-Schema; we would also like to use it in our spec.

[lu-zero]

The current specifications are according to here

• draft-bhutton-json-schema-01 (core)
• draft-bhutton-json-schema-validation-01
• draft-bhutton-relative-json-pointer-00

And since its relationship with the WoT DataSchema is not as clear as it should we'll have to clarify and figure out better how to express which is the supported subset. An implementor as today would have to support the full json-schema (current) and still hope for the best to be interoperable.

[iherman]

See also https://github.com/orgs/json-schema-org/discussions/3#discussioncomment-6290893

[lu-zero]

If they want to stay non-standard and evolving, it would be quite problematic to use it in our specifications.

[jdesrosiers]

Every JSON Schema spec release has a stable URI. What's the benefit of referencing the release notes rather than the spec itself?

• 2020-12 - https://tools.ietf.org/html/draft-bhutton-json-schema-validation-01
• 2019-09 - https://tools.ietf.org/html/draft-handrews-json-schema-validation-02
• draft-07 - https://tools.ietf.org/html/draft-handrews-json-schema-validation-01

JSON Schema is a living specification

This is not correct. Maybe that has led you to believe that there aren't stable references to past releases?

[OR13]

My understanding from @Relequestual was that IETF was not the place to reference for JSON Schema draft versions, and that there are currently no plans to standardize JSON Schema at IETF.

[jdesrosiers]

We're not planning to standardize with IETF, but I don't see why you couldn't reference the drafts we've already published with them. We're going a different direction going forward, but those past drafts will always be hosted by IETF.

[mnot]

@jdesrosiers that's not true -- Internet-Drafts explicitly state:

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."

Until relatively recently, Internet-Drafts were removed from the IETF Web site(s) once they expired. That changed, but there is no guarantee that they will persist.

[jdesrosiers]

there is no guarantee that they will persist.

That's good to know! We've been linking to past drafts from our website forever and never had a problem. The oldest having expired in 2010. We'll have to look into hosting copies ourselves.

And, yes, we are aware that we haven't used I-Ds the way they're intended. It's a big reason we're moving away from using them. Our circumstances don't align well with that process. Our past releases are in I-D format, but they really aren't true I-Ds because we haven't used them as such.

Perhaps the solution is for JSON Schema to self host the old drafts and projects like this one can reference those. We can't change that they are in I-D format. What's done is done and it's all we have. But, linking to them from a JSON Schema domain rather than an IETF domain is probably the best we can do to signal that they don't follow IETF norms. I know it's not perfect, but there has to be some way for it to be referenced because it's all we have.

Sorry for the trouble our misuse of I-Ds has caused. We are working on fixing that going forward.

[plehegar]

JSON Schema is a living specification

This is not correct. Maybe that has led you to believe that there aren't stable references to past releases?

Whether it's a LS or not isn't relevant for the VC/JSON-Schema spec in my mind.

Nevertheless, I appreciate the correction. The reason why I wrote that was because you wrote:
[[
Instead of continuing to release a new immutable and incompatible version of JSON Schema with each release, our next release will be a long-lived version that is stable, but evolving.
[...]
That vision of a stable yet continuously evolving spec doesn't fit well with the IETF process.
]]

[jdesrosiers]

Yes, we're moving in that direction, but we aren't there yet. Anything released in the past doesn't apply to that vision and those past releases are what this discussion is talking about. I only mentioned this in case people thought that it applied to the releases being discussed here. When the time comes to reference a future release, things will be different.

I'll quickly add some clarification to what to expect from JSON Schema in the future, but keep in mind that none of this is set in stone yet. We're emulating ECMAScript's process, so you can generally expect JSON Schema to evolve in a similar way to JavaScript. When ECMAScript 2024 comes out it's not a new version of JavaScript, it's still the same old JavaScript, just with a few new features added. JSON Schema will work the same way rather than the old way of each release being a distinct and potentially incompatible version from the last.

Like ECMAScript, we intend to publish an maintain a full spec snapshot for each release. It won't be necessary to reference the release notes if you need to reference a specific feature set of JSON Schema. There will be, for example, a JSON Schema 2024 spec that you can reference directly.

[OR13]

I'd like to close off this thread with a simple example:

"heres how to cite JSON Schema, and specific versions of JSON Schema at the W3C"... then "mark issue as closed".

I still don't know how to do this.

[Relequestual]

I'd like to avoid marking this as closed for now. Thanks.

[plehegar]

@OR13, as a reminder, context matters here. Depending on how deep one uses JSON Schema will affect how to reference it. I simply suggest to make a proposal for your specific context.

[mnot]

Since this thread is still active :) a few points:

• Martin Thompson has floated a proposal to end expiry of Internet-Drafts. If that gains traction, citing the IETF URLs should be stable (although you still technically need to cite them as 'work in progress').
• The pushback from the IETF JSON community above was a long time ago now, and I suspect things may have changed, especially since the HTTPAPI community is now established there. And, even if the JSON greybeards are unexcited about it, that doesn't necessarily stop it -- especially if there's an active community that wants to bring work to the IETF.
• That said, JSON Schema has now been in development for several years and is widely used. Standards activity in the IETF or the W3C is likely to break things and cause disruption, because change control will pass to the standards body. That makes me think that the right thing to do might be to publish the JSON Schema community's preferred version (whenever it's ready) on the Independent Stream -- you get an RFC that other people (including the IETF and W3C) can cite normatively, but don't pass change control to the IETF. Then you can consider standardising JSON Schema 2.0 (if you like and where you like).

Happy to help / answer questions if there's interest in that path.

[Relequestual]

@mnot This sounds like something we should consider for sure.
I think we would need to pick apart our ADR on decoupling from the IETF, and discuss if we can align on our expectation of travel. (Related comment from @awwright that suggests it would be possible.)

I think the points you make above are enough to be considered sufficient additional information to revisit the decision record results.

[jdesrosiers]

It's great to hear that removing expiration of I-Ds is being considered! Is there any chance this change could be retroactive and apply to previous versions of JSON Schema?

The Independent Stream is certainly a better choice for us in that it keeps change control with us, but I don't think it changes the core incompatibility we have with IETF. The normal process for RFCs is that you work through as many drafts as are necessary and then finish with an RFC. An RFC is expected to be done. Updates are expected to happen occasionally because real life is messy and the world changes, but planned regular updates are not an expected part of the normal process. Our expectation is that JSON Schema will continue to evolve for some time still. We are working toward moving to a model similar to ECMAScript where we introduce compatible changes on a yearly cadence. I've never heard of an RFC that worked that way. I expect we'd be told that if we need to be making updates that often, we're not ready for RFC and we should do more drafts. Meanwhile we're perpetually in the awkward position of developing something as a draft that's being used widely in productions systems. Please let me know if I'm misunderstanding anything here and you think there's a way such a process would fit within IETF.

[mnot]

To be clear, the Independent Stream isn't part of the IETF -- it's a separate user of the RFC Series. For that matter, Internet-Drafts are not monopolised by the IETF either; they're used by indivduals and other organisations (like the IRTF).

Regarding RFCs being "done", we don't consider that the case for HTTP (where four separate RFCs have defined HTTP/1.1 successively - 2068, 2616, 7230, and 9112). It's not the case for other successful protocols either, and in general there are many in the IETF who see the merits of continuous maintenance / "living specs".

You should be aware that the Independent Stream Editor decides what is included on the stream, so you should talk to him about what the process for publishing there would look like, to make sure you're comfortable. Generally, he won't meddle in the details of the document -- he just wants to make sure that what's on the stream is of reasonable quality.

[Relequestual]

To be clear, the Independent Stream isn't part of the IETF -- it's a separate user of the RFC Series. For that matter, Internet-Drafts are not monopolised by the IETF either; they're used by indivduals and other organisations (like the IRTF).

Well, that sounds interesting. What are the practical on the ground differences as a result? Could you like us to maybe a few RFCs from the Independent Stream?
I did a little reading and it looks like the RFC could only be published as "Informational" or "Experimental". Is that correct?

Regarding RFCs being "done", we don't consider that the case for HTTP (where four separate RFCs have defined HTTP/1.1 successively - 2068, 2616, 7230, and 9112). It's not the case for other successful protocols either, and in general there are many in the IETF who see the merits of continuous maintenance / "living specs".

Very good and useful to know. Thank you. The pace of publication for those you mentioned may be slower than what we currently have planned, but that may be fine.

You should be aware that the Independent Stream Editor decides what is included on the stream, so you should talk to him about what the process for publishing there would look like, to make sure you're comfortable. Generally, he won't meddle in the details of the document -- he just wants to make sure that what's on the stream is of reasonable quality.

Sounds promising. Maybe our planned release schedule is something that could work with these type of submissions.
It looks like the preferred contact method is to email rfc-ise@rfc-editor.org, correct?

[jdesrosiers]

@mnot Thank you, that's really useful information!

Regarding RFCs being "done", we don't consider that the case for HTTP (where four separate RFCs have defined HTTP/1.1 successively - 2068, 2616, 7230, and 9112).

Actually, HTTP was what I had in mind when I mentioned reasons for change being the world changing and real life being messy. I know you expect things may need updates from time to time. However, that's not same thing as a continuous maintenance / "living spec" type of process. I haven't seen an RFC Series spec that works like that, but I'm very intrigued to hear that there might be support for such a process within the IETF or the Independent Stream.

[mnot]

Could you like us to maybe a few RFCs from the Independent Stream?

See: https://rfc.fyi/?stream=independent

I did a little reading and it looks like the RFC could only be published as "Informational" or "Experimental". Is that correct?

Correct.

It looks like the preferred contact method is to email rfc-ise@rfc-editor.org, correct?

Yes.

I know you expect things may need updates from time to time. However, that's not same thing as a continuous maintenance / "living spec" type of process.

In my mind, you'd be publishing drafts (either as Internet-Drafts or on your own site) more often, with publications of 'checkpoint' / 'milestone' RFCs less often. YMMV, of course.

[egekorkan]

Dear W3C Strategy team,

This comment represents the collective opinion of the W3C Web of Things (WoT) Working Group (WG) and has been reviewed by the entire group.

The WoT WG is a long-time user and supporter of the JSON Schema activities.
This message has been conveyed in the past via a joint blog post at the JSON Schema blog (see here), which was also reviewed by the W3C MarComm team.

The WoT WG supports the path the JSON Schema Project has decided to take in becoming a self-governed entity that maintains the JSON Schema standards.
We prefer W3C to consider JSON Schema specifications to be normatively referencable and integratable into W3C specifications.
This is supported due to its strong usage within the WoT WG, which is explained below.
The WoT WG needs a firm basis for our current and future standardization work.

Usages within WoT WG

WoT WG uses JSON Schema in the WoT Thing Description Recommendation since its first public working draft, still a major part of the 1.1 Recommendation and will be used in the scope of the next WG charter in similar ways.

Data Modeling and Description

The primary use of JSON Schema is modeling and description of the data that an IoT device sends or receives.
TD document calls it a Data Schema (see this section), which is tightly bound to the JSON Schema specification but also to its lifecycle.
We have even published a JSON Schema ontology at https://www.w3.org/2019/wot/json-schema#

Problem: The WG had to manually copy over a subset of JSON Schema Draft 7 since we are not able to create a normative reference to JSON Schema and tell our readers, users, and implementers that they can just the JSON Schema Draft 7 version. This is problematic because:
- This requires additional work from the WG to write, sync, and maintain it.
- Our readers, users, and implementers get confused. Some implementers started implementing a JSON Schema validator from scratch since they did not realize that they could use an off-the-shelf validator
- There is the potential to create an accidentally incompatible dialect by not staying aligned with JSON Schema.

Validation of TD Instances

Since WoT TDs are usually serialized into JSON-LD, we provide a JSON Schema for validation of TDs.
This is published together with the REC at https://www.w3.org/2022/wot/td-schema/v1.1 and is used by the WoT Discovery specification for validation of TDs submitted to a TD Directory.
Other validations (code, SHACL) are also possible but JSON Schema is a good balance of lightweightness and expressiveness.

JSON Schema is also the core of the tooling used for generating the implementation report, a crucial step in getting to the REC stage.

Scripting Usage

WoT Scripting API uses JSON Schema for two purposes:

1. Generation of TypeScript definitions, which are used by open-source projects such as the Thingweb project in the Eclipse Foundation.
   These can be found at https://github.com/w3c/wot-scripting-api/tree/main/typescript
2. It describes a validation process of payloads at https://www.w3.org/TR/wot-scripting-api/#dfn-check-data-schema, which can be omitted if JSON Schema can be just referenced. This has caused implementation issues in the past since the algorithm was stricter than the validation algorithm used in JSON Schema.

[egekorkan]

Also see https://landscape.json-schema.org/ for JSON Schema's adoption in the industry overall, where WoT is also listed.