Overriding annotation/usage keywords at point of use

[handrews]

Use Case and Motivation

This is one of several proposals to come out of the analysis of the "ban additional properties mode" vs $merge/$patch debate and the various use cases that fed into that.

One common use case identified was specifying information about a type's usage at the point of use rather than as part of the initial type definition. In some cases, it is reasonable to provide some generic usage information in the type definition which serves as default usage documentation, but needs to be overridden in some uses with more specific information.

Additionally, default cannot meaningfully appear in multiple branches of logical keywords such as allOf, as there is no sensible way to choose which value is correct for the specific usage. Conflicting validations in an allOf simply fail the allOf, but conflicting default values are just unusable, independent of validation.

Prior experience

This particular use case has been a major pain point on both large-scale JSON Schema projects with which I have been involved.

One implemented $merge, which was very effective but (as several people have noted, and as I now agree) it is too powerful as it (and $patch) can literally convert any schema into an arbitrarily different schema, or even into a non-schema JSON value. In this project, $merge was used to override default values for default, title, description, and readOnly, or disambiguate among several possible values for those keywords.

The other project ended up redefining many types that should be sharing a definition. In some cases, nearly every schema has had to redefine a particular sub-schema just to change the title and description.

Preserving independent validation

In order to preserve the self-contained nature of validation, usage overriding only applies to the keywords in the "Metadata keywords" section of the validation specification (title, description, and default) plus usage-oriented hyper-schema keywords (readOnly and pathStart).

Breaking self-contained validation has been the main objection to other proposals in this area, either through overly powerful keywords or changing validation modes in ways which are not reflected in the schema itself. This is the only proposal to date that solves this problem with no impact on validation whatsoever.

Existing workarounds

It is possible to use an allOf with a reference to the type schema and a schema that includes only the point-of-use annotations. This is tolerable when reading the schema, but is not good for documentation generation. Each schema within an allOf should be self-contained, which separates the documentation from the thing it's documenting. Additionally, if default annotation fields have been filled out, a documentation tool that attempts to extract and combine annotation data from all allOf branches will just produce conflicting documentation.

Finally, allOf is nonsensical when combining multiple schemas using default, readOnly, or pathStart, as there is no reasonable way to choose which value to use.

The proposal: $use

The keyword $use would be an annotation/hyperschema keyword indicating that a type (via a $ref) is being used with additional annotation and/or hypermedia for this specific use.

$use is an object with two required properties and one optional property; the format for the required properties is borrowed from the $merge and $patch proposals:

• source MUST be a schema, and will nearly always be a $ref as there is no need for $use if you have the literal schema defined in line. Required.
• with should be applied to the resolved source as an application/merge-patch+json instance, which MUST NOT affect any JSON Schema-defined keyword that is not explicitly allowed by the proposal. Required.
  • Non-keyword property names are allowed in order to facilitate JSON Schema extensions that can opt to apply $use to their extensions or not as they choose. Required.

JSON Merge Patch considerations

Declaring the with object to follow JSON Merge Patch semantics allows implementations to use existing libraries, and frees us up from needing to debate yet another schema combination format.

However, a notable limitation of JSON Merge Patch is that you cannot overwrite something to null, as specifying a null value instead deletes the key entirely. A workaround for this that is consistent with existing JSON Schema behavior is to define null somewhere (perhaps right in the $use object since it does not forbid additional properties) and then $ref it. We would need to declare that $use is processed before $ref. Which is fine because $ref often needs lazy evaluation due to circular references anyway.

Elsewhere, there's been an assertion that $ref must always resolve to a schema. This would require breaking that assumption. Since the assumption was not present in Draft 04 (in which JSON Reference was an entirely separate specification), I think this isn't too unreasonable. A compromise could be that it can reference either a schema or null.

If $ref-ing null doesn't gain acceptance, another option would be an additional keyword in the $use object listing JSON pointers to properties that should be set to null.

"$" in name

I chose $use rather than use on the grounds that keywords that manipulate the schema itself should stand out (I will be filing a more comprehensive issue on the use of $ later as it comes up in yet another not-yet-filed proposal, plus there is an open issue to consider $id in place of id).

Example

{
    "type": "object"
    "properties:" {
        "interestTimestamp": {
            "$use": {
                "source": {"$ref": "#/definitions/specialTimestamp"},
                "with": {
                    "title": "Last Event of Interest",
                    "description": "The last time that something interesting happened.  Cannot be directly updated through the API.",
                    "readOnly": true,
                }
            }
        }
    },
    "definitions": {
        "specialTimestamp": {…}
    }
}

This shows that the type can be concisely defined in a shared location, with a clear usage object applied to it.

Meta-schema considerations

This could be added to the meta-schema by moving most of the keyword definitions into the meta-schema's definitions section grouped by type of keyword, and defining with as a schema minus the forbidden keywords.

[epoberezkin]

I miss the point completely - why do you need a special construct to add meta keywords?

Why not just:

{
  "title": "foo",
  "$ref": "bar"
}

as title etc. are ignored anyway?

Or if you want you can:

{
  "title": "foo",
  "allOf": [ { "$ref": "bar" } ]
}

$merge/$patch and banUnknownProperties idea solve additionalProperties issue. $merge/$patch also solve schema extension issue for recursive schemas (meta-schema is just an example, recursive schemas are very common). I agree with the objections against banUnknownProperties. The argument that $merge/$patch is powerful and allows to change the schema is an argument to use it, rather than to avoid.

Leaving aside comparisons to other ideas, what problem that can't be solved already does this proposal solve?

[handrews]

@epoberezkin did you read the linked email thread? It explains exactly how this proposal came about and how it relates to the other issues mentioned, and why it is preferable to $merge/$patch. I'm not going to rewrite it all here, although I can copy in key points if people think it will help.

Issue #15 is the one for $merge/$patch, and it has numerous points related to this.

@tadhgpearson agreed with the need for this use case in this comment.

@seagreen and @awwright cover a lot of the concerns in comments after that.

$merge/$patch also solve schema extension issue for recursive schemas

This issue is not about recursive schemas, please keep the use cases separate. If we happen to adopt $merge/$patch for other reasons, this issue can be closed as moot, but unless that happens (and I've seen no indication that it will), this issue is only about the use case I identified here.

In another issue you suggest that I try out $merge because you think it will grow on me. That's hilarious- as you can see in #15, my previous project made use of $merge well over 200 times in the published schemas alone (and more in internal schemas). This proposal is based on the experience that despite a great deal of "additionalProperties": false use, we never needed $merge to get around that.

[epoberezkin]

@handrews I've read the thread.

It does not answer the question why annotations need re-writing and why simply adding them to $ref as in my examples is not sufficient.

I think the idea of re-writing anything follows from treating $ref as equivalent to inclusion. I think it is a big mistake to treat it in this way, because, as I wrote before:

1. There are recursive cases when inclusion is impossible (and some are mutually recursive, so it's not visible from the URI that they are recursive)
2. Reference resolution is based on the source schema and when you include a fragment it won't have it's own ID, nevertheless it should resolve refs based on the ID of the source schema.

Consider this simplified example:

{
  "id": "schema1",
  "allOf": [ { "$ref": "schema2#/definitions/foo" } ]
}

{
  "id": "schema2",
  "definitions": {
    "foo": { "$ref": "#/definitions/bar" },
    "bar": { "type": "string" }
  }
}

That is a very common use case. If you treat "$ref" as inclusion then "#/definitions/bar" would mean definitions in schema1 and it won't resolve.

This example is equivalent to the test case in JSON-Schema-Test-Suite - if $ref were equivalent to inclusion this test would have fail.

There are use cases when allowing inclusion would be beneficial - it would support polymorphism (same subschemas doing different validation depending on where they are included). If we think it can be beneficial then we may consider an additional keyword that does inclusion (or maybe "$merge" without "with" would do just nicely).

[handrews]

why simply adding them to $ref as in my examples is not sufficient.

Because the only field allowed in a $ref object is $ref. It references, it doesn't merge.

[epoberezkin]

As I wrote above, you can:

{
  "title": "foo",
  "allOf": [ { "$ref": "bar" } ]
}

Why is it not sufficient?

[handrews]

There are recursive cases when inclusion is impossible (and some are mutually recursive, so it's not visible from the URI that they are recursive)

If you want to dig into the behavior of $ref, please do so in issue #66 rather than clogging up this issue with topics that are at best tangential to the proposal at hand.

If there is a specific issue where you see this as a problem with $use, please explain it in terms of $use rather than complaining about general issues with $ref. Otherwise, let's sort out $ref in #66 and then come back to this.

[handrews]

@epoberezkin : What part of "$ref cannot have other keys present because it doesn't merge, it just references" is unclear? I'm not trying to be snarky here, I really can't figure out how else to say it. If $ref behaved as you seem to want, it would more or less be $merge and we wouldn't be arguing over that proposal elsewhere (and pleas leave the $merge arguments in issue #15 ). And in any event, that's not how it behaves, and if you'd like to change its behavior please do that in #66 .

[epoberezkin]

@handrews I was just trying to understand the need for $use as it is defined here and the only explanation I can imagine is that you equate $ref to inclusion. Sure, let's continue in #66. Do you think I need to copy this comment there?

On the substance of the proposal, it simply seems unnecessary. You already can add annotations to $refs by wrapping them into allOf, which is more concise than what is proposed.

Can you explain why the existing solution is not adequate and why an additional vocabulary is needed to do the same you can do without it?

[handrews]

I have a whole section about what the existing workarounds are, with "allOf". It discusses how "allOf"'s behavior with annotations is insufficient for documentation generation, and how "default" simply doesn't work in a meaningful way.

"$ref" is irrelevant- you can't add other keys to it, and using $ref inside of "allOf" vs using a literal schema inside of "allOf" are equivalent for these purposes.

What part of my discussion of the "allOf" alternative is unclear or unconvincing?

[epoberezkin]

I will review it again.

Documentation generation is outside of spec. Whether the doc generator can understand $ref wrapped in allOf or not is the problem of the generator - extending the standard created for validation hardly seems an adequate solution to help doc generators.

"$ref" is irrelevant- you can't add other keys to it, and using $ref inside of "allOf" vs using a literal schema inside of "allOf" are equivalent for these purposes.

Again, please review the cases in the comment above (#98 (comment)) - $ref and using literal schema are not equivalent.

[handrews]

@epoberezkin

Documentation generation is outside of spec.

Um.. no. From the first paragraph of the JSON Schema Core specification:

JSON Schema is intended to define validation, documentation, hyperlink navigation, and interaction control of JSON data.

While we confirmed in #89 that documenting how multiple schemas interact in an API is out-of-scope, documenting individual schemas is explicitly in-scope. Note that I don't mean fancy full I18N/L10N end-user documentation, I just mean reading the schema and presenting it at a developer level.

I'm still not going to discuss your concerns about $ref and inclusion here as it distracts from the main proposal. If you comment on #66, I will address it there.

[epoberezkin]

The way it is written, JSON schema itself is intended as a documentation for JSON data. Deriving some other documentation format from the schema (I assume that's what you mean) is something else.

[handrews]

JSON schema itself is intended as a documentation for JSON data

Right. This is the declared purpose of title and description:

Both of these keywords can be used to decorate a user interface with information about the data produced by this user interface.

An automated system cannot disambiguate conflicting values for these keywords when trying to figure out what to present in such a user interface. That makes schemas using these keywords difficult to re-use, which is the problem being addressed here.

Additionally, completely aside from documentation, the other metadata keyword, "default", is unusable in "allOf" without disambiguation.

Please try to address these problems

• Without framing it in terms of $ref (or wait until we resolve that in Determine behavior of $ref #66 )
• Without relying on $merge/$patch (or wait until we resolve that in $merge and $patch for "schema extensions" #15 )
• Without ignoring the substance of the proposal because you don't like how I use the word "documentation". It's fine to not want to support higher-level documentation generation, but focus on the use case explicitly listed in the specification and the problems are still there