[Proposal] JSON UI Schema

[Anthropic]

EDIT: This is now being tracked at json-schema-org/json-schema-vocabularies#2 as its own project.

Please join us there for further discussion!

---

I propose a separate schema extension for the view definition in the same way hypermedia was separated into its own schema.

JSON Schema Form has a form definition that is array based for ordering, with the majority of its attributes taken from json-schema. It also has titleMap array to define labels for enum fields and can define field types (including custom field types) to use for displaying the field.

As it describes nothing more than a type name it is generator agnostic and not destined to be specific to our Angular implementation. In fact we are already working on making a plain JavaScript version and there is already a React implementation.

The main purpose of raising this here is to highlight a need, in our view, to avoid allowing view based attributes to diminish the data definition purity of the json-schema specification.

Choices and Enum Names would both bring changes to json-schema that would confuse any existing view based frameworks like ours and others while bringing little value and breaking model view separation of concerns. A full json-ui-schema would provide the missing view properties while keeping view and model separate, as they should be in our view.

In addition to a ui-schema, the same way the main json-schema has an extension for hyper media, we would also like consideration for a json-rules-schema for controller/field logic (if/when etc..) and json-flow-schema for workflow/wizard/state management keywords.

I am interested in feedback primarily on the json-ui-schema proposal. But all feedback is welcomed.

Proposal:
JSON Schema UI vocabulary
Child issues:
#252 2017-02-16

[erosb]

I welcome this proposal. It is a much better idea than trying to hack UI-specific keywords into JSON Schema.

[handrews]

I'm definitely in favor of a UI-oriented schema extension. I have no idea whether it is reasonable to include it directly in this core project or whether it should be an independent effort. Either way, it would be very helpful to be able to respond to UI-oriented proposals with "please make this proposal in the UI extension".

We would need to come up with some way of determining what annotation properties, if any, would stay in the existing three parts of JSON Schema. In my opinion, anything that is useful for documentation (like description) or instance processing (like default) should stay in the validation document.

On the other hand, enumeration names should probably go in a UI extension because description is sufficient for human-oriented documentation. The individual enum names are only needed for matching them to values in the UI.

[handrews]

@Anthropic if you would like discussion on the json-rules-schema idea, could you please file it as a separate issue? I think it is too different from json-ui-schema to discuss in the same issue without the whole thing becoming confusingly muddled.

[Anthropic]

@handrews the other extensions I consider as something to discuss after we determine how to approach json-ui-schema. I already created an org of that name just in case.

Regarding properties, I see description as being used more for what you would write in a MySQL field description rather than as a field hint like some use it now. I would like a separate hint field for that.

With json-schema-form we currently allow most json-schema fields to be overwritten in the ui-schema as it is very often the case that an online form will only display part of a data definition. As such we do not validate for fields not added to the ui-schema or removed by logic rules. Obviously a field may still be hidden. You can see our current behaviour at http://github.com/json-schema-form/angular-schema-form documentation.

If interested, I can created a repo in the org I registered and add all json-schema admin interested as admin to that also? Then create a rough draft spec based on our usage as a starting point. Also worth considering to keeping it within this org too, probably easier if it is, just means giving me admin rights on a json-ui-schema repo here to get started.

[Anthropic]

@awwright I finally made the issue for what I discussed with you previously

[awwright]

@Anthropic I've been following this, but you and everyone else seems to have already said pretty much everything I had to say, haha.

I'm very interested in building UI's from a JSON Schema, so I'd like to advance whatever would help accomplish this.

What you describe follows my experience with building user interfaces from JSON Schema pretty well, especially that even for a single JSON document, you'll want to be able to create multiple different forms to edit it in different ways. For example, editing your own comment on a blog, vs. moderating comments on a blog.

There probably should be some way to specify a "default" style, though. If I'm browsing a JSON API annotated with JSON Schema, my user-agent should still be able to show a sensible rendering of an editable JSON document.

[Anthropic]

@awwright well the way that angular-schema-form handles it, we allow users to define an x-schema-form extension object type property on the schema for users who for whatever reason cannot work with two files. But by making it a single property it still clearly maintains an identifiable separation of concerns. We do only recommend it for very small forms though.

If we rename and allow perhaps a ui-schema property to do that, it would still make it part of the ui-schema definition extending the main json-schema spec. Also a schema could always be tied to its ui-schema by a top level ref... or even an array of oneOf ui-schema ref...

[awwright]

JSON Schema Core defines a framework to allow an arbitrary JSON document to be described in any way. I imagine a UI vocabulary would just be another vocabulary following the JSON Schema paradigms, and application/schema+json media type.

Suppose we had entirely different vocabularies for validation and UI. If you want to describe a single document with both vocabularies, you could set two rel=profile links on the instance (one per vocabulary). Or just use one, and do something like:

{
    $schema: "http://json-schema.org/draft-04/schema#",
    type: "object",
    /* ... */
    allOf: [ {
        $schema: "http://json-schema.org/example/json-ui",
        fields: /* ... */,
        /* ... */
    } ]
}

[Anthropic]

@awwright our ui-schema includes all the same values in the main schema with type defining templates to use (so a enum type in the schema can be type select in the ui-schema).

The main differences to core schema in our implementation being:

• for the sake of always maintaining order the definitions are within an array
• it includes a key value pointing to the schema def (the array item can either be the key value or an object with key defined, it isn't currently a jref as at the time it was unclear how to work with arrays at the time)
• there is a titleMap for enum labels (name, value, group and any additional properties)
• some basic conditional logic processing, currently quite "Angular" but looking to change that.

There's more to it, you can see the docs here at the ASF repo

So a vocabs could work, are you thinking json-LD style? I like your example. I'm assuming that fields would be like the array in our definition, but also able to take a $ref?

Obviously I don't expect everything we do to be adopted, far from it, I know some things I don't even want to keep (like functions on the object) so I am very open to making changes! But I was thinking there's enough in our definition to get discussions started depending on where you feel is best for that to take place.

[nicklasb]

Hi @/all,
When @Anthropic, me and some more people discussed the json-rules idea, I created the json-rules org and repo here: https://github.com/json-rules/json-rules

We used gitter, but I guess we've come far enought to discuss publicly, I have started a thread here: json-rules/json-rules#1

[awwright]

@Anthropic I believe it's similar to JSON-LD; the vocabulary in use is selected by "$schema".

When I've been looking for a UI solution, there's two requirements (iirc) that I can think of, that vary slightly from what angular-schema-form is covering:

One is to assist browsing of JSON APIs that would otherwise be unstyled. So if I open up my (JSON-aware) user agent, punch in a URL of a JSON document, I can see it, I can choose to edit it it, and I can persist those changes (with a PUT request). A UI schema would be used to do things like declare data relationships ("this field points to an existing user ID") and processing ("readonly: this field is set exclusively by the server -- don't define it, or whatever it is now, don't change it, changes will be lost")

The other is that I already have an HTML form, and I want to generate a JSON document from it, and provide localized (translated) and field-specific highlights for errors.

Or maybe some hybrid. Maybe I have an HTML website, a JSON instance (e.g. representing a comment on a blog post), and I want to present it to users, editable, with some values read-only or not used at all (for example, their username, IP address, and date of submission).

It seems like angular-schema-form only handles this hybrid case right now.

We need to figure out which of these cases overlap and which are really different technologies.

The big thing is that anything going over the wire gets described by JSON Schema Validation or JSON Hyper-Schema:

• Sometimes abstract resources (a blog post) will have multiple JSON representations. A blog post might have a representation aimed for the public, and a representation for moderators that includes a revision history and IP addresses.
• Instances can describe how they relate to other instances: read only (set only by the server), relationship to another resource that must exist, etc.
• form fields might choose to render themselves differently depending on the validation restrictions (e.g. automatically use <input type="number"/> for type:"number" properties).
• If the value we want to send would be invalid, we can try to trace the invalid value back to a particular form-field and display an error there.

(Some of these features might not be well-defined yet, but they would be good fits for JSON Hyper-Schema.)

So even without a dedicated UI vocabulary, there's still a lot of information for a form generation toolkit to work off of.

What other features do we need, can we come up with a list?

[Anthropic]

@awwright do you want to pick a spot in the wiki for planning to get a start on a requirement/use case list? It could get rather difficult to track if we start posting requirements and use cases in numerous posts on this issue.

I totally agree that any data transmission should be defined by those two schema, that's why when I implemented remote enum option retrieval I used the hyper-media link with a rel of 'options'.

At the same time there is obviously, based on other issues raised, a consideration to re-evaluate the hypermedia schema which could go hand-in-hand with any requirements garnered from a considered requirements list for a ui-schema.

[mmc41]

Good initiative. I am also interested in this. I am however, a bit unsure about how json schema extensibility works (what is possible / what is not) when using a schema extension for defining the view. F.x. Can existing json schema elements be extended in this way or do we have to create new, more powerful alternatives ?

[epoberezkin]

@Anthropic we've been also experimenting with different form definition approaches: in https://github.com/milojs/milo-ui and in https://github.com/jasoniangreen/iso-form

I think it would also need to support grouping and, given than grouping leads to a different view structure from the data structure, the way to map UI elements to data pointers (e.g. dataPath: '/customer/name' in UI element for the first name).

Also, in general case, it's not possible to derive the requirements to a given field from JSON-Schema, so you'll either have to require to use a subset of JSON-schema, similar to what swagger is doing, or to derive JSON-schema from form definition.

I prefer the latter approach, restricting JSON schema is my main objection against swagger.

[Anthropic]

@epoberezkin our approach is working for the most part, we have thousands of users already using the tool and the requests for new features are fairly stable so we tend to know what the users need the most as they keep reminding us we haven't done it yet :)

The main thing we do is merge a schema with a ui-schema in processing to then generate the form. We do not validate items not found in the form definition as we generally find users do not want to validate fields if they are not shown for a selection of users for example. They can still have hidden fields to force such validations. We are migrating our core to a stand alone library which you may find of use if you take a look at it.

The current idea on this proposal after suggestion from @awwright is that we start creating a list of things that need to be managed/supported to then start defining requirements. I'm not sure how @awwright wants me to do that but hopefully he can let me know now so I can get started in a format that will be of use to him :)
ps. if none of that made sense it is 1:15am, forgive me.

[epoberezkin]

I understand. Your approach approach is definitely useful. I just think it is not comprehensive enough to be standardised as is - it leaves many real life scenarios not covered. Flat forms, where the form structure matches data structure, while quite common, don't cover a sufficient share of use cases.

We do not validate items not found in the form definition as we generally find users do not want to validate fields if they are not shown for a selection of users for example.

I said the opposite - for a general JSON-schema it may be impossible to validate item that IS found in the form definition:

• it may be dependent on other field(s)
• it may be within allOf/etc - so while it is possible, it may be difficult to extract schema for a single field.

Given that your library only supports a subset of possible JSON schemas, it is useful to understand which subset it supports.

[Anthropic]

@epoberezkin It's ok, I don't consider it ready to standardise as-is, I only thought of it as a minimum starting point to be expanded and added to... and changed.

I think you know already but to clarify for others, when I said form definition, that is our ui-schema, currently the main feature missing is $ref support, it is possible to make anyOf/oneOf work via an add-on, but it isn't complete as part of the main lib either as we are trying to figure out how to deal with those scenario as they don't lend themselves to easy generation as they are, I'd like to see changes in that area.

As we combine the schema and ui-schema definition we have a contained validation, but we would only validate a parent oneOf/anyOf if it was included in the form definition. So you have choice over how much of the schema you want to validate against. The only issue we see coming up is if there is inclusion of $data references that would cause issues if the related value was not in the form. But at the same time you can't defend against poor implementations, in my opinion it is up to the designer to determine that they include what is required.

[the-t-in-rtf]

Is there a draft of the proposed extension yet?

[Anthropic]

@the-t-in-rtf not yet. We don't have a complete requirements list yet.

edited: changed my mind on where the rest of the comment should have been posted...

[nemesifier]

Can we have a summary about the status of this idea? Is it stalled or is someone who has started working on it?

[Anthropic]

@nemesisdesign, the main focus is on the core and hyperschema draft updates. I am in email conversation with members, but it is slow while they must focus on their current task. If you have goals or interest in the outcome I'm happy to have a discussion through Gitter about it anytime :)

[nemesifier]

I'm the maintainer of a few OpenWISP modules that use JSON-Schema intensively (eg: netjsonconfig and django-netjsonconfig) and I also maintain the specification of the NetJSON data interchange format.

The interest in OpenWISP and NetJSON is growing rapidly, OpenWISP has also been accepted to the Google Summer of Code 2017. Yes I'm interested in contributing, I just need to organize myself a bit better. Probably when the GSOC ends I will have more time to dedicate to this idea. But in OpenWISP we are using a UI that is automatically generated from a JSON-schema and is working pretty well.

[handrews]

@Anthropic we're getting close to releasing the new drafts and there's really just the one big issue that we're finalizing, so now's probably a good time to get this going here, particularly if there is increasing interest :-)

[nemesifier]

Here's the main attributes that we are using heavily for UI generation, some of these are custom implementations of json-editor:

• title
• description
• default: default value pre-filled in the UI
• enum_titles - this one is custom, although I would prefer something like my old proposal choices, but this is better than nothing
• {"format": "checkbox"}, this thing tells the UI generator to render as checkbox, it would be better to use a new attribute for this task, eg: widget
• propertyOrder: this specifies the order of fields

[handrews]

It's occurred to me recently while working on doca (a documentation generation system) that there are quite a few features that would be shared by UI and API doc vocabularies.

For instance propertyOrder is more or less the solution @Relequestual is using for cloudflare/doca#28. And as much as I'd like doca to make a best effort to respect the file order and not have to re-specify all the property names, an interoperable solution would be better.

For enum_titles I still find the oneOf solution pretty compelling now that const is in the spec:

{
    "oneOf": [
        {"const": "foo", "title": "Pick Foo"},
        {"const": "bar", "title": "Pick Bar"},
        {"const": "whatever", "title": "Don't Care"}
    ]
}

This is an easy pattern to recognize, and it also allows associating other keywords like description and examples without having to come up with even more enum-enhancing keywords.

[nemesifier]

Honestly, if I had an alternative I wouldn't use that construct as an alternative to enum_titles. It's much more complicated and it has the negative side of being hard to read for someone who doesn't know JSON-Schema, and UI designers will much likely have to deal with UI schemas and I see this as a very likely adoption issue.

[handrews]

The advantage is that it actually works for both UI and validation. Being able to re-use existing keywords makes it both more flexible and more readable than the positional tuples for "choices", and avoids the duplication required by "enumNames".

Anyway, we can debate that specific issue when we get into the details. One solution might just be an alias for "oneOf" that is friendlier to UI usage so that we don't add much more implementation work for something that should already be present in implementations.

[Anthropic]

We use titleMaps which as you can see below replicate the exact structure of @handrews proposal with only a name change, the things we add though are allowing a group property and data related attributes so that a selection can be used to provide further information. So if the value needs to be the same as the name but a related select needs to be populated with a call using the selection's id then that is available.

{
    "titleMaps": [
        {"value": "foo", "name": "Pick Foo", "group": "Important Foo", "id": 4},
        {"value": "bar", "name": "Pick Bar", "group": "Important Foo", "id": 7},
        {"value": "whatever", "name": "Don't Care", "group": "Additional Options", "id": 42}
    ]
}

So ui:group and ui:data could be in a ui vocabulary to provide the ability to group related items in any context and also provide access to properties for related items as needed.

[Anthropic]

@nemesisdesign we currently use type in our ui-schema for widget, but I want to change it to ui:widget, so I agree that would be easier than using format which I think is preferred for specifying the data format than the ui widget.

[brettz9]

Just adding mention of i18n (internationalization) / l10n (localization) here so this issue will turn up on searches for it (and in case participants haven't considered its relevance).

[handrews]

Thanks, @brettz9, it has come up but searchability is good.

[brettz9]

With UI concerns now targeted to their own spec, I'd like to restate here for i18n that I think it is best for portability between consuming projects, for easier offline support, etc., to allow locales to be expressed in a standard way without need for server-side interpolations (allowing clients to substitute parameters that determine the choice of locale).

To i18nize the example from @handrews , something like the following could allow the documents to be distributed across project with internationalization information intact and consumable or switchable as desired:

{
    "definitions": {
        "locales": {
            "en-US": {
                "fooset": [
                    {"const": "foo", "title": "Pick Foo"},
                    {"const": "bar", "title": "Pick Bar"},
                    {"const": "whatever", "title": "Don't Care"}
                ]
            }
        }
    },
    "oneOf": {"$ref": "#/definitions/locales/~{locale}~/fooset"}
}

The locale variable could be substituted dynamically (including in JavaScript) after determining the user locale.

[brettz9]

Regarding enum-enhancing keywords, for UI purposes, I'd like to see a means of indicating an equivalence between two sets of enums beyond l10n. While grouping and data attributes could be used to implement this, I don't think it would be ideal for making clear the nature of the relationship (equivalence).

For example, a JSON schema of a tabular version of the Bible (with columns for book, chapter, verse, text, annotations, etc.) might wish to express an equivalence for its "book" column between a long name ("Genesis", "Exodus", etc.) and well-known abbreviations ("Gen.", "Ex.", etc.). (Or, with the likes of the Qur'an, frequently-referenced Surah numbers could be mapped to Surah names, e.g., "1" or "The Opening".) More than one set of mappings could be desirable (e.g., those belonging to different sets, e.g., long, short, and medium-length: "Exodus", "Ex." or "Exod." OR those which happen to have idiosyncratic alternatives, e.g., "The Family of 'Imran" or "The House of 'Imran").

A generic book browsing tool could introspect on these equivalences and provide a choice of pull-downs and/or a text box which parses either set of abbreviations for a range of verses/paragraphs (e.g., "Gen 1:5-Exodous 2:7). Likewise, could the user opt for the output to display one set of aliases or another (e.g., to selectively list "Gen.", "Genesis", "Book 1", etc. alongside the chosen selection of verses). User agents could differ in how to present this, but they would have a standard way of detecting that alternative sets of equivalent labels were available.

[romainpiel]

(Sorry I might be deviating from the original topic)

• I am wondering what would be the scope for a JSON UI Schema. Is the purpose to build a JSON specification for web UIs? Or should it be generic enough to also support web and native mobile UIs?
• This thread (and [Proposal] JSON UI Schema: define UI behaviours for all current elements #252) only mention form UIs. Are there also motivations to consider more complex UIs with templates and constraints?

A bit of background aruond me and my motivations: I'm an Android developer and my day to day job involves converting data structures defined in a JSON payload to view models that could be easily bound to Android views. I'm currently doing some research for my personal interest to make my Android apps a bit more dumb and have our backend services expose data structure that are closer to view models, ready to be bound to the views.
This sounds all great and I could just create my own JSON spec. But as I was starting to face issues like the one described in this thread, I thought maybe there could already be "UI-focused" specifications out there.
If I'm not deviating too much and you folks need a pair of eyes from a mobile dev, I'd be happy to help.

[Anthropic]

@romainpiel feel free to contribute to the JSON Schema UI vocabulary proposal based on this issue. I consider it to be more ui than form based myself, but I need more people to provide input like you have done for us to get some further traction on defining a solution and the limits of said solution.

[handrews]

@romainpiel I agree with @Anthropic, who is the point person on this topic. We would really like to get critical mass for discussions so please do comment over on the vocabulary project!

[romainpiel]

@Anthropic @handrews great, will do!

[handrews]

This is now being tracked at json-schema-org/json-schema-vocabularies#2 as its own project.

Please join us there for further discussion!