Form vocabulary special interest group

[yaffol]

Hi!

Do you use JSON Schema to make forms?
Do you run into issues where the schema is cumbersome or not quite suited to your use case?
Do you find yourself working round the schema (perhaps with a side schema of your own) and wish there was more flexibilty and customisation in the core?

We are creating a Special Interest Group around the creation of a custom vocabulary - supported and guided by JSON Schema org - specifically for the generation and validation of forms.

I am hoping to gather here, via comment responses, a laundry-list of all the common (and not so common) limitations and difficulties that people run into when using JSON Schema to generate and validate forms, how you're working round it, and how you'd like it to be better - if you could extend the schema to suit your use cases.

Please do comment below, +1 things you've also experienced, and we can start gathering the raw materials that will go into building a custom JSON Schema form vocabulary.

I'm very happy to take calls or chat direct about it on slack or over email - you can find me on the JSON Schema slack community, or as @yaffol on twitter.

Thank you for your input!

[mwadams]

Excellent! I'm really glad to see that this effort is getting underway.

For me the most important principle to remember is simplicity and incrementality of use. It is super important that more complex scenarios are not implemented at the expense of the simplicity of expression of common scenarios.

In particular, I'm thinking of the common trap of clever generalizations that reduce the complexity of the vocabulary at the expense of simplicity of semantic expression.

[yaffol]

Hey @mwadams! I think this is a really relevant point. There a need to retain the - elegance - of using JSON Schema while finding ways to improve the UX/DX when applying it to forms.

It's going to be really interesting to see the real-world use cases coming out of form development in the wild, and finding a common path to improving them.

Thanks for your input!

[tobilg]

https://rjsf-team.github.io/react-jsonschema-form/ might contain some good ideas

[yaffol]

Hey @tobilg! Yes, definitely. I've reached out to them on their Discord server and hopefully they'll come back with some interesting use cases.

If anyone has any contacts directly within libraries or organisation that make heavy use of JSON Schema for form generation/validation, please do sign-post them here!

[epicfaace]

Hi 👋 , I help maintain react-jsonschema-form. Thanks for making this special interest group!

One thing I can think of off the bat is that the default keyword is really important for rjsf, while there isn't really much guidance for how to handle it in the JSON Schema spec. We created our own, kind of complex, way to handle defaults (see https://react-jsonschema-form.readthedocs.io/en/latest/advanced-customization/internals/#handling-of-schema-defaults), and it might be good to somehow standardize this behavior.

[epicfaace]

We also use a separate uiSchema for all ui customizations that has keys starting with ui:. It would be ideal if we could somehow allow users to specify things in the uiSchema within the JSON Schema: https://react-jsonschema-form.readthedocs.io/en/latest/api-reference/uiSchema/

[solt9029]

I love uiSchema😉

[yaffol]

Hey @epicfaace! Thanks for your input! I've used react-jsonschema-form for a prototype and it was really useful.
It's interesting how a couple of libraries have taken on the idea of using a UISchema - both react-jsonchema-form and JSON Forms do this. Vuetify JSONSchema Forms differs in that it takes the approach of using x-* custom keywords to provide UI hinting.
It will be interesting to see whether we can come up with something that meets both of these use cases in the middle - something that allows you to both specify the UI, layout, specifically, but also brings simplicity to it. Or at least standardisation!
I'm interested in the HAL Forms concept as promoted by @mattbishop - I've only skimmed it thus far but if anything it provides a quite complete implementation of a spec that is trying to do something in this direction, so it's a useful counterpoint at the very least, if not something more concrete (I just don't know at this point)!

[kevin-mitchell]

I'm not sure if this is in the spirit of the type of feedback you're looking for, but it does deal with form validation (and validation in general). The fact that the various implementations have different levels of support for regex as noted here...

The regular expression syntax used is from JavaScript (ECMA 262, specifically). However, that complete syntax is not widely supported, therefore it is recommended that you stick to the subset of that syntax described below.

...can make certain fiddly validation difficult to do. I realize the spec already says ECMA 262, but if anything could be done in this effort to improve regular ol' regex support I think it'd be a really nice improvement in day-to-day usability!

[Relequestual]

Support for the specific regular expression (ECMA 262) is not an absolute requirement, but a "SHOULD".
Some implementations will use whatever the programming language provides, but some implementations will go as far as to use a library which provides ECMA 262 regex in their programming langague when it's not something supported out of the box.

What's "regular ol' regex"?

Are there any specific issues you're facing?

[ryangalamb]

I'd love a keyword that's like "default", but references an instance value. Basically, "If this instance location has a value, use that as the default."

That's the main thing I've needed in my work that JSON Schema doesn't support. I implemented it by borrowing ideas from JSON Hyper-Schema, but there's nothing that works "out of the box" for it. Happy to discuss my experience/observations if people are interested in exploring that functionality.

[yaffol]

Hi @rjmill, thanks for the input! This rings a bell for me in terms of the way AJV and ajv-errors use JSON Pointers to refer to elements in the schema. So that schema values can get inserted into templates for error messages, IIRC. Very happy to hear your observations - please do add them in here!

[albanm]

Maintainer of @koumoul/vjsf here. This is very interesting ! I didn't take the time to look into json-schema custom vocabularies before that, but I think I get the point, and I am willing to participate in the effort.

Vjsf relies heavily on annotations directly on the schema itself (x-* custom keywords), my understanding is that these annotations could be more formally described using one or more json-schema vocabularies. I imagine 2 vocabularies at least, one specific to vjsf for quick work and one shared where the most matured and generic stuff would go.

A few things that come to my mind that need to be expressed when rendering a schema as a form:

• hiding a property (I use x-display=hidden or an option hideReadOnly alongside the standard readOnly)
• choosing an input component (I use x-display=radio/checkbox/switch/file/markdown/tabs/stepper/vertical-stepper/expansion-panels/color-picker)
• adding textual or markup content before and after properties (I use x-slots={before: ..., after: ...})

I will follow the discussion here, and I will try to formalize some of vjsf existing semantics into a vocabulary as a starting point.

[Relequestual]

Welcome @albanm! Feel free to shout if you have any questions on Vocabularies.

[yaffol]

Hi @albanm! Many thanks for your input, your use of x-* keywords is interesting and provides an existing implementation to look at, which is very helpful.
A formal vjsf vocabulary will be enlightening.

[mattbishop]

Please consider copying the concepts from HAL Forms (https://github.com/mamund/hal-forms) as that project has spent considerable effort figuring out many of the same problems this project might. In fact, you might be able to create a vocabulary that HAL forms adopts, similar to how OpenAPI now defers to JSON Schema instead of forking it for their own spec.

[yaffol]

This is really interesting. I've only skimmed it but it looks like it provides a pretty complete spec for doing something very like what might be proposed here, so it merits a close look on those grounds alone.
Do you have experience using it in earnest?

[yaffol]

Thanks for all the feedback so far, it's great to hear such a wide range of views already, I really appreciate it.

I've had some direct messages with suggestions which I wanted to include here for completeness:

"I've always felt that essentially rather than a "form" framework, what is needed is a template framework with form descriptive capabilities. I found a lot of the issues my team had were in getting random bits of content required within the form into any framework specifically designed for forms alone. Like you often see in commercial forms things like banners, info sections and texts. react-jsonschema-form for example doesn't make that really possible without making your schema specific to your output instead of the data."

"My main issues were how to handle the generation of validation rules, I concluded that there could be a need to have an intermediate representation (IR) and that it may be worth standardising what that format would take and how it should handle jsonschema rules which don't often match up.
As in when you only want to validate part of a form you only want to validate that and not the entire form every time, I was considering working with ajv or djv to handle generation of field level subset of validations so an app doesn't slow to a crawl validating the whole thing every time on large forms."

"I was working to generate complex forms in a corporate and I realised the way they did validations was having a completely separate rule set that defined which fields to watch and then ran validations off that, I felt the IR could do something similar."

[yaffol]

And also to note, there is now a #vocab-forms channel over on the JSON Schema Slack for more direct discussion, please do go ahead and use that if you prefer. (Thanks @Relequestual !)

[Relequestual]

Welcome!
Please use the Slack for chatter and record key discussions here.
We DO NOT have access to Slack history, so consider those discussions ephemeral!

[epicfaace]

Also, the folks at @department-of-veterans-affairs (such as @jbalboni) might be interested in this discussion. @dmethvin-gov had a good thread a while back (usds/us-forms-system#141) about the problems with uiSchema.

[yaffol]

Thanks @epicfaace! Yes I noticed they were users of RJSF. It would be really good to get input from users of form libraries as well as maintainers of them. If anyone knows of other users of form libraries who might be interested in getting involved, please signpost them here!

[dmethvin-gov]

There's some good prior art at https://github.com/formio/formio.js, which is at the heart of the commercial form.io product. You can see that whole thing working here, it's pretty impressive: https://formio.github.io/formio.js/app/sandbox

[yaffol]

Thanks for the input @dmethvin-gov! Formio.js is a really interesting reference. It looks - from a quick skim - that they use a custom JSON schema (vs a published version of 'JSON Schema')? The builder is certainly impressive and they have clearly put a lot of work into making a product that is simple and effective to use, but also very customisable. This along with HAL are required reading for me!

[sdirix]

Hi! I'm the project lead of JSON Forms. I would be interested in the scope and goal of this initiative. Especially whether the eventual vocabulary will describe forms in great detail or restrict itself to solve the common problems of form generators / renderers.

I'll just bring up some topics which might be interesting.

Layout

One question is if and how the general layout of the form can be specified or whether it has to follow the structure of the data.

In JSON Forms we use a UI Schema concept to fully decouple the UI from the data schema. The UI Schema describes the layout and points to the JSON Schema for more information where necessary, e.g.

Example:

{
  "type": "VerticalLayout",
  "elements": [
    {
      "type": "Control",
      "scope": "#/properties/name",
    }
  ]
}

Textual descriptions and Internationalization

JSON Schema offers title and description to describe a respective property. Forms however often have three different parts: A label, a description and a placeholder.

• Can this be formalized?
• Some forms render longer descriptions in markdown. Do we also want to support this in some way?

For internationalization purposes often the respective JSON Schema is exchanged with a localized one. This is rather inefficient and requires some special preprocessing. Another approach is to store localization keys directly in the JSON Schema title and/or description which are then transformed by a localization framework of choice.

• Maybe we can define a i18n property which can then for example be prefixed to label, description and placeholder?

Dependent values

Often there are special requirements which can't be properly expressed via JSON Schema. For example when a JSON Schema contains an array of persons and there is also an "enum" which shall exactly represent one of the person instances. Or when it shall be validated that a certain number is larger than another number.

Some values might also be derived from one or more other values.

Visibility and Rules

JSON Schema already supports readOnly. A similar visible property would be useful. In case we formalize layouts it would be interesting whether visible and readOnly can be applied to them and what happens when parent and child properties are specified differently.

Often parts of the form only shall become visible or enabled (i.e. not readOnly) when specific values were entered. In JSON Forms we have specific rules in the UI Schema for this (similar to if/then in JSON Schema). Might be interesting to formalize this. If yes the interaction with the already existing readOnly and maybe visible needs to be specified too.

Validation

JSON Schema validation can be pretty complex. As a consequence often the whole data object needs to be validated on every change although many schemas only have some "required" and "pattern" validations. Would be interesting to allow for some restricted mode with local-only validation. This would then allow validation libraries like AJV to offer much more efficient validation modes.

Combinators and conditional schemas

While oneOf, anyOf and if/then/else make sense from a validation standpoint they are very complex to support in forms.

For example oneOf might be rendered as tabs over all options, but often it can't even be properly decided which of the tabs shall be the active one.

Default values, null and const types

It might be interesting to define some default behavior regarding default values, e.g. are they added by the rendering process? Only initially or whenever the value is undefined or null? Is an explicit undefined different? Shall null values be replaced? Store null, undefined or empty string ('') when a text input is empty? What to do with a standalone const property in the JSON Schema? What about inputs for nested objects which don't exist yet, is the object shell automatically created via the rendering process or only when one of the nested inputs is initially changed?

Enum labels and other patterns

At the moment we use a oneOf construct with const and title to allow for non-string enums as well as differently titled enums, e.g.

oneOf: [
  {
    "const": "my-first-enum",
    "title": "1st enum"
  },
  {
    "const": { foo: "bar" },
    "title": "Other enum"
  },
]

I was wondering whether a more straightforward concept could be specified.

In general it might be interesting to define a list of "standard" patterns (e.g. how a multiple-choice enum is specified) so that the JSON Schema doesn't need to be tweaked when switching between frameworks.

Arrays

When rendering large arrays it often makes sense to render them in some collapsible or reduced way. For these cases it would be nice if one of the contained properties could be marked as the "label" property. For example to then be rendered in the header of a collapsible group or in the overview list.

Steppers

Many forms are wizard-like with different steps and transitional conditions. Would be interesting whether this can or should be formalized.

[jdesrosiers]

Great list!

I encourage this group to keep an eye out for things that don't only apply to forms and try to decouple them. For example, i18n could potentially be useful for things other than form building. Same goes for assertions that can't currently be expressed in JSON Schema.

As for introducing a new enum keyword, I'd advise against it. A big part of the value you get from JSON Schema is that you can use it in many different contexts. You probably want to use the same schema to validate on the backend, but if you introduce new validation keywords, the backend validator probably doesn't understand the form-building vocabulary and those assertions will get ignored.

[Relequestual]

Thanks for showing up and sharing @sdirix!
You're welcome to help shape the scope. What do you think is most attainable?

There's always a path which ends up with maybe even a few new vocabularies, one to cover basics, one more advanced, and one some other use case.

[yaffol]

Thanks for the comprehensive response @sdirix! Those are some very interesting and relevant suggestions. In terms of the scope, I'd echo what @Relequestual says - definition og the scope is very much up for discussion right now!

[thclark]

Hi Folks, 👀 watching this with keen interest!

[yaffol]

Hey @thclark! Thanks for your interest! If you have any experience of difficulty with JSON Schema when building a form please do share! Also there is our #vocab-forms Slack channel for ephemeral discussion. See you there!

[yaffol]

It's great to get a wide range of examples from a multitude of libraries so early on - thanks everyone for your input so far.

In terms of the scope of this initiative, that is a very good question, and exactly what this information-gathering phase is about. I can see that there might be a number of somewhat competing sets of requirements, and as they become clearer, it will be easier to see what possible scopes there are and how they could be implemented.

I feel it would be useful to also clarify how I see my role in all this - I have come to this initiative and JSON Schema in general having done quite a lot of research on JSON Schema form libraries while creating a new UI project - and it became clear in that process that there are areas which are unclear, or create work which has to be duplicated, when using a JSON Schema to create and validate a form. My intention is to facilitate discussion of the issues and potential solutions, and do the 'admin' of shepherding this process onwards. And as I become more familiar with the problem domain, get more involved in the actual iteration on possible solutions.

But a lot of the actual detail of the problems are new to me - I only had an idea that they existed, which is why this clarity of feedback is very useful.

What I would suggest now is a final call for further input - anything that has been really holding you up or is always a pain to implement - then please put it down here!

Then I propose to create a collaborative document, ex: a Google Doc, where I'll collate all the suggestions so far. Then those suggestions can be commented on and refined in-place, and we can move to a situation where we have a clearer set of (possibly competing) requirements. Thereafter, it will be hopefully more possible to identify a course, or set of courses, we can follow to refine those suggestions into some draft spec(s).

If anyone knows of any formalised process we can follow that would help here (I'm thinking RFC style, but don't want to introduce more bureaucratic overhead than is necessary at this stage) I'd be very interested to hear about it.

Thanks again for all the input, really appreciated!

[yaffol]

It's been a while, but I've collated all the responses here into a document.

I've paraphrased and condensed where possibly, appologies if I've mangled your meaning - please submit any corrections in the comments.

The document is open to suggestions so please do add or alter anything you think needs improvement.

[aghosh-evertz]

Hi All,
I have tried to go through the doc to find, if anything like this is supported:

"Support for JSONSchema '"type" as "array", when we have a custom format added to it through code, and the field values are recieved through an async service."

Right now, if I have a schema like this:

JSONSchema:

{
"$schema": "http://json-schema.org/draft-07/schema",
"type": "object",
"title": "Incident Filter",
"definitions": {
"channelIds": {
"type": "array",
"title": "Channel names",
"description": "Channels which will be monitored for incidents"
}
},
"properties": {
"channelIds": {
"$id": "#/properties/channelIds",
"$ref": "#/definitions/channelIds"
}
}
}

The format is getting added through a separate patch file like this:

import { JSONSchema7 } from 'json-schema';

import { JsonSchemaPatches } from '@evutils/formly-utils';

export const updateSchemaFields: JsonSchemaPatches = {
channelIds: updateChannelFieldConfig,
};

function updateChannelFieldConfig(json: JSONSchema7): JSONSchema7 {
return {
...json,
format: 'channel-custom-format',
};
}

**The values for this channel-field is coming from the service, which is later added, when available (async).

It doesn't work and throws random error. How are we supposed to work with a schema field having 'format',whose type is "array", but values are not instantly available??**