Yaml source of truth for JSON schema files

[jack-berg]

Alternative solution for #398 based off conversation in #407 and #400

Idea:

• Define JSON schema files in YAML instead of JSON, for better multi line editing
• This will allow us to merge meta_schema_types.yaml into the schema files so everything is in one place
• Add build tooling to convert YAML source schema files to JSON output schema files, and adjust build validation to work against the JSON files
• On release, publish the JSON version of the schema files

If we like this idea I would follow up with a PR to merge meta_schema_types.yaml into the source YAML files, and update all project tooling, documentation, etc.

[jack-berg]

Marking ready for review after seeing #400 and #396 closed

[jack-berg]

Fun fact - you can tell its a strict refactoring of build tooling because schema-docs.md basically doesn't change besides this trivial bit pointing to the new source location.

[marcalff]

Ok in principle.

The concern is to avoid merge collisions if another PR changes the schema in the mean time, but the generated schema-docs.md file serves as a verification for that, so it looks safe.

[jack-berg]

I realized that maintaining "pre compiled" source for JSON schema definitions complicates the picture for $refs - do refs point at the .yaml or .json (which doesn't exist until you run make compile-schema?

Solved this by adjusting all $refs to point .yaml for a more intuitive editing experience, and replacing .yaml to .json as a part of the process of writing out the JSON schema files.

At the same time, I realized this is a good place to force standardization for the $id and $schema properties each file needs to contain. (see below)

But this got me thinking and worrying (once again) about $id and resolving $refs for tooling. Once solution would be to publish a single .json file containing all of the types in the schema, eliminating the need to do $ref resolution. A single file is also more convenient than a directory, since there's no need to zip it (and unzip it after download). Of course the downside is that one big file with all the types in it is cumbersome to maintain, but.. 1. its not that bad 2. we can improve things with tooling (i.e. lexographical sorting so types are where you expect them).