Publish on schemastore.org

[aslakhellesoy]

The https://github.com/SchemaStore/schemastore project is a repository of JSON Schemas. It has wide industry and a vibrant community of tools around it.

We should publish the schema here. Before we do, let’s discuss naming and versioning schemes.

Publishing should be automated, perhaps a GitHub Action that creates a PR when a tag is pushed.

This is also related to #1 where I suggest we remove the Ruby code. For testing that the schemas accept good json documents and reject bad ones, I suggest we try one of these:

• https://github.com/korzio/djv
• https://github.com/davishmcclurg/json_schemer
• https://github.com/neilpa/yajsv
• https://github.com/json-schema-everywhere/pajv
• https://github.com/ajv-validator/ajv-cli

We should pick the one that gives the best errors.

[TimotheeVille]

We're now testing with json_schemer, it provides error messages that are more or less readable for human. But we can't expect more from a parser right? We're using it also in other products and customers seems satisfied as far as I know.

Concerning the contribution to the schemastore, it sounds great. We don't have to publish anything on the store, we can just add to the catalog the link to the schema in this repo. What do you think @aslakhellesoy?

[aslakhellesoy]

I'm not even sure we need a versioning scheme actually.

I think we do.

Your schema currently fails to reject documents with missing properties. It will actually validate this:

[{}]

That’s not very useful at all. A consumer of the document would not get any reassurances from the schema regarding the presence of properties. As a result, you have to check each the presence of each property in the code. This should be the job of the schema!

You should make most of the properties in the schema required. That’s the whole point of JSON Schema - detect when a document doesn’t have the data you expect.

We need tests for this. In fact, most of the tests should be about what the schema rejects, not about what it accepts.

Once you do this, you will find that the schema will accept documents from certain Cucumber implementations/versions and reject others. At this point we have to make different schemas for each Cucumber implementation and version range.

Different versions and implementations of Cucumber (including some other tools) produce JSON documents that are structured differently. The format was never specified in a schema, and has diverged in different implementations and versions. @vincent-psarga knows all about this.

So you may have to validate a document against multiple schemas until you find one that matches. Then you can say «this one looks like it came from Cucumber-JVM version range 3.2 - 4.5.

With this knowledge, you can process the documents with much more confidence.

[TimotheeVille]

We're now rejecting more invalid JSON.
See : e56e003, ed610e3, 660a4d5 and 1e21ddd

[TimotheeVille]

So you may have to validate a document against multiple schemas until you find one that matches. Then you can say «this one looks like it came from Cucumber-JVM version range 3.2 - 4.5.

In our product, we don't need to know which cucumber framework have produced the report. We just want to make sure we will be able to extract data we need.
What I want to say after validating a report is 'Okay, I can fetch step result from this json', no matter the framework.