Improve validation on Stryker options.

[nicojs]

Right now there is no standardized and complete way (for both plugins as well as the core) to do the validation.

For example, this config file will result in no errors at the moment:

{
  "concurrency": 2,
  "reporter": ["clear-text"]
  "thresholds": {
    "error": 80
   }
}

Solution

I want to allow use JSON Schema to validate. We can use AJV as validator. As for human-readable error messages we can use AJV errors. As a bonus, we could let AJV assign defaults.

Plugins could contribute parts of the schema dynamically by adding a schema/stryker-plugin.json (name pending) file in their respective plugins.

Verifying that no additional properties are present can be done by running the schema validation with additionalProperties: false. They should be logged as warning. We should allow JSON comments by not warning about settings with the postfix "comment". We could allow a --noWarnOnAdditionalConfiguration (name pending) to not warn about additional settings (otherwise people might get frustrated).

[simondel]

Sounds good!

[bartekleon]

I am so happy now that we made this schema :D

[nicojs]

A start has been made here: #2123 . It will validate the core schema, but not yet warn about additional properties. It also won't validate plugins yet, but it's a start.

Feel free to review 😅