Semantic Versioning of canonical data.

[rbasso]

As questioned in #671, we still don't have defined how to version the canonical-data.json files in the new schema. I think we should strive to get something like SemVer.

---

Edit: Below we well keep updated the proposed semantic versioning resulting from the discussion.

Current proposal

Adapting from SemVer 2.0.0 ...

Given a version number MAJOR.MINOR.PATCH, increment the:

• MAJOR version when you make incompatible API changes

Only changes in the test suite in a fundamentally incompatible way should
change the MAJOR version:

• A new "property" (test type).
• Renaming a "property".
• Insertion, deletion or renaming of keys in the test data object.
• Changing the type of one of the test data keys.

MAJOR changes would break even well-behaved test generators, that are not affected by test grouping/nesting.

• MINOR version when you add functionality in a backwards-compatible manner, and

Non-breaking changes that alter the meaning of the test suite, making previously
passing solutions possibly fail, or failing solutions fail at a different spot, should bump the MINOR version:

• Adding or deleting test cases.
• Changing the test cases inputs and/or outputs.
• Changing the test cases ordering.

MINOR changes would never break well-designed test generators, because the test-generation logic remains exactly the same.

• PATCH version when you make backwards-compatible bug fixes.

Anything else that do not change the meaning of the tests should bump the
PATCH version:

• Regrouping/"Renesting" test cases without changing test cases ordering.
• Changing descriptions or comments.
• Changing keys' ordering or formatting (would result in an equivalent JSON file).

[behrtam]

Maybe ..

Given a version number MAJOR.MINOR.PATCH, increment the:
MAJOR version when you add new test cases or change input/output values,
MINOR version when you change descriptions or other descriptive fields, and
PATCH version when you fix small typos or other trivial things.

[rbasso]

The MAJOR number could be reserved for changes in the structure of the tests, things that would probably break any well designed test generator.

[behrtam]

Good point. Than we move the new test cases and input/output changes to MINOR and put the cosmetic changes (rewrite/typos/etc) of descriptive fields into PATCH .

[rbasso]

I'm writing a proposal so that we can discuss the details. There are a lot of corner cases...

[rbasso]

This is what I got until now, @behrtam . I tried do adapt SemVer to the canonical data.:

Given a version number MAJOR.MINOR.PATCH, increment the:

• MAJOR version when you make incompatible API changes

Only changes in the test suite in a fundamentally incompatible way should
change the MAJOR version:

• A new "property" to be tested.
• Insertion, deletion or renaming of keys in the test data object.
• Changing the type of one of the test data keys.

These are the changes that would break even a well behaved test generator.

• MINOR version when you add functionality in a backwards-compatible manner, and

Non-breaking changes that alter the meaning of the tests, making previously
passing solutions possibly fail, should bump the MINOR version:

• Adding or deleting test cases.
• Changing the test cases inputs and/or outputs.

• PATCH version when you make backwards-compatible bug fixes.

Anything else that do not change the meaning of the tests should bump the
PATCH version:

• Reordering, regrouping, renesting test cases. (not sure if this should be MINOR)
• Changing descriptions and comments.
• Changind key orders and formatting (would result in an equivalent JSON file).

What do you think about it?

[rbasso]

I just updated the proposal, @behrtam, because test cases' ordering seems to be important in Exercism, so reordering would change the meaning of the test suite, implying a bump in the MINOR version.

[petertseng]

I wondered if I might have summarised it as:

• Major: You might need to rewrite your test generator.
• Minor: You will definitely not need to rewrite your test generator. You probably want to rerun your test generator though.
• Patch: Rerunning the test generator is probably not necessary.

Under this thought, I would put a reordering in minor.

Then again, some tracks might want to rerun their generator even on patch-level changes, so maybe I have the demarcation between minor and patch incorrect.

[rbasso]

Great summarization, @petertseng.

Under this thought, I would put a reordering in minor.

I completely agree!

Then again, some tracks might want to rerun their generator even on patch-level changes, so maybe I have the demarcation between minor and patch incorrect.

What matter here is conveying information to the tracks about what changed:

• MAJOR - Test data structure changed. (API break)
• MINOR - Test data changed / test cases reordered. (test content changed)
• PATCH - Non-API-breaking, non-test-content-changing modifications (test organization, cosmetic changes)
• PATCH - json-content-irrelevant changes (json-irrelevant changes)

The only drawback here is being forced to bump the PATCH on json-irrelevant changes, but they are so uncommon that I don't see the need to add one more level of versioning just for that, which would make everything even more confusing.

[ErikSchierboom]

@rbasso @petertseng I agree with the proposed setup. Looks great.

[ErikSchierboom]

@rbasso Have we documented the proposed semantic versioning scheme somewhere?

[rbasso]

I don't think we did, @ErikSchierboom. I just added the policy label and updated the first comment to reflect what was discussed.

[ErikSchierboom]

Hmmm, I would to see it documented somewhere, or at least a link to this document be added in a logical place (but where). What do you think?

[rbasso]

Considering that the documentation for the new schema is in the README.md, it should probably be inserted there. Maybe a reference in the CONTRIBUTING.md would be necessary...not sure.