Validate the test data version as part of a travis build

[behrtam]

Thanks to the work of @rbasso (see exercism/problem-specifications#602, exercism/problem-specifications#625, exercism/problem-specifications#673, etc) and many more, we now not only have a schema but also a test data version.

In exercism/problem-specifications#524 @robkeim started to think about how to propagate changes from x-common to the tracks. In this issue @ErikSchierboom wrote:

Yes, sending an update each time something changes in x-common can be tiresome, but the alternative (manually keeping track of x-common) is not a whole lot better.

So, I thought about a way that doesn't generate a lot of mails, issues or notifications, but makes sure that at least every time someone actually is working on a test file it is up to date.

I started to work on a script (check-testversion.py) for the python track that checks the versions for all exercises that are part of a travis build against the newest version in the x-common repo, a bit like a linter or stylechecker. It still needs a lot of cleanup and testing, but it at least proves that the basic idea is working.

The basic concept is ...

1. get a list of changed files git diff --name-only $TRAVIS_COMMIT_RANGE
2. get the newest version from canonical-data.json through https://api.github.com/repos/...
3. get the actual version from a comment in the test file # test @ version: 1.0.0
   (maybe it should be excluded into a separate JSON file as part of the exercise directory)
4. compare the versions and output some infos

As a test I changed the anagram version back from 1.0.1 to 1.0.0 in this commit: behrtam/python@3dc7acd
Which resulted in a failed build as expected: https://travis-ci.org/behrtam/xpython/jobs/219306945#L163

I don't know if this will finally be integrated into the python track or if this is a good idea at all, but I thought maybe some parts of this could also be interesting for other tracks.

[kytrinyx]

I like this idea a lot. I'd love to hear if any other tracks try it out and what you think.

[petertseng]

Nice. I have https://github.com/petertseng/x-common/tree/up-to-date going but haven't gone to the length of putting it in the CI for the respective tracks yet (probably shouldn't, since why would Ruby code go in non-Ruby tracks?)

That gives colour coded output of whether things are up to date, like (sorry, can't show the colour coding on Github markdown)

                  bracket-push: 1.0.0 -> 1.1.0
                       hamming: 1.0.0 == 1.0.0
                       anagram: 1.0.1 == 1.0.1
        largest-series-product: 1.0.0 == 1.0.0
                         sieve: 1.0.0 == 1.0.0
                         react: 1.0.0 == 1.0.0
                          leap: 1.0.0 == 1.0.0

[hilary]

xruby is working towards something similar. We now have the test data version in our generated test files:

• test template
• generated tests

[stevejb71]

xocaml is also doing this for generated tests, with a comment at the top of each test.ml file:

(* Test/exercise version: "1.0.0" *)

[petertseng]

I'm trying to figure out the best cadence for running a script that ensures exercises are up to date. I thought of a few choices:

• The maintainers simply have to remember to run the script on a manual but regular basis. This of course adds no complexity to the CI. The obvious disadvantage is the human element - what if someone forgets?
• CI checks changed files only. This leaves open the possibility that a file will go unchanged and therefore unchecked for a long time. How to deal with this? Combine it with the above strategy?
• CI checks all files. A big disadvantage here is that a previously-passing build may turn into failing even though nothing has changed in the repo being checked (only x-common).

I would be interested to hear any of y'all's experiences on that.

[stevejb71]

I would not want to enable the 3rd option, it would take the focus off the PR I'm working on.

Would it be possible for a central script to raise /update a single issue in each stream repo for version mismatches?

[robkeim]

@stevejb71 we have a script that can do that and we discussed enabling something like that here, although there was concern about the additional burden for track maintainers.

[stevejb71]

Can it be done on a per track basis? I'd be fine with it for the ocaml track.

[rpottsoh]

I wouldn't mind issues being created / updated in the Delphi track as long as the issue pertained to an exercise my track currently offers. Tracks should be able to opt out of the feature.

[cmccandless]

If exercism/meta#97 comes to fruition, I would prefer that solution to CI checks. While both could be done together, to me using both might feel redundant.

[kytrinyx]

I agree that the bot would be a better solution—it will not slow down CI or break the build.

[Stargator]

Not sure about having the test version in a comment, why not in the config.json for each exercise?

[kytrinyx]

How about putting the test version inside a file in the .meta dir?

[coriolinus]

If test version lived in a separate file in .meta, I'd find it pretty easy to accidentally forget to update it when updating an exercise. A comment near the top of the file containing the tests has the property of showing up as soon as I open that file in a text editor.

[wolf99]

That would be more visible, but also would require changes to existing files in every exercise in most tracks.... :-\

[coriolinus]

Yes. Happily, the required changes are of the sort which should be very easy to automate. It's a matter of writing up an appropriate script, running it once, committing the changes, and then being done with it.

…

On Fri, Feb 23, 2018 at 6:00 PM, toby ***@***.***> wrote: That would be more visible, but also would require changes to existing files in every exercise in most tracks.... :-\ — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHdeTgdQVVtgsEjRMlSZdjPUVftBNp0fks5tXu6ggaJpZM4M1wB0> .

[kytrinyx]

@coriolinus if you automate the change, then it wouldn't matter where it is, right?

[coriolinus]

@kytrinyx I was talking about automating the change to insert the version comment without considering maintenance. You're quite right: if you insert the version string automatically, it doesn't matter where it is. However, I believe that my earlier point still stands: after the big update to insert tracking version strings, it does still matter where it lives; putting it near the top of the file most likely to be edited makes it more likely to be remembered than putting it in an unrelated file.

@petertseng also brought up an interesting point in a different discussion: there may be track-specific canonical locations where version strings are expected. Rust, for example, expects to find version info in Cargo.toml in the project root; it would not surprise me if other tracks now or in the future also had project-specific canonical version locations. Adding versioning information elsewhere would require either keeping the two version strings synchronized, or depreciating the track-specific location in favor of the exercism-specific one.

The utility of having a bot such as proposed in exercism/meta#97 would outweigh the hassle of taking either of those courses, but a really good bot might be configurable to look in the track-specific location anyway.

[kytrinyx]

@coriolinus Thanks for the clarification.

I think the bot is the best first step, and having the bot look for version information in a configurable location would be a nice add-on.

[kytrinyx]

We've imported this issue to the https://github.com/exercism/exercism.io repository.