Sync Up Existing Test Suites with x-common

[abo64]

There are continuous changes of existing tests in x-common.
Is there a good way to keep track and update the Scala tests?

Just to list some recent changes:

• scrabble-score
• nucleotide-count
• phone-number
• prime-factors
• etl
• secret-handshake

[ErikSchierboom]

Coincidentally, an issue for this topic has only recently been opened: exercism/problem-specifications#524 Perhaps you could chip in there?

[abo64]

What do you think about the idea to directly parse the canonical-data.json file as suggested in this Haskell issue?

[ricemery]

I think we should generate the tests programmatically. Some of the test suites are already built in this way. Take a look in the testgen directory and https://github.com/exercism/xscala/blob/master/README.md . The testgen code I have written isn't all that clean, but it works.

…

On Wed, Feb 1, 2017 at 10:48 PM abo64 ***@***.***> wrote: What do you think about the idea to directly parse the canonical-data.json file as suggested in this Haskell issue <exercism/haskell#487>? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#331 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABeHGwNgWrPhZPSCe8UemOP8f5_UNmv-ks5rYW4SgaJpZM4LyiJn> .

[abo64]

@ricemery Yes, that looks like the right idea.

I wonder whether we should even bother to generate extra Scala source code from the TestCases as it complicates matters quite a bit.
Maybe just use the TestCases to run the tests immediately? (very similar to the Haskell PR).

(btw, you use play.api.libs.json. Why?)

[ricemery]

If I understand the Haskell PR correctly, they want their test code to read the JSON and build/execute test cases dynamically.
I'm not sure I like that idea. It add complexity for the end user trying to figure out how to code their solution to meet the test requirements. When I solve exercism problems, I always look at the test code to figure out how to proceed. I suppose this is TDD.
I think having the test cases obfuscated within the JSON makes doing TDD more difficult.

Though I do agree it would make maintaining the test suites easier.

I am using play json because I knew how to use the api. It is in no way a statement on play json being the best way to parse json.

[abo64]

True, this approach works only if a closer look at the test suite is not required (which is the case in the Haskell track). Some issues:

• Every exercise includes a stub with the signature for the solution;
  or if we don't want this we need a unified format of the tests that makes it very clear how the solution signature must be: For example always having a (possibly overloaded) function named callMUT or so that then calls and reveals the required signature(s).
• The test failure messages must be self-explanatory including input, expected and actual output.
• There are no pending exercises (which I personally would welcome - one usually tries to go through the tests from top to bottom anyway to make them green).
• Maybe some additional general information about all this on how to run a test suite.

I agree that these are a quite disruptive changes, and that might be reason enough to not go this way.
On the other hand the extra step of creating a Scala source file adds quite some complexity and work for ourselves. Is it really worth the effort?

[ErikSchierboom]

Creating a Scala source is more work, but I really feel we should keep it that way. Every single exercise I try to solve, I start out with examining the tests. I really like having them be explicit and being able to run or debug them individually.

To solve the problem of automatically keeping up-to-date with the canonical data, I think a better approach would be to write a generator that takes the canonical data and automatically creates the test file.

[abo64]

All right, if that is the majority opinion I will of course follow.
I will look upon it as a challenge to simplify this translation process. :)

[ricemery]

I'm sure you can do a much better job of building a generator than I did! I will admit that my generators are ugly.

…

On Fri, Feb 3, 2017 at 1:46 AM abo64 ***@***.***> wrote: All right, if that is the majority opinion I will of course follow. I will look upon it as a challenge to simplify this translation process. :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#331 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABeHGyzWhZHm6DXAi-FJFnLrB8Tku04Lks5rYulIgaJpZM4LyiJn> .

[abo64]

No, that is fine.
I was thinking to extract some utility functions or so. Your generators are a good start!

[abo64]

Now the x-common JSON schema has become more standardized.

My suggestion is:
We track progress of this issue and update the Scala test suites more or less in sync with it, ideally adding respective test generators.
Even better would be to make use of the schema versions somehow?

What do you think?

[ErikSchierboom]

So are you suggesting that we should make the basic test generator conform to the updated JSON schema?

[abo64]

@ErikSchierboom Not really, but when I think about it: This is actually a good idea, too! :)
We could do this first. Probably with a clone of TestBuilder (named TestSuiteBuilder, say) in order not to interfere with existing generators.

As this issue is about being in sync with changes in x-common: My initial thought was only to monitor the issue and update our generators and test suites more or less in sync with it.
Or if we somehow keep track of the json file versionIds (for example as a comment in the test suite?) we might even be able to automatically generate the list of outdated test suites?

[rbasso]

We are not supposed to change the test data in exercism/problem-specifications#625, except for adding the property key and maybe make other minor changes to pass validation. I'm also taking the opportunity to reformat the files in separated commits.

If any PR there changes the test contents, it is a bug. 🐛

Of course, all the current generators will almost certainly stop functioning.

[abo64]

@rbasso You are right. Maybe I should then say:
Let's use the changes of tests in exercism/problem-specifications#625 to

• write/adapt Scala test generators for them.
• cover any changes of outdated Scala test suites on the fly.
• detect any future changes automatically if we can to keep the Scala test suites up to date.

[ErikSchierboom]

Sounds like a good plan!

[abo64]

TestCaseGen should be modified to contain all possible json schema fields.
And the Json parser can be included in TestSuiteBuilder to always (?!) create the Seq[TestCaseGen].

Then hopefully the default generator will just work for most test suites.

[rbasso]

TestCaseGen should be modified to contain all possible json schema fields.
And the Json parser can be included in TestSuiteBuilder to always (?!) create the Seq[TestCaseGen].

Then hopefully the default generator will just work for most test suites.

I just remembered one important thing...

One of the central features of the new schema is that test data is completely separated from test grouping, so it is possible to write generators that are not sensible to test grouping, and they would continue working if the tests are regrouped/"renested", which is not considered a MAJOR change. In fact, if the test ordering is not changed, regrouping is considered simply a PATCH by the proposed semantic versioning.