Order of tests should be optimized for Test Driven Development

[mk-mxp]

Problem

Exercism states in some places, that the "test first" approach it uses, is like TDD. This does not hold true for the order of tests in problem specifications.

While making over minesweeper I followed TDD to develop the new example solution. There the tests jump from no dimension to both dimensions and come back later to one dimension. This leads to having to write the complete solution early to satisfy a test, without being guided through all the steps towards the final solution. And the tests coming after this early 2-dimensional test then do pass with no new code being written.

Problem specs in general group tests by a theme or a common property. This does not want to be the right order for TDD. It follows different rules.

Also, PHPUnit does not guarantee execution order of tests. Following FIRST principles, order must not matter. But that is not true for the process of TDD, only for using the tests later to prove correctness.

I think it is very advanced TDD knowledge to know which test needs to be passed next. So presenting students with an unordered list of failing tests does not help them to find their way through implementing a solution. At least, the tests should be ordered according to a good path through the TDD journey. It would be even better to always have only one failing test - which is a core property of TDD.

With this in mind, I think PHP track should have:

• A way to order tests optimized for TDD
• A way to keep / enforce the order when an exercise is sync'ed to problem specs
• Optionally a way to present students with only one failing test instead of a wall of failures

Suggested solution

PHPUnit allows dependencies (@depends) between tests. These are not explicitly meant to enforce an execution order, but by definition a test depending on the result of a previous test ensures an execution order for those tests. As a side effect PHPUnit skips tests when the tests fail they depend upon. This technically solves the way how to have the tests ordered and only fail one test at a time.

To know the designed order of tests, we may add a key testOrder to .meta/config.json with an ordered list of UUIDs. This list must be considered by test generators as well as contributors. When it does not exist, we fall back to the order in problem specs.

[mk-mxp]

It was agreed upon in the forum, that the order of tests in canonical data should be followed and that the order should be optimized there. Also the grouping was seen as useless for most tracks and was questioned, but there was no agreement to flatten the data.