Rethink test strategy

[mnonnenmacher]

Currently most functional tests work by comparing analyzer or scan results with a YAML file that contains the expected serialized result.
The problem with this approach is, that comparing the complete file often compares much more than is required to test a single feature. As a result simple refactorings can break a lot of tests which do not test the refactored code, and it is required to change a lot of YAML test files to fix this, which makes refactorings very expensive.
A way to fix this would be to reduce the amount of YAML based tests and to make sure that a test for a specific feature only tests this feature and nothing else. This would make it necessary to write a lot more code in the tests themselves, but with the gain that refactorings become cheaper and issues with related tests can often be found at compile time, not only when executing tests.

[sschuberth]

Also see #2450 (comment).

[mnonnenmacher]

Another issue with the YAML based tests is that it is difficult to fix them when the model is changed. Issues with expected result files are discovered only when executing tests, and manually fixing them can be a lot of effort.
One approach to improve this situation is to create the expected results in Kotlin code, this way issues with model refactorings would already be detected at compile time. This is similar to the test data that is already used for some other tests, see:

• https://github.com/oss-review-toolkit/ort/blob/5162e306caaf81478f32ee597a193f1ec815410b/model/src/test/kotlin/licenses/TestData.kt
• https://github.com/oss-review-toolkit/ort/blob/142aa8c1a727c22fd3a70ae73d73fd4e169c57d1/evaluator/src/test/kotlin/TestData.kt
• https://github.com/oss-review-toolkit/ort/blob/142aa8c1a727c22fd3a70ae73d73fd4e169c57d1/reporter/src/funTest/kotlin/TestData.kt

To amount of code required to create this test data could be reduced by creating a simple DSL to create ORT results.

[nnobelis]

One can use DSL builder in Kotlin to build expected results :
https://kotlinlang.org/docs/type-safe-builders.html

[sschuberth]

One can use DSL builder in Kotlin to build expected results : https://kotlinlang.org/docs/type-safe-builders.html

Indeed, and a start was committed with 0f26841.

[sschuberth]

Closed as part of backlog grooming. Feel free to comment if you would like to contribute to this.