Repeat mode for integration tests

[odisseus]

To use it, add the option -PrepeatTests to the command line. Example:

./gradlew integrationTest -PrepeatTests=20

Test plan

1. Introduce non-deterministic failures, like this:

   val x = Random.nextInt() % 10
   if (x < 2) {
       fail()
   } else if (x < 4) {
       1 / 0
   } else if (x < 6) {
       throw IllegalStateException("$x")
   }

2. Run the tests with high enough repetition count
3. Verify that different stack traces are present in the test report

[pkukielka]

If you need it to be a system property you don't need to do that, but instead you can run gradle with -DrepeatTests instead of -PrepeatTests

[odisseus]

According to Stackoverflow, the -D argument won't necessarily pass through to the tests.

[pkukielka]

Maybe also something with Repeatable or Repeat in name? Custom doesn't bring too much context

[odisseus]

This class doesn't really repeat anything or make anything repeatable. I don't mind renaming it, but I can't think of any good name.

[pkukielka]

Could you somehow attach example off test report or at least screenshot of it?

[mkondratek]

Is it easily doable to replace

DocumentCodeTest. testGetsFoldingRanges
DocumentCodeTest. testGetsFoldingRanges
DocumentCodeTest. testGetsFoldingRanges
...

with

DocumentCodeTest. testGetsFoldingRanges (0)
DocumentCodeTest. testGetsFoldingRanges (1)
DocumentCodeTest. testGetsFoldingRanges (2)
...

❓

[odisseus]

Not without writing a lot more code. I will try to do that in a follow-up PR.