Tests fail due to timeouts after introduction of parallel execution + unusable logs

[supersilverlemonhaze]

1. Issue: Since version 1.4.3 and specifically due to the parallel execution, when executing a larger number of tests the run fails due to timeouts. We have at this moment only a low number of test devices. Thus if all tests start to run in parallel and start at the same time and some devices with the necessary tags are already in use, then the tests that are waiting will fail if timeout set too low. At this moment we have only 12 test cases running, but what if we add more? How much higher do we have to set the timeout then? Please introduce the option for us to enable/disable parallel execution. I've found this in the pom.xml: false
   Setting it to any value is ignored, everything runs in parallel.

2. Also: since the introduction of parallel execution the logs are all over the place, which renders them pretty much unusable, since the causality is lost and im not gonna piece the logs together by their timestamps.

[Beff42]

Hey there Lemonhaze,

the new configuration stems from an JUnit update. There should be no parallel execution, although JUnit now supports it, it is disabled by default.
The property you are referring to in the pom is claimParallel? This is independent from the JUnit execution and refers to a Testsuite mechanism where several devices can be claimed parallelly, to circumvent cases where claiming takes a long time (e.g. in case emulated devices are used).

I agree, the logging is not ideal. We will investigate this and I will let you know, what we can do about it.

Meanwhile another possibility for the timeout issue might be some delay in unclaiming the device? We also do some cleanup before unclaiming, like deleting rooms etc. maybe this is considerably slow? In the Serenityreport you can see the execution time for each individual test step. Each API call should also have a timestamp, there it should also be visible, that tests are not started parallel.

[supersilverlemonhaze]

Thanks for the quick response. Then I just wonder if its disabled by default, why is it then executed in parallel?

[Beff42]

Well the logging only looks as if it is executed parallel. That's why I suggested checking the Serenity report and the individual timestamps of the requests. There it will be visible, that they are not executed in parallel.

If you want to share private information (e.g. a Serenity report) you can also open a service Ticket for the TI-Messenger at https://service.gematik.de/servicedesk/customer/portal/24

[supersilverlemonhaze]

Thanks again for the advice, we will check serenity reports for that. However it would be great, if the logs came out neatly as well.

[Beff42]

Yes, agreed. We are working on a solution for the logging. This depends a bit on the Serenity Library we use, but maybe we'll be able to provide a temporary workaround.