New xml results file created for every test when running with Orchestrator

[joehoag]

Hello. I am attempting to use xray-junit-extensions for an Android project. When we enable Orchestrator, I think that each test is interpreted as an individual test plan, and so we get a separate XML results file for each test / test case. So if we ran 1000 tests, we would get 1000 xml results files.

When we disable Orchestrator, we get the expected result: a single XML results file that contains all test cases.

We need to have Orchestrator enabled. Can you give me any advice as to how to overcome the one-results-file-per-test-case problem?

[bitcoder]

What is Orchestrator? I'm not aware of it

[joehoag]

https://developer.android.com/training/testing/instrumented-tests/androidx-test-libraries/runner#use-android

Orchestrator clears out all app state before each test, and runs each test in its own instrumentation "sandbox". It is used to make sure that each test is run "fresh" and is not affected by leftover state from previous tests.

[bitcoder]

One thing that you could do is to merge the Junit XML reports together.
I remember the junit-merge tool but I haven't tested in a while. There are other though. Maybe that addresses your goal.
I have no experience with Orchestractor.

[joehoag]

Thanks. We would pursue a merging solution as a last resort. Right now I'm working on finding some sort of hook that tells me when all of my tests have completed running. With that, I could tweak EnhancedLegacyXmlReportGeneratingListener.java to only produce the results file once all tests have completed.

[bitcoder]

@joehoag if you find it, that would be great if it could generalized and eventually included on this project. Can you get a sample and simple project that showcases the current behaviour so I can eventually try to replicate it?

[bitcoder]

any feedback on this @joehoag ?

[joehoag]

Hi Sergio. There were a couple of developments on our end: - We were never able to find a good solution to the one-results-file-per-test issue, so we ended up adding some logic to combine all separate results files into an aggregate results file. - We decided that we would rather not convert our test code to JUnit 5, so we converted your code to be based on the JUnit4 RunListener rather than the JUnit5 TestExecutionListener. So our code would probably look somewhat familiar to you, but it's undergone some pretty significant overhaul. Regards, --Joe

…

On Thu, Dec 26, 2024 at 3:39 AM Sergio Freire ***@***.***> wrote: any feedback on this @joehoag <https://github.com/joehoag> ? — Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7LDQNQ6AHAMK2BIKI7PD2HPMFRAVCNFSM6AAAAABQO5SR7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRSGQZDIMZXGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>