[Feature]: Run same tests in multiple different browsers in one invocation

[DerChris173]

🚀 Feature Request

The node.js version of playwright has a very nice feature to run the same tests in multiple browsers (i.e. Firefox, Chrome, Webkit) in a single test runner invocation, see https://playwright.dev/docs/test-projects . This does not seem to be supported in the Windows version right now.

It would be nice to enable this somehow as part of a @testtemplate .

Example

Options::browserName could be changed to accept a list as a parameter.

Then, a new ArgumentsSource could be used together with existing @ParameterizedTest annotation to inject different BrowserContexts based on configuration.

@ParamterizedTest
@BrowserSource
public void runPlaywrightTest(Browser browser) {
... invoked once for each configured browser
}

Based on that, multiple test invocations could be triggered with the set of configured browsers.

Motivation

Simpler integration of test runs of multiple browsers into CI/CD pipelines.

[uchagani]

This will require some thought. JUnit supports repeating a test via the @RepeatedTest annotation but i'm not sure if that exposes anything that will enable us to pick up different configurations.

For now, what I would recommend doing is running in different browsers via a system property. something like mvn test -Dbrowser=$browserNameFromSomeShellLoop

and in your OptionsFactory, set the browser according to what the system property has.

[uchagani]

I don't think we will ever be able to get it to the same point has the playwright-test runner is without writing our own test runner. the above suggestion is probably the best for now.

[DerChris173]

I think it might not be too much work to build such a feature.
We would need an implementation of ArgumentsProvider interface, which somehow accesses the Playwright configuration and create/get new Browser instances. Maybe it could directly invoke BrowserExtension::getOrCreateBrowser? That method could be retrofitted to also allow selection of a specific browser via parameter instead of taking it from options.browserName

Hardest part might be figure out a design which integrates well into existing browser selection.

Would patches be accepted for this?

[uchagani]

It would be up to the playwright team whether patches would be accepted for it. I only worked on the JUnit integration so was speaking from that perspective.

You could also create your own extension and see if it works. One of my goals was to allow people to create their own extensions since I do that a lot for various functionalities. If you run into a situation where the current implementation is preventing that, I think it's something that we should look into accommodating.

[DerChris173]

Please have a look at the prototype I built here DerChris173#1

I had to do some modification in BrowserExtension to not automatically use the browser from the option, but from an extension to give some external control.
I ran a small test, it seems to work fine.

Is there a way to move forward with this approach and get it integrated? At least the parts that require changes in the main codebase...

Thanks!