request: APIs and infrastructure to simplify cross-browser automated testing of extensions

[twschiller]

Context

Automated testing tools, e.g., Selenium, Cypress, Puppeteer, and Playwright are invaluable for cross browser web application testing. There's limited support for them for testing browser extensions

There are mocks (e.g,. jest-chrome) for unit testing, but these don't capture the complexity of extension communication and browser quirks

Automated cross-browser testing will be valuable for checking the progress of the browsers vs. this w3c group's proposals.

Request

I don't have a clear recommendation for what's needed here. My goal is to get the conversation kicked off (with people more knowledgable than me!) On the PixieBrix project, we're just now trying to set up our cross-browser browser automated testing infrastructure

I suspect there's needs to be some mix of:

• Exposing new APIs: e.g., for error detection, for adjusting permissions without user interaction, etc.
• The extension community contributing to the automated testing tool projects (code and documentation)
• Providing examples/documentation of testing different features across browsers

Related context:

• Extension testing in browser / puppeteer pixiebrix/pixiebrix-extension#534
• https://groups.google.com/a/chromium.org/g/chromium-extensions/c/2GU7e-OKwtQ/m/CeLBiMe_BwAJ
• Firefox extensions with v2.1.0 puppeteer puppeteer/puppeteer#5532 (comment)
• https://groups.google.com/a/chromium.org/g/chromium-extensions/c/a2xNMNu3DCc/m/gEqUyOIXCAAJ

[Techdojo]

I’m especially interested in this as we’re currently working on ways to implement automated testing on browser extensions, feel free to reach out to me and we can discuss our respective strategies. Jon…

…

Sent from my iPhone

On 23 Jun 2021, at 18:37, Todd Schiller ***@***.***> wrote:  Context Automated testing tools, e.g., Selenium, Cypress, Puppeteer, and Playwright are invaluable for cross browser web application testing. There's limited support for them for testing browser extensions There are mocks (e.g,. jest-chrome) for unit testing, but these don't capture the complexity of extension communication and browser quirks Automated cross-browser testing will be important for checking the progress of the browsers vs. this w3c groups proposals. Request I don't have a clear recommendation for what's needed here. My goal is to get the conversation kicked off (with people more knowledgable than me!) On the PixieBrix project, we're just now trying to set up our cross-browser browser automated testing infrastructure I suspect there's needs to be some mix of: Exposing new APIs: e.g., for error detection, for adjusting permissions without user interaction, etc. The extension community contributing to the automated testing tool projects (code and documentation) Providing examples of testing different features across browsers Related context: pixiebrix/pixiebrix-extension#534 https://groups.google.com/a/chromium.org/g/chromium-extensions/c/2GU7e-OKwtQ/m/CeLBiMe_BwAJ puppeteer/puppeteer#5532 (comment) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[ghostwords]

At Privacy Badger, we use Selenium to continuously test functionality in Chrome and Firefox. The benefit of Selenium is that we exercise the actual extension in an actual browser.

Off the top of my head, we wish we had the ability to interact with browser chrome, in particular programmatically open the browser popup. As is, we have to resort to various hacks (examples: opening the popup, opening new browser windows).

[dotproto]

I’m interested in digging in here to better understand how automated end to end and unit testing of browser extensions can be improved. At the moment the biggest gap I know of is the highlighted by @ghostwords - there’s no way in WebDriver or Chrome DevTools Protocol to trigger a browser action. What other gaps are there?

[fregante]

@dotproto it seems that no tool currently allows running code in specific contexts, for example in the background page/worker, a devtool_page, or a content script. From what I gather, the only way to do so currently is to:

1. Load a page under chrome-extension:// in a new tab
2. Communicate to the other contexts from there
3. Have each context actually run the code locally

This means that in order to run test code in one of these contexts, the code needs to exist in the extension itself — at least during the testing phase.

Privacy Badger above seems to use getBackgroundPage() to access its window, but the code is still not executed in the background page (which may be a requirement in some cases).

Most guides online focus on loading the extension in the browser, but there's almost nothing about actually running extension code.

[tregagnon]

@dotproto Yes the main issue is about the Popup (or any browser chrome UI that can display an extension, like the Sidebar in Firefox).

Without a command to trigger the browser action, and open the Popup, usually the workaround for extension developers is to load the Popup page in a new tab using an extension URL (like chrome-extension://).

The problem with this method, is that it makes irrelevant any tests about features that access information about the current tab. Because extension URLs (chrome-extension://) are usually treated differently. And for proper testing, it would be better to test different situations.

[ghostwords]

Privacy Badger above seems to use getBackgroundPage() to access its window, but the code is still not executed in the background page (which may be a requirement in some cases).

What cases are those? Are you mixing up unit testing with functional testing?

[fregante]

Privacy Badger above seems to use getBackgroundPage() to access its window, but the code is still not executed in the background page (which may be a requirement in some cases).

What cases are those? Are you mixing up unit testing with functional testing?

I have several npm modules for WebExtensions so it'd be great that they are tested against real browsers and not just against mocks, which don't reflect the continuous changes, bugfixes and regressions that browsers get.

One explicit example:

• https://github.com/fregante/webext-detect-page has a test to detect the code is run in a specific context. This is generally used to share code/modules across contexts, but ensuring that they're not called the wrong context

But also:

• Ensuring that the content script is receiving messages from the background script and and it's filtering messages from any other page

[dotproto]

Last week I opened issue 3848 on the ChromeDriver project. My understanding at the time was that WebDriver was a better fit for this capability than CDP, but, as noted in that issue, after some discussion with the relevant team it seems like Chrome may experiment with this request in the Chrome DevTools Protocol.

[xeenon]

I would prefer we tackle this in WebDriver, since that is more at the browser level than the dev tool protocol level.

[ariasuni]

For reference, the «8. Testing Extensions with WebDriver» section of the draft of the previous CG:
https://browserext.github.io/browserext/#wd

[dotproto]

@xeenon, for clarity, are you referring to information gathering or where these features would be implemented? For information gathering, I think the WECG may be a good place to discuss use cases that are not well served today, to discuss potential solutions, and potentially even draft suggestions for consideration by other WGs. On the other hand if you meant that it should be implemented in WebDriver, I don't disagree.

The idea of using Chrome DevTools Protocol (CDP) was for experimentation, not as a long term solution. As I understand it, the Chrome DevTools team maintains CDP and may also be taking over ChromeDriver. Since CDP is a browser-specific feature, Puppeteer is built on CDP & is used in automated testing workflows, and that team has relevant expertise with both, it seemed like a good opportunity to explore concrete ways of addressing extension testing gaps, then bringing those findings to the WebDriver WG for discussion.

[ariasuni]

I just want to mention that there’s webextensions-api-mock who’s using the JSON schema used internally in Chrome and Firefox to automatically mock the whole APIs, and unit testing is a thing that should be taken into consideration also.

[cuylerstuwe]

The nature of browser extensions is that most of them intend to cause side-effects within the environment in which they're intended to run.

Therefore, for most extensions, end-to-end testing is overwhelmingly preferred, as they're the often the only test cases that prove anything meaningful.

[dotproto]

This morning I attended the Browser Tools and Testing Working Group's monthly WebDriver BiDi meeting (minutes). My goal in attending was to better understand the group's current focus, raise the feature requests we've seen (namely #19 and #140), and get an initial impression on how to proceed.

WG Feedback

Initial feedback from the group indicates that there are currently two incarnations of WebDriver: Classic and BiDi (bi-directional). Classic is primarily focused on interacting with web content and is no longer being actively developed. The WG is currently focusing on BiDi; this specification is likely the right target for deeper integrations with the browser and new, low-level testing capabilities.

In both cases, the WebDriver specs both support the concept of extensions provided by browser vendors or other specifications (Classic, BiDi). It will most likely be best for us to create a Classic or BiDi extension to provide the capabilities extension developers would like to see. The Testing & Tools WG obviously has expertise here and indicated that they are open to reviewing and contributing to our extension specifications.

Next steps

Given this context, it looks like we'll will need to create and maintain our own specification(s) to address the needs of WebExtension authors. Our next steps are roughly…

1. Create a concrete set of objectives that we can share and discuss with the Browser Testing and Tools WG. Essentially a "this is what we need" document.
2. Share the Need Doc to the WG to request feedback and guidance.
3. Begin drafting one or more specifications to address gaps in the automated testing experience for WebExtension authors.
4. Iterate on these specifications based on additional feedback from the Browser Testing and Tools Working Group, extension authors, and browser vendors.
5. Profit!

Additional links / notes

Meeting attendees shared a handful of links that may be useful as we proceed with specifying WebDriver extensions.

• https://www.w3.org/wiki/WebDriver/2022-01-BiDi

  • 2022-01-12 meeting agenda

• https://w3c.github.io/permissions-automation/

  • A WebDriver extension that "facilitates testing of the Permissions API."

• Exposing vendor specific API for testing (e.g. terminateServiceWorker) web-platform-tests/wpt#6866

  • Related to Add support for manually triggering service workers being suspended to allow automated testing #140

There were also a couple other points came up that I wanted to quickly highlight.

• @shs96c: "Back in the day geckodriver had the ability to switch between contexts. There's probably some prior art there. But the best way to do is probably to do a WebDriver extension".
• @jgraham: "A lot of things like network intercept don't have any standards now. We'd definitely welcome collaboration on that sort of thing."
  • I see potential overlap between extension debugger API capabilities and BiDi. This could potentially also be used for webRequest.
  • There may well be other areas of overlap. For example, "Sandboxed Script Execution" (Specify sandboxed script execution webdriver-bidi#144), WebExtension content scripts, and WebExtension support for user scripts in the scripting API.