Run integration tests on CI

[Dexterp37]

In #16 we added running unit test coverage and linting for the addon on CircleCI.

We now need a way to run integration tests using real browsers (headless?).

[rhelmer]

I've been looking at what the major extensions like uBlock and AdBlockPlus and others do, they seem to have automation for building the extension and running unit tests but I'm not finding a ton on integration tests.

The Lighthouse extension from Google does this, and uses Puppeteer with headless Chrome: GoogleChrome/lighthouse#4640

There are various examples of using Mocha with Puppeteer and Selenium. I'm still trying to understand the differences between the two, but it looks like Puppeteer might be easier to set up and provide nicer integration into various browser features so it's probably worth seriously considering since it supports Firefox as well, which was one of the big sticking points for using Selenium for a while.

[rhelmer]

OK here's an example of using Puppeteer with web-ext:

const path = require("path");
const puppeteer = require("puppeteer");
const webExt = require("web-ext");

async function run() {
  webExt.util.logger.consoleStream.makeVerbose();
  const PORT = 1234;
  const extensionPath = "./";
  const runner = await webExt.cmd.run(
    {
      sourceDir: path.join(__dirname, extensionPath),
      firefox: {
        darwin: "/Applications/Firefox Nightly.app/Contents/MacOS/firefox-bin",
        win32: `${process.env.PROGRAMFILES}\\Firefox Nightly\\firefox.exe`,
      }[process.platform],
      args: [`--remote-debugging-port=${PORT}`],
    },
    {
      shouldExitProgram: false,
    }
  );
  const { debuggerPort } = runner.extensionRunners[0].runningInfo;
  console.log(debuggerPort);
  return await puppeteer.connect({
    browserWSEndpoint: `ws://localhost:${PORT}`,
  });
}

run()
  .then((result) => console.debug("running"))
  .catch((err) => console.error(err));

The puppeteer.connect() bit allows us to then use the Puppeteer API which is now hooked into the Firefox devtools:
https://github.com/puppeteer/puppeteer/blob/main/docs/api.md#puppeteerconnectoptions

I think Chrome doesn't allow extensions in headless mode, so we might have to run in "headfull" mode (which should be fine we can use a screen buffer or something). Chrome has a more direct way to load extensions vs. using web-ext (--load-extension) but if we can just use web-ext everywhere it will probably make our lives simpler.

[rhelmer]

So I think the remaining steps are:

• hook up a test target in package.json for this (maybe npm run test-integration)
• ensure that this runs in CI in head-full mode
• add at least one mocha test

[Dexterp37]

OK here's an example of using Puppeteer with web-ext:

Woha, this looks great and simple!

[rhelmer]

Ugh, OK. One big stumbling block here for me is that Chrome extension install works very differently than Firefox in general, and Puppeteer doesn't really officially document (as far as I can find) how this should work not that Firefox support is official.

It's kind of a weird trade-off, actually - Chrome has a command-line option to temp-install unsigned add-ons, but it does not work in headless mode which is rough. web-ext can work with both, and it uses the remote debugging protocol that they share to do the automation, but this means that many examples you'll find in the docs and web won't really be quite the same which is frustrating.

Back when Firefox support was experimental, this section of the doc existed which covered using web-ext:
https://github.com/puppeteer/puppeteer/blob/f26bb7f4c44d9b7db5cc73c4af32db6fa5bcd3a2/experimental/puppeteer-firefox/README.md#add-ons

However that page has been changed to say that Firefox is no longer experimental and that section was removed. From github discussions it looks like this is still the only way forward so I'm going to forge ahead, but it sure is frustrating :)

Outside of Mozilla's Firefox automation I haven't turned up much in the realm of official support for this sort of thing, which is frustrating. It's a different story for Chrome unsurprisingly.

[pdehaan]

I wonder if Playwright would be any easier than Puppeteer.

[Dexterp37]

Ugh, OK. One big stumbling block here for me is that Chrome extension install works very differently than Firefox in general, and Puppeteer doesn't really officially document (as far as I can find) how this should work not that Firefox support is official.

It's kind of a weird trade-off, actually - Chrome has a command-line option to temp-install unsigned add-ons, but it does not work in headless mode which is rough. web-ext can work with both, and it uses the remote debugging protocol that they share to do the automation, but this means that many examples you'll find in the docs and web won't really be quite the same which is frustrating.

Eugh :( I understand your frustration here.

I wonder if Playwright would be any easier than Puppeteer.

@pdehaan raises a good point! Maybe Playwright is worth trying: it seems to support easy integration with the Mocha testrunner. What do you think, @rhelmer ?

[Dexterp37]

@rhelmer @pdehaan I quickly experimented with Playwright. While it is super easy to setup and make it work (great job devs!), it does not support webextension testing. The devs are very explicitly stating it's out of scope for playwright. So this seems to be a no go.

[rhelmer]

OK so the more I look into this, I'm not sure how useful puppeteer will be for testing the multi-addon flow that we need to... Puppeteer works by connecting to devtools (AFAICT) using a remote debugging protocol, and I don't think Chrome provides a way to do install extensions via this mechanism, we probably need to use some lower-level instrumentation. It's worth double-checking this assumption. Firefox does have a way to do this IIRC.

The way I see other folks using Puppeteer here is to just pass the --load-extension flag to Chrome and load a single extension, or use web-ext to load a single extension, and then run mocha tests with the extension loaded.

However, what we need to do is:

1. navigate to website
2. install core add-on
3. navigate to options page and install study add-on

So I think that we need something like what we use for mochitests (Marionette I think?), since we need to interact with browser chrome. The higher-level API for this is WebDriver so I will dig into this a bit and see where I get.

We also might want to consider foisting this off to a Firefox mochitest for expediency, especially if we're going to land the core add-on in Firefox then that's something we'll want anyway. Still worth investigating if we could use something like mocha there and look at what WebDriver lets us do on other browsers. Getting around signature verification on Chrome without using temporary add-ons might be a big issue there...

[rhelmer]

@Dexterp37 given our time constraints, I think we should write a mochitest and run it in our CI (artifact build). This is something we've done in the past and it's Firefox only but it has Marionette etc. already set up.

We should still work on getting a more browser-agnostic setup but since we need to interact with browser UI it's going to be more complex than we originally hoped, so I want to unblock us in the short term.

wdyt?

[Dexterp37]

@rhelmer I think that's fair!

[Dexterp37]

This can be closed thanks to #212.