Add some integration tests using puppeteer #12668

calixteman · 2020-11-30T17:12:43Z

run with gulp integrationtest

timvandermeij

Nice idea! I primarily have some comments that should make this more generally usable. We should think about if we want to make this part of the bot/Travis tests already or to do that later.

test/unit/integration.js

gulpfile.js

test/test.js

timvandermeij · 2020-11-30T21:01:05Z

test/test.js

- const page = pages[0];
- await page.goto(startUrl, { timeout: 0 });
+ };
+ if (browserName === "chrome") {


Could you add a comment why these additional Chrome and Firefox preferences are necessary?

About option for Chrome, they're required at least on my machine because without them chrome is crashing!
And as a side effect I just discovered that unit tests are running on chrome too.

test/unit/integration.js

brendandahl

For getting various events when a page is rendered and forms are ready we can enable the dom dispatch. It's currently only enabled for development and mozcentral builds, so we'd need to add a new build flag or option to

pdf.js/web/ui_utils.js

Line 806 in a618b02

if (typeof PDFJSDev === "undefined" || PDFJSDev.test("MOZCENTRAL")) {

brendandahl · 2020-12-03T00:02:07Z

test/integration/integration-boot.js

+
+const describes = [];
+
+global.describe = (desc, filename, selector, callback) => {


Instead of adding a new test framework, can we load jasmine (which we already use)? I was hoping to use the integration tests for more than just form tests which this is pretty tailored too. This current method would then be a helper that a test would use.

Snuffleupagus · 2020-12-03T12:38:04Z

For getting various events when a page is rendered and forms are ready we can enable the dom dispatch.

While we used to dispatch events to the DOM, that was purposely removed since it's not a great API in the general case (and only existed for backwards compatibility); hence can we please not re-add that sort of functionality here!

If it's turns out to be absolutely necessary, with no way around it, please at least limit this to only being enabled (using pre-processor statements) for TESTING-builds.

Edit: The supported way of accessing events globally, in a GENERIC-build of the default viewer (which is what we should be testing here), would be along the lines of https://github.com/mozilla/pdf.js/wiki/Third-party-viewer-usage, i.e.

document.addEventListener("webviewerloaded", function() {
  PDFViewerApplication.initializedPromise.then(function() {
    // The viewer has now been initialized, we can access e.g. the `EventBus`-instance.
    PDFViewerApplication.eventBus.on("pagerendered", evt => {
      console.log(`Rendered page: ${evt.pageNumber);
    });
  });
});

It's currently only enabled for development and mozcentral builds, so we'd need to add a new build flag or option to

The only reason that we even kept this functionality at all, limited to mozilla-central (and nowhere else), was to not break the existing tests in https://searchfox.org/mozilla-central/source/toolkit/components/pdfjs/test

timvandermeij · 2020-12-08T21:39:50Z

Related: mozilla/botio-files-pdfjs#28

brendandahl · 2020-12-09T19:15:56Z

/botio-linux integrationtest

pdfjsbot · 2020-12-09T19:15:58Z

From: Bot.io (Linux m4)

Received

Command cmd_integrationtest from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/eb1d4532db58d02/output.txt

pdfjsbot · 2020-12-09T19:18:29Z

From: Bot.io (Linux m4)

Success

Full output at http://54.67.70.0:8877/eb1d4532db58d02/output.txt

Total script time: 2.52 mins

Integration Tests: Passed

brendandahl · 2020-12-09T19:22:56Z

/botio-windows integrationtest

pdfjsbot · 2020-12-09T19:22:58Z

From: Bot.io (Windows)

Received

Command cmd_integrationtest from @brendandahl received. Current queue size: 0

Live output at: http://3.101.106.178:8877/b4b089de48464a6/output.txt

pdfjsbot · 2020-12-09T19:26:20Z

From: Bot.io (Windows)

Failed

Full output at http://3.101.106.178:8877/b4b089de48464a6/output.txt

Total script time: 3.36 mins

Integration Tests: FAILED

brendandahl · 2020-12-10T03:24:04Z

For the windows failure we need to increase the timeout for the beforeAll (second argument). There's suppose to be a way to change it for everything with jasmine.DEFAULT_TIMEOUT_INTERVAL but I can't seem to get that to apply.

calixteman · 2020-12-10T12:23:18Z

/botio-linux integrationtest

pdfjsbot · 2020-12-10T12:23:20Z

From: Bot.io (Linux m4)

Received

Command cmd_integrationtest from @calixteman received. Current queue size: 0

Live output at: http://54.67.70.0:8877/ee4b383af23bc29/output.txt

gulpfile.js

test/integration/annotation_spec.js

pdfjsbot · 2020-12-10T12:25:44Z

From: Bot.io (Linux m4)

Success

Full output at http://54.67.70.0:8877/ee4b383af23bc29/output.txt

Total script time: 2.40 mins

Integration Tests: Passed

Snuffleupagus · 2020-12-10T12:26:28Z

web/app_options.js

+ typeof PDFJSDev === "undefined" ||
+ PDFJSDev.test("!PRODUCTION && !INTEGRATIONTEST")


Also, are these changes actually necessary since there should be no build-target setting PRODUCTION = false anyway?

Good question, there was a !PDFJSDev.test("PRODUCTION"), is this value different of PDFJSDev.test("!PRODUCTION") ?
And there's a similar use here:
https://github.com/mozilla/pdf.js/blob/master/web/app_options.js#L248

there was a !PDFJSDev.test("PRODUCTION"), is this value different of PDFJSDev.test("!PRODUCTION") ?

Those two formats are equivalent.

pdfjsbot · 2020-12-10T12:57:18Z

From: Bot.io (Windows)

Received

Command cmd_integrationtest from @calixteman received. Current queue size: 0

Live output at: http://3.101.106.178:8877/faba073cd9610f0/output.txt

pdfjsbot · 2020-12-10T13:00:42Z

From: Bot.io (Windows)

Success

Full output at http://3.101.106.178:8877/faba073cd9610f0/output.txt

Total script time: 3.38 mins

Integration Tests: Passed

brendandahl · 2020-12-10T17:32:41Z

/botio test

pdfjsbot · 2020-12-10T17:32:42Z

From: Bot.io (Windows)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://3.101.106.178:8877/c9a964717dc16bb/output.txt

pdfjsbot · 2020-12-10T17:32:42Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/bd7a2596622c823/output.txt

brendandahl

Let's merge this if the full test suite runs fine, so we can unbreak tests for other PR's.

pdfjsbot · 2020-12-10T17:58:49Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/bd7a2596622c823/output.txt

Total script time: 26.10 mins

Font tests: Passed
Unit tests: FAILED
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/bd7a2596622c823/reftest-analyzer.html#web=eq.log

pdfjsbot · 2020-12-10T18:03:15Z

From: Bot.io (Windows)

Failed

Full output at http://3.101.106.178:8877/c9a964717dc16bb/output.txt

Total script time: 30.55 mins

Font tests: Passed
Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://3.101.106.178:8877/c9a964717dc16bb/reftest-analyzer.html#web=eq.log

calixteman · 2020-12-10T18:23:15Z

/botio test

pdfjsbot · 2020-12-10T18:23:16Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 1

Live output at: http://54.67.70.0:8877/7154439e10f2187/output.txt

pdfjsbot · 2020-12-10T18:23:17Z

From: Bot.io (Windows)

Received

Command cmd_test from @calixteman received. Current queue size: 1

Live output at: http://3.101.106.178:8877/95dd2507981f2ec/output.txt

pdfjsbot · 2020-12-10T18:50:45Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/7154439e10f2187/output.txt

Total script time: 26.05 mins

Font tests: Passed
Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/7154439e10f2187/reftest-analyzer.html#web=eq.log

calixteman · 2020-12-10T18:57:53Z

/botio test

pdfjsbot · 2020-12-10T18:57:54Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.67.70.0:8877/838f315cdd2283c/output.txt

pdfjsbot · 2020-12-10T18:57:55Z

From: Bot.io (Windows)

Received

Command cmd_test from @calixteman received. Current queue size: 1

Live output at: http://3.101.106.178:8877/0c190421f889fcc/output.txt

pdfjsbot · 2020-12-10T19:04:12Z

From: Bot.io (Windows)

Failed

Full output at http://3.101.106.178:8877/95dd2507981f2ec/output.txt

Total script time: 30.33 mins

Font tests: Passed
Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://3.101.106.178:8877/95dd2507981f2ec/reftest-analyzer.html#web=eq.log

pdfjsbot · 2020-12-10T19:24:03Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/838f315cdd2283c/output.txt

Total script time: 26.13 mins

Font tests: Passed
Unit tests: FAILED
Integration Tests: Passed
Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/838f315cdd2283c/reftest-analyzer.html#web=eq.log

pdfjsbot · 2020-12-10T19:36:26Z

From: Bot.io (Windows)

Failed

Full output at http://3.101.106.178:8877/0c190421f889fcc/output.txt

Total script time: 32.25 mins

Font tests: Passed
Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

Image differences available at: http://3.101.106.178:8877/0c190421f889fcc/reftest-analyzer.html#web=eq.log

* run with `gulp integrationtest`

brendandahl · 2020-12-10T20:01:30Z

/botio test

pdfjsbot · 2020-12-10T20:01:31Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/90b20ee0335da91/output.txt

pdfjsbot · 2020-12-10T20:01:31Z

From: Bot.io (Windows)

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://3.101.106.178:8877/83d68d3ce4c659e/output.txt

pdfjsbot · 2020-12-10T20:27:42Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/90b20ee0335da91/output.txt

Total script time: 26.17 mins

Font tests: Passed
Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/90b20ee0335da91/reftest-analyzer.html#web=eq.log

pdfjsbot · 2020-12-10T20:31:27Z

From: Bot.io (Windows)

Failed

Full output at http://3.101.106.178:8877/83d68d3ce4c659e/output.txt

Total script time: 29.92 mins

Font tests: Passed
Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

Image differences available at: http://3.101.106.178:8877/83d68d3ce4c659e/reftest-analyzer.html#web=eq.log

timvandermeij · 2020-12-10T21:01:29Z

Thank you for setting this up! This will be very useful so we can actually start writing integration tests for the viewer now.

timvandermeij reviewed Nov 30, 2020

View reviewed changes

timvandermeij added the test label Nov 30, 2020

calixteman force-pushed the interaction branch 2 times, most recently from a148246 to 0e59e99 Compare December 1, 2020 15:36

brendandahl suggested changes Dec 3, 2020

View reviewed changes

calixteman force-pushed the interaction branch 3 times, most recently from 110ae24 to d3595f6 Compare December 8, 2020 16:42

brendandahl mentioned this pull request Dec 8, 2020

Add support for the integration tests. mozilla/botio-files-pdfjs#28

Merged

calixteman force-pushed the interaction branch from d3595f6 to 2b2776a Compare December 8, 2020 17:26

calixteman force-pushed the interaction branch 3 times, most recently from 18f6a34 to c1f96de Compare December 9, 2020 18:49

calixteman force-pushed the interaction branch 2 times, most recently from fcc1697 to d3cb3ab Compare December 10, 2020 12:09

Snuffleupagus requested changes Dec 10, 2020

View reviewed changes

gulpfile.js Outdated Show resolved Hide resolved

gulpfile.js Outdated Show resolved Hide resolved

test/integration/annotation_spec.js Outdated Show resolved Hide resolved

Snuffleupagus reviewed Dec 10, 2020

View reviewed changes

calixteman force-pushed the interaction branch from d3cb3ab to f614e0f Compare December 10, 2020 12:56

brendandahl approved these changes Dec 10, 2020

View reviewed changes

calixteman force-pushed the interaction branch from f614e0f to cfe554f Compare December 10, 2020 18:22

calixteman force-pushed the interaction branch from cfe554f to 1e2cc1f Compare December 10, 2020 18:57

Add some integration tests using puppeteer and Jasmine

5b42ac3

* run with `gulp integrationtest`

calixteman force-pushed the interaction branch from 1e2cc1f to 5b42ac3 Compare December 10, 2020 19:55

brendandahl merged commit 31ea30a into mozilla:master Dec 10, 2020

Snuffleupagus mentioned this pull request May 13, 2021

Tests for Viewer #6505

Closed


		const describes = [];

		global.describe = (desc, filename, selector, callback) => {

		typeof PDFJSDev === "undefined" \|\|
		PDFJSDev.test("!PRODUCTION && !INTEGRATIONTEST")

Add some integration tests using puppeteer #12668

Add some integration tests using puppeteer #12668

Conversation

calixteman commented Nov 30, 2020

timvandermeij left a comment • edited Loading

Choose a reason for hiding this comment

timvandermeij Nov 30, 2020

Choose a reason for hiding this comment

calixteman Dec 1, 2020

Choose a reason for hiding this comment

brendandahl left a comment

Choose a reason for hiding this comment

brendandahl Dec 3, 2020

Choose a reason for hiding this comment

Snuffleupagus commented Dec 3, 2020 • edited Loading

timvandermeij commented Dec 8, 2020

brendandahl commented Dec 9, 2020

pdfjsbot commented Dec 9, 2020

From: Bot.io (Linux m4)

Received

pdfjsbot commented Dec 9, 2020

From: Bot.io (Linux m4)

Success

brendandahl commented Dec 9, 2020

pdfjsbot commented Dec 9, 2020

From: Bot.io (Windows)

Received

pdfjsbot commented Dec 9, 2020

From: Bot.io (Windows)

Failed

brendandahl commented Dec 10, 2020

calixteman commented Dec 10, 2020

pdfjsbot commented Dec 10, 2020

From: Bot.io (Linux m4)

Received

pdfjsbot commented Dec 10, 2020

From: Bot.io (Linux m4)

Success

Snuffleupagus Dec 10, 2020

Choose a reason for hiding this comment

calixteman Dec 10, 2020

Choose a reason for hiding this comment

Snuffleupagus Dec 10, 2020

Choose a reason for hiding this comment

pdfjsbot commented Dec 10, 2020

From: Bot.io (Windows)

Received

pdfjsbot commented Dec 10, 2020

From: Bot.io (Windows)

Success

brendandahl commented Dec 10, 2020

pdfjsbot commented Dec 10, 2020

From: Bot.io (Windows)

Received

pdfjsbot commented Dec 10, 2020

From: Bot.io (Linux m4)

Received

brendandahl left a comment

Choose a reason for hiding this comment

pdfjsbot commented Dec 10, 2020

From: Bot.io (Linux m4)

Failed

pdfjsbot commented Dec 10, 2020

From: Bot.io (Windows)

Failed

calixteman commented Dec 10, 2020

pdfjsbot commented Dec 10, 2020

From: Bot.io (Linux m4)

Received

pdfjsbot commented Dec 10, 2020

From: Bot.io (Windows)

Received

pdfjsbot commented Dec 10, 2020

From: Bot.io (Linux m4)

Failed

calixteman commented Dec 10, 2020

pdfjsbot commented Dec 10, 2020

From: Bot.io (Linux m4)

Received

pdfjsbot commented Dec 10, 2020

timvandermeij left a comment •

edited

Loading

Snuffleupagus commented Dec 3, 2020 •

edited

Loading