Puppeteer dev branch! #704

garris · 2018-03-16T16:07:36Z

This is happening!

This branch...
https://github.com/garris/BackstopJS/tree/puppeteer

is tracking this branch...
https://github.com/krisdigital/BackstopJS

Interested in contributing? ...please coordinate with @krisdigital.

What you need to know...

The Puppeteer engine option will be released as soon as we are at feature parity with Chromy and Phantom.
We will continue to support Phantom, Chromy and Puppeteer moving forward.
You are awesome.

krisdigital · 2018-03-16T19:46:04Z

Hello to all potential puppet players!

Here is a little road map for all who want to try it out or even contribute. In general, for me it works much more reliable and it is also faster.

Screnshots for viewport and document
Engine scripts onBefore and onReady, loadCookies and clickAndHover
ReadyEvent, readySelector, hide- and removeSelectors
Screenshots of selectors, selectorExpansion
let Puppeteer and Chromy coexist
start clean up by pulling common functions between Chromy and Puppeteer in a separate module
more cleanup and testing

Take it for a spin an let me know what you think 🚀

garris · 2018-03-20T18:23:25Z

Just bumped to 3.2.0 and pushed to NPM @beta!

TESTERS & COMMENTS! 🙏

npm install backstopjs@beta

Aeotrin · 2018-03-21T20:41:37Z

Works for me :) NOTE: the engine needs to be puppet
Node version requirement of > v8 as well.

This also fixed the issue of a white screenshot I was having when running against a local (.local or .dev) domain #706 .

matthiaskomarek · 2018-03-23T09:57:39Z

I use the reference urls for our test suite with the following command:

backstop reference && backstop test

And it looks like that the test suite hangs up when an error occurs. In the console i get

COMMAND | Command `reference` ended with an error after [95.102s]
COMMAND | Error: Navigation Timeout Exceeded: 60000ms exceeded
                    at Promise.then (/Users/matthiaskomarek/Documents/Projects/zurich/visual-regression/node_modules/puppeteer/lib/NavigatorWatcher.js:71:21)
                    at <anonymous>
(node:35439) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Navigation Timeout Exceeded: 60000ms exceeded
(node:35439) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

After this, it just stops running. But doesn't terminate the terminal process.

If no error occurs, everything is running fine!

krisdigital · 2018-03-23T11:17:05Z

@matthiaskomarek do you get a different behavior using Chromy in the stable branch? Not sure if it has something to do with Puppeteer..

AkA84 · 2018-03-25T06:42:30Z

Hey @krisdigital , thanks a lot for doing this! I'm testing it out and I saw that when I set debugWindow: true, there is no chrome windows opened when I run the tests. Has the param name changed?

EDIT: I also see no difference between debug: false and debug: true

krisdigital · 2018-03-25T13:23:14Z

@AkA84 Both true, did not implement the debugWindow option, but it is not a problem to do... I never could find any use for the debug option though, as it throws just too much text out, I think it only makes sense if you have no debugWindow option like phantomJS.. #713

AkA84 · 2018-03-26T07:11:46Z

@krisdigital thanks a lot again, and yeah I agree I didn't find the debug option useful unless i wanted to stare at a big dump of html for some reason!

matthiaskomarek · 2018-03-26T13:57:35Z

Hey @krisdigital i have tried the same with the current 3.1.21 branch and if something fails with the chrome engine. The script goes on and just shows an empty entry in the report.
So this is an issue (the hung up script) with the puppeteer engine.

krisdigital · 2018-03-26T14:13:46Z

@matthiaskomarek Thank's for checking, I will look into it!

AkA84 · 2018-03-26T15:02:22Z

anyone has any problems with running captures in parallel (say, 5 at a time)? I have random errors (puppeteer that can't find an element, for example) for scenarios with onReady scripts that run just fine when doing one capture at a time.

I'm still not sure if this is something wrong with my setup/scripts, but just wanted to bring it up in case someone encountered the same issue. At the moment I can rely on the results only if i don't do parallel capturing, which really is a shame :(

garris · 2018-03-26T15:19:06Z

@krisdigital this issue will probably be addressed by the work happening in #712.

garris · 2018-03-26T20:20:02Z

@AkA84 we are still working on the migration— once all of our feature tests pass we’ll be looking at why some parallel cases seem to work fine but others have issues. This could be a performance issue in backstop— but it could also be on the network, the server or something else in the environment.

iain17 · 2018-03-27T08:45:30Z

Hey @krisdigital We're super excited for puppeteer. I'm currently trying to get it to run in Docker with the latest changes from https://github.com/garris/BackstopJS/tree/puppeteer.

One thing I noticed is that we seem to create a new browser instance for each test case. We're currently experimenting with just having one browser instance and creating only new page objects (tabs) for each test. Let me know what you think.

My repo is at: https://github.com/iain17/BackstopJS/tree/puppeteer

kiran-redhat · 2018-03-27T09:09:58Z

This will increase the speed exponentially. having browser opened and keep using same instance

…

On Tue, Mar 27, 2018, 9:45 PM Iain Munro ***@***.***> wrote: Hey @krisdigital <https://github.com/krisdigital> We're super excited for puppeteer. I'm currently trying to get it to run in Docker with the latest changes from https://github.com/garris/BackstopJS/tree/puppeteer. One thing I noticed is that we seem to create a new browser instance for each test case. We're currently experimenting with just having one browser instance and creating only new page objects (tabs) for each test. Let me know what you think. My repo is at: https://github.com/iain17/BackstopJS/tree/puppeteer — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#704 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQUrI1R8a6oVu4fYTqyOwiE05fU7IR0rks5tifwvgaJpZM4SuC7d> .

krisdigital · 2018-03-27T09:40:37Z

@iain17 The current behaviour with new browser instances mirrors the behaviour of Chromy. Opening a tab is probably more ressource friendly, see this discussion: puppeteer/puppeteer#85
Maybe it could be provided as an option for users who do not need a new session for each scenario, afaik this would be also possible with Chromy.

krisdigital · 2018-03-27T20:18:58Z

@garris @matthiaskomarek Just made a pull request that should handle that problem #718

garris · 2018-03-28T01:02:25Z

📣 Puppeteer branch is merged to master and pushed to npm @canary! 🎆

npm install backstopjs@canary

krisdigital · 2018-03-28T07:47:12Z

🎉🍾🍻

cactusa · 2018-04-06T12:41:30Z

Just trying puppeteer out in v3.2.3 on a docker setup and got an error:

BackstopJS v3.2.3
Loading config:  /src/backstop.config.js 

COMMAND | Executing core for `test`
createBitmaps | Selected 45 of 45 scenarios.
      COMMAND | Command `test` ended with an error after [0.06s]
      COMMAND | Error: Failed to launch chrome!
                [0406/123126.823853:ERROR:zygote_host_impl_linux.cc(88)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
                
                
                TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md
                
                    at onClose (/usr/local/lib/node_modules/backstopjs/node_modules/puppeteer/lib/Launcher.js:246:14)
                    at Interface.helper.addEventListener (/usr/local/lib/node_modules/backstopjs/node_modules/puppeteer/lib/Launcher.js:235:50)
                    at emitNone (events.js:110:20)
                    at Interface.emit (events.js:207:7)
                    at Interface.close (readline.js:367:8)
                    at Socket.onend (readline.js:147:10)
                    at emitNone (events.js:110:20)
                    at Socket.emit (events.js:207:7)
                    at endReadableNT (_stream_readable.js:1059:12)
                    at _combinedTickCallback (internal/process/next_tick.js:138:11)
>> Exited with code: 1.
>> Error executing child process: Error: Process exited with code 1.

Anyone experiencing the same?

krisdigital · 2018-04-06T12:53:20Z

@cactusa see the link in the error: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md, especially the sandbox section, also this PR #722

cactusa · 2018-04-09T16:26:35Z

@krisdigital Just pulled the latest commit to try out the PR you linked above, but it didn't work for me, absolutely no change at all. I still get that same error. And I rebuild the docker image as well and followed the documentation from the readme.

FYI @VesterDe

garris · 2018-04-09T19:06:45Z

@krisdigital or @here Anybody know how to handle CSP issues with puppeteer? I am getting errors like...

######## Error running Puppeteer ######### Error: Evaluation failed: EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive ...

VesterDe · 2018-04-09T19:31:41Z

@cactusa Hey, did you also include the engineOptions to your config file?

Specifically, this line:

"engineOptions": {"args": ["--no-sandbox", "--disable-setuid-sandbox"]}

That solved your exact problem for me. I'm not sure how to help if that doesn't change the error, at least...

garris · 2018-04-10T02:40:20Z

@VesterDe unfortunately no. same exact behavior. 😢

This is weird -- I didn't have this problem with the Chromy engine. 🤔

krisdigital · 2018-04-10T08:30:06Z

@garris This seems to be a known issue puppeteer/puppeteer#1814 ...

cactusa · 2018-04-10T12:13:36Z

@VesterDe Yes I did indeed include that exact line in my config. 😞

garris · 2018-04-10T14:13:43Z

Thanks guys! Fixed by adding this onBefore script 👉 67801c2

garris · 2018-04-10T15:09:54Z

I am hoping this will be a much better fix in the near future! puppeteer/puppeteer#2324

cactusa · 2018-04-18T14:08:02Z

Cool.
Update from me - I finally had time to make backstop run with puppeteer, but most of my scenarios now fail for two reasons.
For some reason the font renders differently now, which shouldn't be possible if I am running a docker container with consistent versions.

The second and bigger issue us that for most scenarios the captured test image is just grey, nothing else on it. Naturally it detects close to 100% difference.

So far I have tried adding a delay to each scenario also tried with 'asyncCaptureLimit': 1
Couldn't see any other issues posted with this same problem.
Any suggestions?

VesterDe · 2018-04-18T15:14:57Z

I can confirm that I get both of those errors intermittently as well, even if running reference and test one after another on the same machine.

Are your test images gray or empty? Mine are actually just WIDTHxHEIGHT of empty pixels, maybe the gray is just the css on the diffing tool?

Do any of your reference images appear empty as well? I also get empty references sometimes...

Although, as far as I can tell, this never happens if there are no selectors present for the run. So, if I'm just capturing the entire website in one image. In that case, it's never empty...

cactusa · 2018-04-18T15:42:38Z

Not sure, but I think it's just grey image of the same size as the reference. I have spotted some of the tests images are half captured and half grey. I have one example where the test capture is of an area slightly above the intended selector...

I am also getting this error at the very start:

>> (node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit

And this error on a couple scenarios where the test image is completely missing:

ENGINE ERROR: Navigation Timeout Exceeded: 60000ms exceeded

I guess am not switching to puppeteer just yet. I was really hoping this to work because it doesn't randomly hang executions as chromy does and is faster 😢

VesterDe · 2018-04-20T11:29:41Z

Ok, I've found a bit of a curve ball here... I'd be really grateful if someone can confirm this for me.

I've isolated the failing cases on one of my projects, and it seems that the screenshot will be empty if any of the numbers pertaining to the bounding box of the selected element have decimals. So, if I log the bounding boxes these values produce a screenshot:

Screenshoting header:
{"x":0,"y":0,"width":800,"height":66}
Screenshoting footer:
{"x":0,"y":948,"width":800,"height":703}

And these produce an empty file:

Screenshoting #testimonials:
{"x":0,"y":3475.296875,"width":800,"height":140}
Screenshoting #cta1:
{"x":0,"y":421.203125,"width":800,"height":918.109375}

This is... unusual... But seems consistent on my machine. What also happens is that as soon as one selector in an array fails because of this reason, all the remaining selector in the scenario fail.

It does provide an insight into the randomness we've been seeing, because while width and height are constant, X and Y are relative to the current viewport, and may have different values on different runs.

I've tested this out on a standalone puppeteer run and it seems to happen as well. If someone could reproduce these findings we can make an issue with the puppeteer team.

Edit: Perhaps this issue will lead to a good outcome.

Edit2: Indeed, as per the above issue, Puppeteer@1.1.1 doesn't have this problem, it seems to start with 1.2.0.

krisdigital · 2018-04-20T22:18:08Z

@VesterDe Great debugging! Maybe the Puppeteer version should be locked to 1.1.1 until the bug is fixed then..

garris · 2018-04-21T01:53:47Z

@VesterDe Thank you for taking time to investigate this!

cactusa · 2018-05-09T12:34:56Z

@garris is it OK to move puppeteer to 1.1.1 as it is more stable? I can test and make a pull request.

garris · 2018-05-09T14:02:35Z

Looks like puppeteer is at 1.4.0. Yes, we definitely want to bump this.

cactusa · 2018-05-10T11:52:39Z

@garris Sadly version 1.4.0 still exhibits the same bug. Some test images are blank some are partially captured. Will try with 1.1.1 next

Update: Weirdly v1.1.1 has the exact same problem. Maybe puppeteer is not the problem here 😭

jsmecham · 2018-05-10T13:44:28Z

I am also experiencing this issue with the latest BackstopJS and Puppeteer. Sometimes the images will be blank, sometimes they will be cropped incorrectly and sometimes they will display part of the content and white for the rest of it. Have not been able to find a workaround, yet (tried delays, etc).

garris · 2018-05-10T13:56:48Z

Is this fixable by adding a Math.floor() to our capture flow?

cactusa · 2018-05-14T11:22:30Z

@garris could you point me to the file I need to the change in and I'll try to tests this as soon as I have some spare time.

garris · 2018-05-14T15:20:06Z

Here is where selector screenshots are called by backstop...

https://github.com/garris/BackstopJS/blob/master/core/util/runPuppet.js#L365

You could look into changing this to a different puppeteer call where you specify the screen coordinates explicitly using rounded boundingBox values.

cactusa · 2018-06-20T14:11:50Z

@VesterDe I have been doing some debugging of my own and I managed to reproduce the same results as you, but in my case I get the empty screenshots even for entries that do not have the decimals in their boundingBox object. I think the issue is caused by something else.

@garris I am testing a bug fix for this and will be making a pull request shortly. I have basically done what you suggested, but without rounding the numbers and it worked.

cactusa · 2018-06-20T15:55:38Z

My pull request #808

VesterDe · 2018-06-20T20:00:02Z

Cool, can't wait to test it out.

Edit: At first glance, your pull request seems to fix it for me! Great job @cactusa :)

garris added enhancement help wanted labels Mar 16, 2018

garris mentioned this issue Mar 20, 2018

Not really reliable results using engine Chromy #696

Closed

AkA84 mentioned this issue Mar 27, 2018

PCHR-3221: Migrate BackstopJS test suite to headless Chrome + Puppeteer compucorp/civihr#2535

Merged

garris mentioned this issue Apr 8, 2018

Replacing chromy with puppeteer (originally: Failing with no images in report, and no screenshot for test taken) #724

Open

garris mentioned this issue Apr 13, 2018

Inconsistent handling of script errors between the three engines #731

Closed

AkA84 mentioned this issue May 16, 2018

[New Feature Request] Retry Function #761

Open

This was referenced Jun 20, 2018

Puppeteer captures wrong location of selectors / Chromy doesn't #800

Open

Puppeteer fix for creating blank (off page) screenshots #808

Merged

jonathanloske mentioned this issue Jan 31, 2019

Puppeteer: hoverSelector not working #977

Closed

Puppeteer dev branch! #704

Puppeteer dev branch! #704

Comments

garris commented Mar 16, 2018 • edited Loading

krisdigital commented Mar 16, 2018 • edited Loading

garris commented Mar 20, 2018 • edited Loading

Aeotrin commented Mar 21, 2018 • edited Loading

matthiaskomarek commented Mar 23, 2018 • edited Loading

krisdigital commented Mar 23, 2018

AkA84 commented Mar 25, 2018 • edited Loading

krisdigital commented Mar 25, 2018

AkA84 commented Mar 26, 2018

matthiaskomarek commented Mar 26, 2018

krisdigital commented Mar 26, 2018

AkA84 commented Mar 26, 2018 • edited Loading

garris commented Mar 26, 2018

garris commented Mar 26, 2018

iain17 commented Mar 27, 2018

kiran-redhat commented Mar 27, 2018 via email

krisdigital commented Mar 27, 2018 • edited Loading

krisdigital commented Mar 27, 2018 • edited Loading

garris commented Mar 28, 2018 • edited Loading

krisdigital commented Mar 28, 2018

cactusa commented Apr 6, 2018

krisdigital commented Apr 6, 2018

cactusa commented Apr 9, 2018

garris commented Apr 9, 2018

VesterDe commented Apr 9, 2018

garris commented Apr 10, 2018 • edited Loading

krisdigital commented Apr 10, 2018

cactusa commented Apr 10, 2018

garris commented Apr 10, 2018

garris commented Apr 10, 2018

cactusa commented Apr 18, 2018

VesterDe commented Apr 18, 2018 • edited Loading

cactusa commented Apr 18, 2018 • edited Loading

VesterDe commented Apr 20, 2018 • edited Loading

krisdigital commented Apr 20, 2018

garris commented Apr 21, 2018

cactusa commented May 9, 2018

garris commented May 9, 2018

cactusa commented May 10, 2018 • edited Loading

jsmecham commented May 10, 2018

garris commented May 10, 2018

cactusa commented May 14, 2018

garris commented May 14, 2018

cactusa commented Jun 20, 2018 • edited Loading

cactusa commented Jun 20, 2018

VesterDe commented Jun 20, 2018 • edited Loading

garris commented Mar 16, 2018 •

edited

Loading

krisdigital commented Mar 16, 2018 •

edited

Loading

garris commented Mar 20, 2018 •

edited

Loading

Aeotrin commented Mar 21, 2018 •

edited

Loading

matthiaskomarek commented Mar 23, 2018 •

edited

Loading

AkA84 commented Mar 25, 2018 •

edited

Loading

AkA84 commented Mar 26, 2018 •

edited

Loading

krisdigital commented Mar 27, 2018 •

edited

Loading

krisdigital commented Mar 27, 2018 •

edited

Loading

garris commented Mar 28, 2018 •

edited

Loading

garris commented Apr 10, 2018 •

edited

Loading

VesterDe commented Apr 18, 2018 •

edited

Loading

cactusa commented Apr 18, 2018 •

edited

Loading

VesterDe commented Apr 20, 2018 •

edited

Loading

cactusa commented May 10, 2018 •

edited

Loading

cactusa commented Jun 20, 2018 •

edited

Loading

VesterDe commented Jun 20, 2018 •

edited

Loading