Selenium Server w/ PhantomJS : Trying to achieve a Visual Regression Testing Strategy

[justinlevi]

I'm in pursuit of a "Unified Development Environment", mixed Mac & Windows devs. I've been super happy with the Drupal-VM project and with everyone's help so far on my other questions. The last piece I'm trying to get in place is figuring out a strategy for visual regression testing.

@geerlingguy, Per your post here, you mentioned that you don't recommend the Selenium PhantomJS route on the VM:

#367

Yes, you can get it working inside Drupal VM with the ability to do screenshots through the phantomjs driver... but I usually recommend people either set up CI with that level of detail/automation on a separate server (e.g. Jenkins) or on their host machine, since it will use an actual browser instance, like FireFox. Drupal VM is much better imo for building the tests with a headless browser, though you can get it working either way (if you want screenshots).

I'm wondering why? From an unified development environment perspective, seems like it would be better if everyone on the team could fire up a VM, with all the necessary config, do a git pull to their share filesystem folder, maybe a database sync, and be up and running...

Continuous Integration During Development

I'd like some sort of watch task running while styling and site building that would run the behat tests and another task runner for CSS themers. I haven't really figured out when it makes logical sense to run diffs. Every save? Every commit? only on release? I could see potential benefit to each. That seems like something we can figure out given our workflow if I can get everything up and running though.

I am pretty confident I will want to run all behat tests and visual regression tests on commit as well as a pre-commit hook before deployment to prod. Any tips/pointers for how to get this setup? I'd like to do this on Travis CI but not really sure where to go from here.

In writing this all out I'm realizing I'm probably asking like 5 questions here and only a few of them are directly related to this project. This would probably also better be a blog post at this point but oh well...

Here is just some context of what I did to get things setup and working on my local host

• I'm able to run behat tests on the VM via the drupal-extension. Yay! 👍
• After getting the following prerequisites installed I'm able to generate screenshots and diffs via webdriverio, phantomjs, and webdrivercss. Yay! x 💯
  • node v5.5 on my local host (mac)
  • $ `npm install -g phantomjs@">1.0 <2.0"
  • $ `npm install -g selenium-standalone"
  • $ `npm install -g webdriverio@2.4.5
  • $ brew install imagemagick // I think this was already installed but not 100%
  • $ brew install graphicsmagick // This was a fix to an odd gm convert error

Here's my globally installed node packages:
$ npm list -g --depth=0

├── npm@3.5.3
├── phantomjs@1.9.19
├── selenium-standalone@4.9.0
├── webdriverio@2.4.5

Here's my package.json:

{
  "name": "vtests",
  "version": "1.0.0",
  "main": "webdrivercsstest.js",
  "dependencies": {
    "jquery": "^1.11.3",
    "phantomcss": "^0.10.4",
    "phantomcss-gitdiff": "^0.0.5",
    "webdrivercss": "^1.1.9"
  },
}

Here's my actual webdrivercsstest.js

var assert = require('assert');

// init WebdriverIO
var client = require('webdriverio').remote({desiredCapabilities:{browserName: 'phantomjs'}})
// init WebdriverCSS
require('webdrivercss').init(client, {
  screenWidth: [320,480,768,960,1280]
});

client
    .init()
    .url('http://drupalvm.dev')
    .webdrivercss('startpage',[
        {
            name: 'page',
            elem: '#page',
            exclude:  [
             ".container-to-exclude",
            ]
        }
    ], function(err, res) {
        assert.ifError(err);
        assert.ok(res.page[0].isWithinMisMatchTolerance);
    })
    .end();

The screen captures are generated as expected if I start the selenium server on my local machine via:

$ selenium-standalone start

and in another terminal run
$ node webdrivercsstest.js

I need to credit @lhridley https://github.com/lhridley for most of this based on the workshop she gave to the WebNY group a few months ago. This is in large part me just trying to grok the whole thing in my own way...

[geerlingguy]

I think it's mostly a matter of documentation; we could point people to how to get things adjusted so it's easier to do all the testing and screenshots within the VM.

[justinlevi]

Just for reference, to start the selenium server on the VM, this gets things running.

$ sudo /etc/init.d/selenium start

I also needed to install graphicsmagick to get the above test to run correctly

$ sudo apt-get install graphicsmagick

I have since updated my config.yml file to inlude these

extra_packages:
  - unzip
  - graphicsmagick
  - graphicsmagick-imagemagick-compat

[kevinquillen]

Just to add a comment here.. I had a lot of success using the SeleniumHQ Docker and running Selenium as its own Docker VM and letting that connect to DrupalVM. Just pass host name and IPs to the Docker client when initializing it and it will set that in its host file. It was much much easier than ssh into the DrupalVM and attempting to get all the necessary things. Plus it lets Docker act as a dedicated Selenium hub. From there I just told Behat the IP to the selenium server and it was good to go. It also comes with VNC client(s) built in so you can watch the browser run through the tests if you have doubts or need to debug. I haven't figured out how to take the next step and get it on Jenkins yet, but this lets anyone working on the project write and run their own tests locally in the VMs with no known requirements on the host machine beyond Docker and Vagrant.

I also needed to run Selenium without PhantomJS so I could test AJAX or responses that are served through BigPipe in D8 otherwise tests fail.

I am however interested in visual regression testing, and would like to try this out and see what the result is.

One thing I had an issue with was updating NodeJS in DrupalVM to 4.x+ from config.yml. Couldn't quite get that to happen

Update: setting the version in config.yml to 4.3.0 successfully installs NodeJS for me. 5.6.0 did not work.

[geerlingguy]

For this and #367, I've decided to add links to the issues in the docs for Behat, and leave it at that... I've found that many projects set up Selenium, PhantomJS, Node, Behat, and the Drupal plugins slightly differently, and there's no way I want to try to account for all the flavors in Drupal VM's docs.

Rather, as long as the base components are able to work, I point to resources like the official docs, implementations like BLT, and issues like these, and I leave it up to the reader to decide how to implement automated tests into their own projects.

[fede-green]

Hello, guys... sorry i bring back to life this old post, maybe nowadays you do things in a totally different way, but one never knows :)

Lately i decided to write again some behat tests and we were thinking of doing also visual regression tests. Now: it is pretty easy to take screenshots using behat, so my question is: do you know guys is there a way of simply comparing those screenshots with a tool in stead of writing new tests from scratch that in the end most likely reflect the same behaviors one already writes in behat/mink tests?

thanks :)

[lhridley]

Check out the library dependencies that power PhantomCSS. There is one JS library in particular that handles the screenshot comparisons. You may be able to leverage that.

https://huddle.github.io/Resemble.js/

[kentr]

I use Wraith for this. It fits in to a workflow for testing after changes.

There are some Ansible tasks for installing Wraith and running basic tests here:
https://github.com/kentr/drupal-vm/tree/dev/scripts/misc/tasks