Document using Behat with Drupal VM

[grasmash]

Draft docs

Drupal VM and Behat tests

Using the Drupal Extension's "drupal" driver with Drupal VM

The Drupal Extension for Behat has an inherent limitation: it cannot use the 'drupal' driver to bootstrap Drupal on a remote server. If you're using Drupal VM and would like to execute Behat tests using the 'drupal' driver, you must execute them from within the VM. This is a break of the established pattern of running all BLT commands outside of the VM.

To execute Behat tests using the 'drupal' driver on a Drupal VM environment, you must do the following:

1. Update tests/behat/local.yml with the absolute file path to your project inside the VM. I.e., find and replace all instances of host/machine/path/to/repo with /vm/path/to/repo, which should look something like var/www/[project.machine_name].
2. SSH into the VM vagrant ssh.
3. Change to your project directory cd /var/www/[project.machine_name].
4. Assert that PhantomJS is installed for VM: composer run-script install-phantomjs
5. Execute behat tests ./blt.sh tests:behat

Using the Drupal Extension's "drush" driver with Drupal VM

You may choose to write only behat tests that utilize the Drupal Extension's "drupal" driver.

[justinlevi]

I think there is an issue running @javascript behat tests on Drupal-vm as well right? The example feature throws the error below when running from within the stock Drupal-vm

@javascript
Feature: Selenium Test
  In order to test if Selenium is working
  As a user
  I need to be able to load the homepage

  Scenario: Load a page
    Given I am on "/"
    Then I should see the text "Log in"

vagrant@data:/var/www/data$ ./vendor/bin/behat tests/behat/features/Examples.feature --config=tests/behat/l ocal.yml -p local
@javascript
Feature: Selenium Test
  In order to test if Selenium is working
  As a user
  I need to be able to load the homepage

  Scenario: Load a page                 # features/Examples.feature:7
    Given I am on "/"                   # Drupal\DrupalExtension\Context\MinkContext::visit()
      Could not open connection: The path to the driver executable must be set by the webdriver.chrome.driver system property; for more information, see https://github.com/SeleniumHQ/selenium/wiki/ChromeDriver. The latest version can be downloaded from http://chromedriver.storage.googleapis.com/index.html (Behat\Mink\Exception\DriverException)
    Then I should see the text "Log in" # Drupal\DrupalExtension\Context\MinkContext::assertTextVisible()

--- Failed scenarios:

    features/Examples.feature:7

1 scenario (1 failed)
2 steps (1 failed, 1 skipped)
0m10.20s (24.46Mb)

I don't believe the chrome-driver is installed by default so even if you have selenium is up and running it won't work as expected.

[justinlevi]

Update: this may have been an oversight on my part. Given the following local.yml, my @javascript tests seem to execute as expected after starting up my selenium server from within the VM

sudo /etc/init.d/selenium start

# To generate a local.yml file using this the example template, execute:
# `./blt.sh setup:behat` from the project root.
imports:
  - behat.yml

local:
  suites:
    default:
      paths:
        # Set features to repo root so that .feature files belonging to contrib
        # modules, themes, and profiles can be discovered. Enable these
        # additional paths in build/phing/build.yml.
        features: /var/www/data
        bootstrap: /var/www/data/tests/behat/features/bootstrap
      contexts:
        - Drupal\FeatureContext:
          parameters:
            environment:
              # absolute path to local directory to store screenshots - do not include trailing slash
              screenshot_dir: '/Users/username/Desktop/test'
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\DrushContext
  extensions:
    Behat\MinkExtension:
      javascript_session: selenium2
      # configure the base url for your site
      base_url: http://data.vm
      # set default command for "Show last response" step.
      show_cmd: "open %s"
      goutte:
        guzzle_parameters:
          redirect.disable: true
          ssl.certificate_authority: system
          curl.options:
            CURLOPT_SSL_VERIFYPEER: false
            CURLOPT_SSL_VERIFYHOST: false
            CURLOPT_CERTINFO: false
            CURLOPT_TIMEOUT: 120
      # ADDITIONAL OPTIONS FOR WEB DRIVERS
      # use the following 3 lines to use phantomjs
      browser_name: phantomjs
      selenium2:
        wd_host: "http://localhost:4444/wd/hub"
      #  browser: chrome


    Drupal\DrupalExtension:
      blackbox: ~
      api_driver: 'drupal'
      drush:
        alias: '@data.vm'
      drupal:
        # This must be an absolute path.
        drupal_root: /var/www/data/docroot
      drush:
        alias: '@data.vm'
        global_options: '--strict=0'
      region_map:
        header: "#header"
        content: "#content"
        footer: "#footer"
      selectors:
        message_selector: '.messages'
        error_message_selector: '.messages.messages-error'
        success_message_selector: '.messages.messages-status'

# example-multisite:
#   extensions:
#     Behat\MinkExtension:
#       javascript_session: selenium2
#       # Configure the base url for your site. For multisite, ensure that this
#       # corresponds to a valid domain entry in sites.php.
#       base_url: http://127.0.0.1:8889
#       # set default command for "Show last response" step.
#       show_cmd: "open %s"
#       selenium2:
#         wd_host: http://127.0.0.1:4444/wd/hub
#         browser: chrome
#     Drupal\DrupalExtension:
#       drupal:
#         # This must be an absolute path.
#         drupal_root: /var/www/data/docroot
#       drush:
#         alias: '@self'

[justinlevi]

Took me a minute to figure out how to run the ./blt.sh behat:tests:execute command so thought I'd throw an example in here for that.

/blt.sh tests:behat:execute -D behat.path=$(pwd)/tests/behat/features

Running this should try and execute all tests in the tests/behat/features directory.

[grasmash]

@justinlevi You shouldn't actually need selenium server at all. By default, the tests are run via PhantomJS, which "spoofs" a selenium server. Running blt tests:behat will start PhantomJS on the default port and then run all Behat tests in the behat.paths array, which is set to a default value here: https://github.com/acquia/blt/blob/8.x/template/build/core/phing/build.yml#L12.

You can override the paths in build/custom/phing/build.yml.

[justinlevi]

@grasmash Thanks so much for the help! I'm still super new to behat testing so I'm sure I'm missing something obvious here.

Given the default local.yml settings, I can't seem to get the Example.feature to run.

The following command on my host machine, where I don't have selenium running, gives me the following error:

$ ./vendor/bin/behat -c tests/behat/local.yml tests/behat/features/Examples.feature

Error 💥:

 [Symfony\Component\Config\Definition\Exception\InvalidConfigurationException]                  
  The javascript session must be one of the enabled javascript sessions ([]), but got selenium2  

Here's what my local.yml looks like (which is pretty much the stock local.yml I think):

# To generate a local.yml file using this the example template, execute:
# `./blt.sh setup:behat` from the project root.
imports:
  - behat.yml

local:
  suites:
    default:
      paths:
        # Set features to repo root so that .feature files belonging to contrib
        # modules, themes, and profiles can be discovered. Enable these
        # additional paths in build/phing/build.yml.
        features: /Users/justinwinter/Sites/data
        bootstrap: /Users/justinwinter/Sites/data/tests/behat/features/bootstrap
      contexts:
        - Drupal\FeatureContext:
          parameters:
            environment:
              # absolute path to local directory to store screenshots - do not include trailing slash
              screenshot_dir: '/Users/justinwinter/Sites/data/tests'
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\DrushContext
  extensions:
    Behat\MinkExtension:
      javascript_session: selenium2
      # configure the base url for your site
      base_url: http://data.dev
      # set default command for "Show last response" step.
      show_cmd: "open %s"
      # ADDITIONAL OPTIONS FOR WEB DRIVERS
      # use the following 3 lines to user phantomjs
      browser_name: phantomjs
      selenium2:
         wd_host: "http://127.0.0.1:4444/wd/hub"
         browser: chrome
      # use the following lines to disable SSL authentication for goutte.
      goutte:
        guzzle_parameters:
          redirect.disable: true
          ssl.certificate_authority: system
          curl.options:
               CURLOPT_SSL_VERIFYPEER: false
               CURLOPT_SSL_VERIFYHOST: false
               CURLOPT_CERTINFO: false
               CURLOPT_TIMEOUT: 120
    Drupal\DrupalExtension:
      drupal:
        # This must be an absolute path.
        drupal_root: /Users/justinwinter/Sites/data/docroot
      drush:
        alias: '@self'
        global_options: '--strict=0'

The error exception doesn't mean much to me and I'm not really sure where I should be looking for more information on this. ❔

Should this Example.feature work out of the box?

Update: I was able to finally get this running on my host machine based on the post here:
http://www.inanzzz.com/index.php/post/13p9/running-behat-tests-with-phantomjs

Seems that I need specify the local profile with -p in order for it to work. 😂
./vendor/bin/behat -c tests/behat/local.yml tests/behat/features/Examples.feature -p local

I'm definitely not on solid ground but at least I have the tests running.

So the following questions remain for me:

• Should this get added to the documentation then?
• Why would I need to add the -p here?
• How do get this to run using the phing task then?
• How do I get this to work on Drupal-VM?

[justinlevi]

Always cracks me up a little when I find some answers to questions that I figured out a few months prior, in a different thread or blog post, I was part of... 😊

For reference:
geerlingguy/drupal-vm#481

Here's the gist for installing the chromedriver on your drupal-vm instance.
https://gist.github.com/justinlevi/a770db4b4f616c71bb8ddffc43bfb499

sudo apt-get install libxss1 libappindicator1 libindicator7
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

sudo dpkg -i google-chrome*.deb
sudo apt-get install -f
sudo apt-get install xvfb
sudo apt-get install unzip

wget -N http://chromedriver.storage.googleapis.com/2.22/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
chmod +x chromedriver

sudo mv -f chromedriver /usr/local/share/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver
sudo apt-get install python-pip

You should be able to download and run this via:

curl -O http://bit.ly/1W7AAP9 && chmod +x chrome-driver-install.sh && yes | ./chrome-driver-install.sh

If you have the drupal-vm selenium installed_extras uncommented and provisioned, you should then be able to run :

$ sudo /etc/init.d/selenium start

This should then allow you to run your @javascript tests using the selenium server and the chromedriver.

Still feel like it would be useful to document more from the official "Acquia Professional Services" stance, as far as the best way to approach this. I know from a windows perspective getting this all packaged up in drupal-vm would probably be preferable given the difficulties of getting all the pieces to work otherwise.

Also, I'm still unsure how to get the phing task to work with the -p local parameter and why it's necessary in my case.

*UPDATE: * Whoa... I just saw this command ./blt.sh tests:phantomjs:launch which seems to negate any need to install selenium. Wondering if if would be helpful to have a readme dedicated to some help on every phing command?

When I run
./blt.sh tests:phantomjs:launch && ./vendor/bin/behat -c tests/behat/local.yml tests/behat/features/Examples.feature -p local

The @javascript Example.feature runs as expected.

[grasmash]

@justinlevi Running blt tests:behat will automatically start Phantom JS and run Behat. So, you should never need to directly run tests:phantomjs:launch or tests:behat:execute or install Selenium or the Chrome driver. Do you encounter an error when simply running blt tests:behat ?

[grasmash]

To answer your question regarding -p local, it simply instructs behat to use the "local" profile, which should exist in tests/behat/local.yml (automatically generated by the local:setup target) which is populated using this example file: https://github.com/acquia/blt/blob/8.x/template/tests/behat/example.local.yml#L6

[grasmash]

I've verified that running the following commands results in a brand-new site with a fully functioning suite of Behat tests:

cd blt
blt clean
blt configure
blt create
cd ../blted8
blt vm:init
vagrant up
blt local:setup
# update tests/behat/local.yml to point to docroot _within_ VM
vagrant ssh
cd /var/www/blted8
composer run-script install-phantom
blt tests:behat

Notice two things:

• composer run-script install-phantom. This is a new script I've added which allows you to quickly install phantomJS within the VM. Unfortunately, when you composer install outside the VM it will install PhantomJS to the host (kind of).
• "update tests/behat/local.yml to point to docroot within VM" is a manual step that should be added to vm:init.

[justinlevi]

@grasmash Again, thanks for your help.

So I have three feature files in my tests/behat/features folder:

best-practices.feature

Feature: Best Practices
  In order to assure drupal is setup and working the way it should
  As developers
  We need to be aware if we forgot to do something critical

  Scenario: Node Modules should not be accessible to anonymous users
    # Drupal\DrupalExtension\Context\MinkContext::visit()
    Given I am on "node"
    # Drupal\DrupalExtension\Context\MinkContext::assertResponseStatus()
    Then the response status code should be 404

 # @api @database @terms
 # Scenario: Taxonomy/term/%id pages are NOT accessible
    # Drupal\DrupalExtension\Context\DrupalContext::createTerm()
 #   Given a "Category" term with the name "test"
    # FeatureContext::theResponseStatusCodeForTheTaxonomyTermPageShouldBe($term, $code)
 #   Then the response status code for the taxonomy/term "test" page should be 403
    # The page should return an access denied response

drush-api.feature

@api
Feature: Drush driver
  In order to show functionality added by the Drush driver
  As a authenticated user
  I need to be able to edit my account

  Scenario: Drush alias
    Given I am logged in as a user with the "authenticated user" role
    When I click "My account"
    Then I should see the link "Edit"

And the Bolt included Examples.feature

@javascript
Feature: Selenium Test
  In order to test if Selenium is working
  As a user
  I need to be able to load the homepage

  Scenario: Load a page
    Given I am on "/"
    Then I should see the text "Data"

# @todo Add @api example.

When I run $ ./blt.sh tests:behat, from within a Drupal-vm ssh session, my best-practice feature passes. This is using the MinkContext with the phantomjs driver?

The other two features throw a ton of errors. copy/pasting below - sorry for the length:

 vagrant@data:/var/www/data$ ./blt.sh tests:behat
Buildfile: /var/www/data/build/custom/phing/build.xml
     [echo] You are running BLT with xdebug enabled. This has a major impact on
        runtime performance.
 [property] Loading /var/www/data/project.yml
 [property] Loading /var/www/data/build/core/phing/build.yml
     [copy] default.local.settings.php omitted, is up to date
     [echo] Executing commands against multisite "default"
     [echo] Using settings file /var/www/data/docroot/sites/default/settings/local.settings.php
 [property] Loading /var/www/data/build/custom/phing/build.yml

project > tests:run-server:

     [echo] NOT using `drush runserver' for tests.

project > tests:phantomjs:launch:

     [echo] Launching PhantomJS GhostDriver.

project > tests:behat:

     [echo] You are running BLT with xdebug enabled. This has a major impact on
        runtime performance.
 [property] Loading /var/www/data/project.yml
 [property] Loading /var/www/data/build/core/phing/build.yml
     [copy] default.local.settings.php omitted, is up to date
     [echo] Executing commands against multisite "default"
     [echo] Using settings file /var/www/data/docroot/sites/default/settings/local.settings.php
 [property] Loading /var/www/data/build/custom/phing/build.yml

project > tests:behat:execute:

     [echo] No Behat tests were found in /var/www/data/docroot/profiles
     [echo] You are running BLT with xdebug enabled. This has a major impact on
        runtime performance.
 [property] Loading /var/www/data/project.yml
 [property] Loading /var/www/data/build/core/phing/build.yml
     [copy] default.local.settings.php omitted, is up to date
     [echo] Executing commands against multisite "default"
     [echo] Using settings file /var/www/data/docroot/sites/default/settings/local.settings.php
 [property] Loading /var/www/data/build/custom/phing/build.yml

project > tests:behat:execute:

     [exec] /var/www/data/tests/behat/features/best-practices.feature
    [behat] Running '/var/www/data/vendor/bin/behat /var/www/data/tests/behat --config=/var/www/data/tests/behat/local.yml --tags=~ajax --format=pretty --format=junit  --out=std --out=/var/www/data/reports  --profile=local --strict --verbose --colors'
Feature: Best Practices
  In order to assure drupal is setup and working the way it should
  As developers
  We need to be aware if we forgot to do something critical

  Scenario: Node Modules should not be accessible to anonymous users # features/best-practices.feature:6
    Given I am on "node"                                             # Drupal\DrupalExtension\Context\MinkContext::visit()
    Then the response status code should be 404                      # Drupal\DrupalExtension\Context\MinkContext::assertResponseStatus()

@api
Feature: Drush driver
  In order to show functionality added by the Drush driver
  As a authenticated user
  I need to be able to edit my account

  Scenario: Drush alias                                               # features/drush-api.feature:7
    Given I am logged in as a user with the "authenticated user" role # Drupal\DrupalExtension\Context\DrupalContext::assertAuthenticatedByRole()
      exception 'Exception' with message 'Failed to log in as user 'sC18pSXU' with role 'authenticated user'' in /var/www/data/vendor/drupal/drupal-extension/src/Drupal/DrupalExtension/Context/RawDrupalContext.php:430
      Stack trace:
      #0 /var/www/data/vendor/drupal/drupal-extension/src/Drupal/DrupalExtension/Context/DrupalContext.php(64): Drupal\DrupalExtension\Context\RawDrupalContext->login()
      #1 [internal function]: Drupal\DrupalExtension\Context\DrupalContext->assertAuthenticatedByRole('authenticated u...')

      # etc, etc, etc, .... Truncating here so this thread is actually a little easier to follow. 

      #30 /var/www/data/vendor/behat/behat/bin/behat(31): Symfony\Component\Console\Application->run()
      #31 {main}
    When I click "My account"                                         # Drupal\DrupalExtension\Context\MinkContext::assertClick()
    Then I should see the link "Edit"                                 # Drupal\DrupalExtension\Context\MinkContext::assertLinkVisible()

@javascript
Feature: Selenium Test
  In order to test if Selenium is working
  As a user
  I need to be able to load the homepage

  Scenario: Load a page               # features/Examples.feature:7
    Given I am on "/"                 # Drupal\DrupalExtension\Context\MinkContext::visit()
      exception 'WebDriver\Exception\UnknownError' with message 'The path to the driver executable must be set by the webdriver.chrome.driver system property; for more information, see https://github.com/SeleniumHQ/selenium/wiki/ChromeDriver. The latest version can be downloaded from http://chromedriver.storage.googleapis.com/index.html' in /var/www/data/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:155
      Stack trace:
      #0 /var/www/data/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php(140): WebDriver\Exception::factory(13, 'The path to the...')
      #1 /var/www/data/vendor/instaclick/php-webdriver/lib/WebDriver/WebDriver.php(77): WebDriver\AbstractWebDriver->curl('POST', '/session', Array, Array)

      # etc, etc, etc, .... Truncating here so this thread is actually a little easier to follow. 

      #36 /var/www/data/vendor/symfony/console/Application.php(123): Behat\Testwork\Cli\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
      #37 /var/www/data/vendor/behat/behat/bin/behat(31): Symfony\Component\Console\Application->run()
      #38 {main}

      Next exception 'Behat\Mink\Exception\DriverException' with message 'Could not open connection: The path to the driver executable must be set by the webdriver.chrome.driver system property; for more information, see https://github.com/SeleniumHQ/selenium/wiki/ChromeDriver. The latest version can be downloaded from http://chromedriver.storage.googleapis.com/index.html' in /var/www/data/vendor/behat/mink-selenium2-driver/src/Behat/Mink/Driver/Selenium2Driver.php:306
      Stack trace:
      #0 /var/www/data/vendor/behat/mink/src/Behat/Mink/Session.php(70): Behat\Mink\Driver\Selenium2Driver->start()
      #1 /var/www/data/vendor/behat/mink/src/Behat/Mink/Mink.php(117): Behat\Mink\Session->start()

      # etc, etc, etc, .... Truncating here so this thread is actually a little easier to follow. 

      #34 /var/www/data/vendor/behat/behat/bin/behat(31): Symfony\Component\Console\Application->run()
      #35 {main}
    Then I should see the text "Data" # Drupal\DrupalExtension\Context\MinkContext::assertTextVisible()

--- Failed scenarios:

    features/drush-api.feature:7
    features/Examples.feature:7

3 scenarios (1 passed, 2 failed)
7 steps (2 passed, 2 failed, 3 skipped)
0m40.02s (47.59Mb)
  [foreach] /var/www/data/build/core/phing/tasks/tests.xml:19:8: /var/www/data/build/core/phing/tasks/tests.xml:35:12: /var/www/data/build/core/phing/tasks/tests.xml:39:26: One ore more Behat tests failed.

BUILD FAILED
/var/www/data/build/core/phing/tasks/tests.xml:10:31: Execution of the target buildfile failed. Aborting.

Total time: 43.9003 seconds

[grasmash]

Can you confirm that you ran composer install from within the VM?

[justinlevi]

Yes, definitely. Just re-ran composer install then ./blt.sh tests:behat with the same results.

[justinlevi]

@grasmash I was able to go through the steps you listed above, with a couple of small modifications, and get all behat including @javascript tests to run.

cd blt
./blt.sh clean
./blt.sh configure

# I updated my project.yml local > hostname to `${project.machine_name}.vm` due to a conflict with my local dnsmasq settings not liking *.localhost

./blt.sh create
cd ../blted8
./blt.sh vm:init
vagrant up
./blt.sh local:setup

# update tests/behat/local.yml to point to docroot _within_ VM - I also had to update all the paths in local.yml to be relative to the server root on the VM  from `/Users/justinwinter/Sites/blted8/` to `/var/www/blted8`

vagrant ssh

# I have been running a lot of VMs lately on 192.168.88.88 so my ~/.ssh/known_hosts was giving me some errors so I had to remove entries here. 

cd /var/www/blted8
composer run-script install-phantomjs #NOTE: the run-script is install-phantomjs not install-phantom
./blt.sh tests:behat

I am still perplexed why I couldn't get this to run from within my own Drupal-VM instance though so I might need to try and go back and compare a bit more. Worth noting is that I did not use the ./blt.sh vm:init as that wasn't available when I first started playing around.

So this process allows you to run the tests from within the VM. Is there a way to run these tests outside the VM then? Also, this all worked on my mac but getting this all to work on a Windows 7 box is still proving elusive.

[grasmash]

@justinlevi If you'd like to run tests outside of the VM, you can only run Behat tests that using the 'drush' driver. This is an inherent limitation of the Behat Extension. See "Run tests and site on different servers" values in https://behat-drupal-extension.readthedocs.io/en/3.1/drivers.html.

Windows is not officially supported by BLT, though we accept pull requests to improve compatibility.

[grasmash]

Docs were added in #186.