Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add explanation for testing on travis #57

Merged
merged 7 commits into from
Mar 20, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 100 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -437,7 +437,7 @@ and uploaded to S3 when tests succeed.

### Running your Nightwatch tests on your _Continuous Integration_ (CI)

####Running your Nightwatch tests on CI is easy on CodeShip.
#### Running your Nightwatch tests on CI is easy on CodeShip.
We usually set the required (_minimum_) node version in our
`package.json` e.g:
```js
Expand All @@ -459,12 +459,106 @@ npm test
```
That's it.

> ***Note***: while the tests run seamlessly on CodeShip we were unable
to get Selenium standalone working on Travis-CI
if you have time to ***help us***, please see:
https://github.com/dwyl/learn-nightwatch/issues/8
#### Running your Nightwatch tests on Travis-CI with sauce connect
Since we are testing on the localhost we have to make sure that the server is started before the tests are run and closes after the tests finish. So we need to boot up a server to serve our content. Travis makes this easy enough via a before_script task. In the task we will just start a python simple server and give it a few seconds to boot. The ampersand at the end of the python line tells travis to run the process in the background instead of blocking the execution thread, allowing us to run tasks at the same time.

####Running your Nightwatch tests on CircleCi.
```yml
language: node_js
before_script:
- python -m SimpleHTTPServer &
- sleep 2
node_js:
- "6.0"
```

One other way to run a server before running a test is to use the `before` and `after` methods present in nightwatch.

```js
module.exports = {
before: function (browser, done) {
server = require('../server')(done) // done is a callback that executes when the server is started
},

after: function () {
server.close()
},

'Demo test': function (browser) {
browser
.url('localhost:3000') // visit the local url
.waitForElementVisible('body'); // wait for the body to be rendered

browser
.assert.containsText('body','hello') // assert contains
.saveScreenshot(conf.imgpath(browser) + 'dwyl.png')
.end()
}
}
```

The `server.js` can be a simple express server.

```js
function makeServer(done) {
var express = require('express');
var path = require('path');
var app = express();

app.get('/', function (req, res) {
res.status(200).sendFile(`index.html`, {root: path.resolve()});
});
var server = app.listen(3000, function () {
var port = server.address().port;
done()
});
return server;
}
module.exports = makeServer;
```
**Note** : In the above example you can see that the port is fixed. It will run fine if you are running tests on single device but in case you are running tests on multiple devices on saucelabs, this will give you an error that the port is already in use as all the devices try to start the server on same port (in our current approach). So we need to dynamically allot available ports to prevent this error. You can use [get-port](https://github.com/sindresorhus/get-port) for this.

This is all we need to run a test on browser/s. Now we have set up saucelabs on travis.

To run the test on Travis-CI and use sauce connect you need to add a addon to you .travis.yml
```
addons:
sauce_connect: true
```

The `username` and `access_key` can be optionally stored in `.travis.yml` or can be stored on travis-ci website as environment variables. There are various methods of storing the `username` and `access_key` of saucelabs and you can read more about them [here](https://docs.travis-ci.com/user/sauce-connect/). In our case we have preferred to save it on travis website so that our `.travis.yml` is simple.

Now you have to make some changes in `nightwatch.conf.js`

```js
const TRAVIS_JOB_NUMBER = process.env.TRAVIS_JOB_NUMBER;
// in test_settings.default:
default: {
launch_url: 'http://ondemand.saucelabs.com:80',

username : process.env.SAUCE_USERNAME,
access_key : process.env.SAUCE_ACCESS_KEY,
...
desiredCapabilities: {
build: `build-${TRAVIS_JOB_NUMBER}`,
'tunnel-identifier': TRAVIS_JOB_NUMBER,
},
}
```
See the modified final config [here](./nightwatch.conf.TRAVIS.js)
You can run multiple test commands i.e.
```
- npm run test:unit; npm run test:e2e
```
You can see the working code [here](https://github.com/ritz078/embed.js/pull/228/files) and the corresponding test on travis [here](https://travis-ci.org/ritz078/embed.js/builds/211089816)

**Note-1**: Tests on the PRs of _forked repos_ will fail as the secured environment variables are not accessible to them on travis. You will recieve authentication error in that case.

**Note-2**: Running tests on IE still seems tricky. Will have to explore more. Any help is appreciated.

**Note-3**: If you are recieving timeout error, maybe you are running tests on many devices. Try to adjust the time or decrease the number of devices.


#### Running your Nightwatch tests on CircleCi.
To run the test on circle ci you need to make some adjustments to you circle.yml
Here is an Example from the circle ci [docs](https://circleci.com/docs/browser-testing-with-sauce-labs/)
```
Expand Down
160 changes: 160 additions & 0 deletions nightwatch.conf.TRAVIS.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
require('env2')('.env'); // optionally store your environment variables in .env
const PKG = require('./package.json'); // so we can get the version of the project
const BINPATH = './node_modules/nightwatch/bin/'; // change if required.
const SCREENSHOT_PATH = "./node_modules/nightwatch/screenshots/" + PKG.version + "/";

const config = { // we use a nightwatch.conf.js file so we can include comments and helper functions
"src_folders": [
"test/e2e" // we use '/test' as the name of our test directory by default. So 'test/e2e' for 'e2e'.
],
"output_folder": "./node_modules/nightwatch/reports", // reports (test outcome) output by Nightwatch
"selenium": {
"start_process": true,
"server_path": BINPATH + "selenium.jar", // downloaded by selenium-download module (see below)
"log_path": "",
"host": "127.0.0.1",
"port": 4444,
"cli_args": {
"webdriver.chrome.driver" : BINPATH + "chromedriver" // also downloaded by selenium-download
}
},
"test_workers" : {"enabled" : true, "workers" : "auto"}, // perform tests in parallel where possible
"test_settings": {
"default": {
"launch_url": "http://ondemand.saucelabs.com:80", // we're testing a local site on Saucelabs
"selenium_port": 80,
"selenium_host": "ondemand.saucelabs.com",
"silent": true,
"screenshots": {
"enabled": true, // save screenshots to this directory (excluded by .gitignore)
"path": SCREENSHOT_PATH
},
"username" : process.env.SAUCE_USERNAME, // if you want to use Saucelabs remember to
"access_key" : process.env.SAUCE_ACCESS_KEY, // export your environment variables (see readme)
"globals": {
"waitForConditionTimeout": 10000 // wait for content on the page before continuing
},
"desiredCapabilities": {
"tunnel-identifier": process.env.TRAVIS_JOB_NUMBER, // needed for sauce-connect, i.e for testing localhost on saucelabs
build: `build-${process.env.TRAVIS_JOB_NUMBER}` // needed for sauce-connect
}
},
"local": {
"launch_url": "http://localhost",
"selenium_port": 4444,
"selenium_host": "127.0.0.1",
"silent": true,
"screenshots": {
"enabled": true, // save screenshots taken here
"path": SCREENSHOT_PATH
}, // this allows us to control the
"globals": {
"waitForConditionTimeout": 15000 // on localhost sometimes internet is slow so wait...
},
"desiredCapabilities": {
"browserName": "chrome",
"chromeOptions": {
"args": [
`Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) AppleWebKit/534.46
(KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3`,
"--window-size=640,1136" // iphone 5
]
},
"javascriptEnabled": true,
"acceptSslCerts": true
}
},
"chrome": { // your local Chrome browser (chromedriver)
"desiredCapabilities": {
"browserName": "chrome",
"javascriptEnabled": true,
"acceptSslCerts": true
}
},
"chromemac": { // browsers used on saucelabs:
"desiredCapabilities": {
"browserName": "chrome",
"platform": "OS X 10.11",
"version": "47"
}
},
"ie11": {
"desiredCapabilities": {
"browserName": "internet explorer",
"platform": "Windows 10",
"version": "11.0"
}
},
"firefox" : {
"desiredCapabilities": {
"platform": "XP",
"browserName": "firefox",
"version": "33"
}
},
"internet_explorer_10" : {
"desiredCapabilities": {
"platform": "Windows 7",
"browserName": "internet explorer",
"version": "10"
}
},
"android_s4_emulator": {
"desiredCapabilities": {
"browserName": "android",
"deviceOrientation": "portrait",
"deviceName": "Samsung Galaxy S4 Emulator",
"version": "4.4"
}
},
"iphone_6_simulator": {
"desiredCapabilities": {
"browserName": "iPhone",
"deviceOrientation": "portrait",
"deviceName": "iPhone 6",
"platform": "OSX 10.10",
"version": "8.4"
}
}
}
}
module.exports = config;

/**
* selenium-download does exactly what it's name suggests;
* downloads (or updates) the version of Selenium (& chromedriver)
* on your localhost where it will be used by Nightwatch.
*/
require('fs').stat(BINPATH + 'selenium.jar', function (err, stat) { // got it?
if (err || !stat || stat.size < 1) {
require('selenium-download').ensure(BINPATH, function(error) {
if (error) throw new Error(error); // no point continuing so exit!
console.log('✔ Selenium & Chromedriver downloaded to:', BINPATH);
});
}
});

function padLeft (count) { // theregister.co.uk/2016/03/23/npm_left_pad_chaos/
return count < 10 ? '0' + count : count.toString();
}

var FILECOUNT = 0; // "global" screenshot file count
/**
* The default is to save screenshots to the root of your project even though
* there is a screenshots path in the config object above! ... so we need a
* function that returns the correct path for storing our screenshots.
* While we're at it, we are adding some meta-data to the filename, specifically
* the Platform/Browser where the test was run and the test (file) name.
*/
function imgpath (browser) {
var a = browser.options.desiredCapabilities;
var meta = [a.platform];
meta.push(a.browserName ? a.browserName : 'any');
meta.push(a.version ? a.version : 'any');
meta.push(a.name); // this is the test filename so always exists.
var metadata = meta.join('~').toLowerCase().replace(/ /g, '');
return SCREENSHOT_PATH + metadata + '_' + padLeft(FILECOUNT++) + '_';
}

module.exports.imgpath = imgpath;
module.exports.SCREENSHOT_PATH = SCREENSHOT_PATH;