Skip to content

Commit d9b8abe

Browse files
davidjnelsondavidjnelson
committed
initial commit
1 parent 1e40762 commit d9b8abe

29 files changed

+1969
-2
lines changed

‎.gitignore

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
2+
.DS_Store
3+
.idea
4+
node_modules
5+
/npm-debug.log
6+
7+
css-visual-test/component-library/
8+
css-visual-test/ApplitoolsEyesParallelDebug.log
9+
css-visual-test/lib/PrivateConfig.js

‎Makefile

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# applications
2+
NODE ?= node
3+
NPM ?= $(NODE) $(shell which npm)
4+
5+
# The `install` task is the default rule in the Makefile.
6+
install: node_modules
7+
8+
# ensures that the `node_modules` directory is installed and up-to-date with
9+
# the dependencies listed in the "package.json" file.
10+
node_modules: package.json
11+
@$(NPM) prune
12+
@$(NPM) install
13+
@touch node_modules
14+
15+
project-unit-test: install
16+
node_modules/.bin/mocha css-visual-test/test
17+
18+
# run the component library server, building the public app first
19+
componentlibrary: install
20+
@$(NODE) css-visual-test/component-library-generator.js
21+
@$(NODE) css-visual-test/component-library-server.js
22+
23+
# run the css visual test tool, building the public app first
24+
visualtest: install
25+
@$(NODE) css-visual-test/component-library-server.js &
26+
@$(NODE) css-visual-test/run-visual-tests.js

‎README.md

+160-2
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,160 @@
1-
# css-visual-test
2-
Test your front end components visually. Currently supports components built with react and commonjs / browserify.
1+
Css Visual Test
2+
===============
3+
4+
Overall Goal
5+
------------
6+
7+
How do we know that if css changes for all the components comprising an application, that the application looks as expected for every state mutation the application supports?
8+
9+
Usage
10+
-----
11+
12+
`make componentlibrary`
13+
14+
This will run a build, then generate the component library and a server to view it on port 4000 to be used for visual testing, manual inspection to understand the codebase, and/or debugging creation of visual tests. Running node css-visual-test/component-library-server.js will regenerate the component library if needed as well if you change something.
15+
16+
`make visualtest`
17+
18+
This will run all the visual tests using Applitools Eyes for image comparison, and Sauce Labs for browser automation. If differences are noted between runs or new visual tests are added, a failed test is reported. When tests fail, make returns an Error 1 code, which will fail the build. This will run the componentlibrary task along with the component library server if it's not already running before it runs.
19+
20+
You'll need to create an account at both Sauce Labs and Applitools to use this tool. They both have free trials.
21+
22+
[Sauce Labs Signup Page](http://saucelabs.com/signup)
23+
24+
[Applitools Signup Page](https://applitools.com/sign-up)
25+
26+
From the Applitools UI you can approve a new test as a baseline, or see any screenshots marked as failures. You can also change the algorithm used for image comparison, mark areas of the viewport to ignore from the screenshot, and apply changes to the configuration in batches.
27+
28+
Open Files Limit
29+
----------------
30+
31+
It is recommended to increase your open files limit before running massively parallelized tests. Do this with this command: `ulimit -n 8192`
32+
33+
Creating New Visual Tests
34+
-------------------------
35+
36+
To create new visual tests, create a `visualtests` directory as a subdirectory under any directory that is a child of the `client` directory. Create any number of files you want with the format `<direction>-<testDataNumber>.jsx` where `direction` is either `ltr` for right to left languages, `rtl` for right to left languages, and `testDataNumber` is any number from 1 to n which represents a variation of the component's data that you would like to test.
37+
38+
For example, if you look at the client/card directory, you can see a subdirectory called visualtests there. You can see four files:
39+
40+
- ltr-1.jsx is an instantiation of the card component with a single word "test" in left to right direction.
41+
- ltr-2.jsx is an instantiation of the card component with a roughly fifty repetitions of the word "test" in left to right
42+
direction.
43+
- rtl-1.jsx is an instantiation of the card component with a single word "test" in right to left direction.
44+
- rtl-2.jsx is an instantiation of the card component with a roughly fifty repetitions of the word "test" in right to left
45+
direction.
46+
47+
Here is an image which shows this:
48+
49+
![Visual Tests Directory Structure](https://cloud.githubusercontent.com/assets/381633/7289360/c9d8ac5e-e923-11e4-9101-e8fea1e5c45c.png)
50+
51+
The test runner will automatically pick up files if you follow this format. There is no need to create a separate manifest file listing tests. Also, you can write the component bootstrap files ( ie: ltr-1.jsx ) just like you would in the real application.
52+
53+
The jsx will be transpiled, the requires will be resolved. The css is from the output of the scss being compiled via make build. The template to host the components in the component library uses a jade template.
54+
55+
Configuring Secret Keys
56+
-----------------------
57+
58+
Add a file to `css-visual-test/lib` called `PrivateConfig.js`, as in `css-visual-test/lib/PrivateConfig.js`.
59+
60+
Copy and paste the following and fill in your Sauce Labs username and access key, and your Applitools access key:
61+
62+
module.exports = {
63+
sauceLabsUsername: '',
64+
sauceLabsAccessKey: '',
65+
applitoolsEyesAccessKey: ''
66+
};
67+
68+
69+
Running Visual Tests That Pass
70+
------------------------------
71+
72+
`make visualtest` output:
73+
74+
![Make visualtest output](https://cloud.githubusercontent.com/assets/381633/7289373/1489c706-e924-11e4-89d2-ae825a4a79eb.png)
75+
76+
Sauce Labs selenium grid runner status UI output:
77+
78+
![Sauce Labs selenium grid runner status UI output](https://cloud.githubusercontent.com/assets/381633/7289394/46e2b168-e924-11e4-9b17-5b4bd31bbcc9.png)
79+
80+
A single passing test for one environment in the Applitools Eyes test runner status UI:
81+
82+
![A single passing test for one environment in the Applitools Eyes test runner status UI](https://cloud.githubusercontent.com/assets/381633/7289693/739ff798-e928-11e4-9a17-497cf4643c95.png)
83+
84+
Running visual tests that fail
85+
-------------------------------------
86+
87+
After changing the `public/style.css` from:
88+
89+
@media only screen and (min-width: 480px) {
90+
.card {
91+
margin-bottom: 16px;
92+
padding: 24px; } }
93+
94+
to:
95+
96+
@media only screen and (min-width: 480px) {
97+
.card {
98+
margin-bottom: 16px;
99+
padding: 240px; } }
100+
101+
`make visualtest` output:
102+
103+
![screen shot 2015-04-08 at 6 10 47 pm](https://cloud.githubusercontent.com/assets/381633/7289786/ae96aa80-e929-11e4-80b1-3bca406c47a5.png)
104+
105+
Applitools Eyes test runner status UI shows the failure:
106+
107+
![screen shot 2015-04-08 at 6 12 00 pm](https://cloud.githubusercontent.com/assets/381633/7289805/ede6befa-e929-11e4-85c3-f52d5f27b35a.png)
108+
109+
A single failing test for one environment in the Applitools Eyes test runner status UI:
110+
111+
![screen shot 2015-04-08 at 6 13 00 pm](https://cloud.githubusercontent.com/assets/381633/7289815/1ca37c74-e92a-11e4-9e62-ab5ddb365ff8.png)
112+
113+
Generating Your CSS
114+
-------------------
115+
116+
It's helpful sometimes to use a tool like less or sass to aid css development. This tool allows you to plug that in
117+
however you see fit, but doesn't require one. Just output your left to right stylesheet into the `public/style.css`
118+
file and your right to left stylesheet into the `public/style-rtl.css` using whatever tool you choose to use.
119+
120+
Running The Project's Unit Tests
121+
--------------------------------
122+
123+
`make project-unit-test`
124+
125+
Example Project
126+
---------------
127+
128+
The included example project uses a few tools you might want to swap out with your own preferred tools:
129+
130+
- commonjs
131+
- browserify
132+
- react
133+
- make
134+
135+
Cost Considerations For Various Configurations
136+
----------------------------------------------
137+
138+
The ideal way to use the tool is to purchase as many parallel vms as you can from Sauce Labs, and as many licenses as you
139+
can from Applitools. If this is too expensive, you have various options to reduce costs:
140+
141+
- Use less parallel vms. This will make your test suite run slower.
142+
- Only run the Applitools tests on merges to master. This will give you slower feedback cycles.
143+
144+
Future Goals ( Pull Requests Appreciated! ) :-)
145+
------------------------------------------------
146+
147+
- A mode to only run by comparing master to current branch for local usage, and non merge to master ci usage.
148+
- Simplifying the jsx naming scheme.
149+
- Add plugin hooks for non commonjs dependency management systems.
150+
- Add plugin hooks for non react based projects.
151+
- Add an example project for angular.
152+
- Add a file system watcher to regenerate the component library when css, js, and templates are changed.
153+
- Add an integration test using the local only run feature.
154+
- Add a functional test using Sauce Labs And Applitools.
155+
- Group all components into a single page which is used for batch diffing the components with links to each individual page still.
156+
- Create a shared layout wrapper component for each component on the index grouped page, host this as it’s own npm project to allow selective upgrades.
157+
- For the react / commonjs example, load react from the host pages while still making them work from the require calls, to speed generation time of the component library.
158+
- Add a feature where you can have less vms than selenium environments in case you want to run many os / browser / device combos but not pay for a ton of parallel vms.
159+
- Change the time estimation calculation to account for vm startup time, which is what takes the longest.
160+
- Add a debug or local mode where if a sauce connect tunnel is already open it doesn’t try to reopen it

‎client/card/card.jsx

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
var React = require( 'react/addons' );
2+
3+
module.exports = React.createClass( {
4+
displayName: 'Card',
5+
6+
render: function() {
7+
return React.createElement( 'div', { className: 'card' }, this.props.children );
8+
}
9+
} );

‎client/card/visualtests/ltr-1.jsx

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
var React = require( 'react' ),
2+
Card = require( 'card/card' );
3+
4+
React.render( <Card>test</Card>, document.getElementById( 'root' ) );

‎client/card/visualtests/ltr-2.jsx

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
var React = require( 'react' ),
2+
Card = require( 'card/card' );
3+
4+
React.render( <Card>test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test </Card>, document.getElementById( 'root' ) );

‎client/card/visualtests/rtl-1.jsx

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
var React = require( 'react' ),
2+
Card = require( 'card/card' );
3+
4+
React.render( <Card>test</Card>, document.getElementById( 'root' ) );

‎client/card/visualtests/rtl-2.jsx

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
var React = require( 'react' ),
2+
Card = require( 'card/card' );
3+
4+
React.render( <Card>test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test </Card>, document.getElementById( 'root' ) );

‎css-visual-test/Makefile

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
REPORTER ?= spec
2+
MOCHA ?= ../node_modules/.bin/mocha
3+
4+
# In order to simply stub modules, add test to the NODE_PATH
5+
test:
6+
@NODE_ENV=test $(MOCHA) --reporter $(REPORTER)
7+
8+
.PHONY: test

‎css-visual-test/SupportedSeleniumEnvironments.js

+63
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
// The highest value here for the lowest cost at the highest speed is to use the most popular operations systems,
2+
// and the most popular rendering engines across six parallel virtual machines on sauce labs:
3+
4+
// Windows 7 Chrome 41/Latest, Blink rendering engine, Windows 7 operating system
5+
// Windows 7 Firefox 37/Latest, Gecko rendering engine
6+
// Windows 7 IE 11/Latest, Trident Rendering Engine
7+
// IOS Safari 8.2/Latest, Webkit Rendering Engine, IOS operating system
8+
// Android 4.4, Android operating system
9+
// OSX Chrome 41/Latest, OSX Operating System
10+
module.exports = {
11+
"environments": [
12+
{
13+
"browserName": "chrome",
14+
"platform": "Windows 7",
15+
"version": "41.0",
16+
"commandTimeout": 600,
17+
"idleTimeout": 600
18+
}
19+
// Applitools free plan allows for one concurrent vm and sauce labs free plan allows for two concurrent vms.
20+
// Therefore, leave the most useful variations commented out in this file. You can add as many as you want
21+
// using the Sauce Labs Platform Configurator Tool ( select nodejs ): https://docs.saucelabs.com/reference/platforms-configurator/#/
22+
/*,
23+
{
24+
"browserName": "firefox",
25+
"platform": "Windows 7",
26+
"version": "37.0",
27+
"commandTimeout": 600,
28+
"idleTimeout": 600
29+
},
30+
{
31+
"browserName": "internet explorer",
32+
"platform": "Windows 7",
33+
"version": "11.0",
34+
"commandTimeout": 600,
35+
"idleTimeout": 600
36+
},
37+
{
38+
"browserName": "iphone",
39+
"platform": "OS X 10.10",
40+
"version": "8.2",
41+
"deviceName": "iPhone Simulator",
42+
"device-orientation": "portrait",
43+
"commandTimeout": 600,
44+
"idleTimeout": 600
45+
},
46+
{
47+
"browserName": "android",
48+
"platform": "Linux",
49+
"version": "4.4",
50+
"deviceName": "Android Emulator",
51+
"device-orientation": "portrait",
52+
"commandTimeout": 600,
53+
"idleTimeout": 600
54+
},
55+
{
56+
"browserName": "chrome",
57+
"platform": "OS X 10.10",
58+
"version": "41.0",
59+
"commandTimeout": 600,
60+
"idleTimeout": 600
61+
}*/
62+
]
63+
};

‎css-visual-test/component-library-generator.js

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
var componentLibraryGenerator = require( __dirname + '/lib/ComponentLibraryGenerator' ),
2+
manifestBuilder = require( __dirname + '/lib/ManifestBuilder' );
3+
4+
manifestBuilder.buildTestManifest( 'Calypso' )
5+
.then( componentLibraryGenerator.generateComponentLibrary )
6+
.then( function() {
7+
process.exit( 0 );
8+
} )
9+
.catch( function( error ) {
10+
console.log( error.stack );
11+
process.exit( 1 );
12+
} );

‎css-visual-test/component-library-server.js

+59
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
var path = require( 'path' ),
2+
connect = require( path.resolve( 'node_modules' ) + '/connect' ),
3+
winston = require( path.resolve( 'node_modules' ) + '/winston' ),
4+
portScanner = require( path.resolve( 'node_modules' ) + '/portscanner' ),
5+
psAux = require( path.resolve( 'node_modules' ) + '/ps-aux' ),
6+
config = require( __dirname + '/lib/Config' ),
7+
componentLibraryServerRunning = false,
8+
port9200Open = false,
9+
componentLibraryServerPort = config.componentLibraryPort,
10+
printToConsoleAndExit = function( message, code ) {
11+
winston.error( message );
12+
process.exit( code );
13+
};
14+
15+
psAux.singleton.parsed(function ( error, processList ) {
16+
var serverProcessesRunning = 0;
17+
18+
if( error ) {
19+
printToConsoleAndExit( 'Error calling ps aux: ' + error.message );
20+
}
21+
22+
processList.forEach( function( process ) {
23+
if( process.command.indexOf( 'css-visual-test/component-library-server.js' ) > 0 ) {
24+
serverProcessesRunning++;
25+
}
26+
} );
27+
28+
// exclude the process running the check
29+
if( serverProcessesRunning > 1 ) {
30+
componentLibraryServerRunning = true;
31+
}
32+
33+
portScanner.checkPortStatus( componentLibraryServerPort, 'localhost', function( error, status ) {
34+
if( status === 'open' ) {
35+
port9200Open = true;
36+
}
37+
38+
// I'm intentionally not including the case where the server is running on a port that is not 9200,
39+
// because the tests depend on that port.
40+
if( port9200Open &&
41+
! componentLibraryServerRunning ) {
42+
// the port being in use is a fatal error and should stop excecution of the rest of the steps in the test
43+
printToConsoleAndExit( 'Port ' + componentLibraryServerPort + ' is in use by something other than the Component Library ' +
44+
'Server. Please shut down the program using this port and try again.', 1 );
45+
} else if( ! port9200Open &&
46+
! componentLibraryServerRunning ) {
47+
connect(
48+
connect.static( path.join( __dirname, 'component-library' ) ) // Serve the component library
49+
).listen( componentLibraryServerPort );
50+
51+
winston.info( 'Component Library Server started on port ' + componentLibraryServerPort + '.' );
52+
} else if( port9200Open &&
53+
componentLibraryServerRunning ) {
54+
printToConsoleAndExit( 'Component Library Server is already running on port ' + componentLibraryServerPort +
55+
'. Server not started.', 0 );
56+
}
57+
});
58+
} );
59+

0 commit comments

Comments
 (0)