-
Notifications
You must be signed in to change notification settings - Fork 8
/
Copy pathguidelines.html
205 lines (154 loc) · 10.9 KB
/
guidelines.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html;charset=utf-8" />
<title>Guidelines for writing device independent tests</title>
<link href="http://www.w3.org/StyleSheets/TR/base.css" rel="stylesheet" type="text/css" />
<style type='text/css'>
.figure { float: left; padding-right: 1em; }
h3 { clear: left; }
.guidelines li {
padding:1em;
background-color:#7fc380;
margin-bottom:0.5em;
}
</style>
</head>
<body>
<p><img alt="W3C" src="http://www.w3.org/Icons/w3c_home" width="72" height="48" /></p>
<div class="head">
<h1>Guidelines for writing device independent tests</h1>
<dl>
<dt>Editors:</dt>
<dd>Dominique Hazaël-Massieux, W3C</dd>
<dd>Carmelo Montanez, National Institute of Standards and Technology</dd>
</dl><!--begin-include "../copyright.inc"-->
<p class="copyright"><a href="http://www.w3.org/Consortium/Legal/ipr-notice-20000612#Copyright">Copyright</a>
© 2008-2009 <a href="http://www.w3.org/"><abbr title="World Wide Web Consortium">W3C</abbr></a><sup>®</sup>
(<a href="http://www.csail.mit.edu/"><abbr title="Massachusetts Institute of Technology">MIT</abbr></a>,
<abbr title="European Research Consortium for Informatics and Mathematics"><a href=
"http://www.ercim.org/">ERCIM</a></abbr>, <a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C
<a href="http://www.w3.org/Consortium/Legal/ipr-notice-20000612#Legal_Disclaimer">liability</a>, <a href=
"http://www.w3.org/Consortium/Legal/ipr-notice-20000612#W3C_Trademarks">trademark</a>, <a href=
"http://www.w3.org/Consortium/Legal/copyright-documents-19990405">document use</a> rules apply.</p>
<!--end-include-->
<hr title="Separator for header" />
</div>
<h2 class="no-num no-toc" id="abstract">Abstract</h2>
<p>As support for Web technologies grows, it is important that tests writers
develop test suites that will work as well as possible across devices. This
document offers guidance in the form of simple guidelines to follow to create
device-independent tests.</p>
<h2 class="no-num no-toc" id="status">Status of this document</h2>
<p>These guidelines were produced by members of the <a href="http://www.w3.org/2005/MWI/Tests/">Mobile Web Test Suite Working Group</a> which is part of the <a href=
"http://www.w3.org/Mobile/">Mobile Web Initiative</a> (see <a href="http://www.w3.org/Mobile/About">About</a>).</p>
<p>Comments on, and discussions of this document can be sent on the (<a href=
"http://lists.w3.org/Archives/Public/public-mwts/">archived</a>) public mailing list <a href=
"mailto:public-mwts@w3.org">public-mwts@w3.org</a> (see <a href="http://www.w3.org/Mail/Request">instructions</a>).
W3C Members can also send comments directly to the Mobile Web Test Suites working group.</p>
<p>These guidelines represent the current thinking of the working group and as such may be updated, replaced or
rendered obsolete by other W3C documents at any time. Its publication does not imply endorsement by the W3C
membership or the <a href="http://www.w3.org/2005/MWI/Tests/">Mobile Web Test Working Group.</a></p>
<p>Patent disclosures relevant to the Mobile Web Test Suites Working Group may be found on the Working Group's public
<a href="http://www.w3.org/2006/02/mwi-test-charter.html#Patent">patent disclosure page.</a></p>
<h2 id="overview">Introduction</h2>
<p>This document provide a set of guidelines for writing test cases that can be run effectively across devices, in particular on mobile devices.</p>
<h3>Existing work</h3>
<p>The <a href="http://www.w3.org/TR/2003/NOTE-acdi-20030901/">Authoring Challenges for Device Independence</a> [<a href="#ACDI">ACDI</a>] explore these different limitations in the general context of writing device independent content, and the <a href="http://www.w3.org/TR/mobile-bp/">Mobile Web Best Practices 1.0</a> [<a href="#MWBP">MWBP</a>] give specific guidance on writing content targeted at mobile devices.</p>
<p>The <a href="http://www.w3.org/Style/CSS/Test/guidelines.html">CSS2.1 Test Case Authoring Guidelines</a> [<a href="#CSSTCAG">CSSTCAG</a>] provides guidance on how to write test cases, and sets device-independence as a goal. The <a href="http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm#TestReviewGuidelines">SVG Test Suite Manual</a> [<a href="#SVGTS">SVGTS</a>] also offers advices on writing test cases.</p>
<p>Inspired by this existing work, and based on the experience gathered by the Mobile Web Test Suites Working Group while reviewing test cases and their fitness to mobile devices [<a href="#MWTSSURVEY">MWTSSURVEY</a>], this document explores the specific aspects to take into account when writing test cases to ensure greater device independence.</p>
<h3>Lowest common denominator</h3>
<p>Consider recording the browser identifier with the widely implemented
<code>window.navigator.userAgent</code>, as there might be several browsers
on one device.</p>
<p>When designing device-independent test cases, it is important to
acknowledge the limitations of most devices:</p>
<ol>
<li>Screen</li>
<li>Memory available</li>
<li>Network bandwidth, latency and cost</li>
<li>CPU power</li>
</ol>
<p>For tests that require interaction (either for running the test or for submitting results), consider:</p>
<ol>
<li>Keyboard or pointing device access and ease of use</li>
<li>Human cost of correctly submitting the results</li>
<li>Start your test automatically with <code><body onload="beginTest();"></code></li>
</ol>
<h2 id="guidelines">Guidelines</h2>
<h3>Screen limitations</h3>
<p>Screen size matters when designing visual test cases - e.g. where the tester needs to assess whether the rendering of a test cases matches a reference rendering.</p>
<p>Across devices, the following screen parameters vary widely:</p>
<ol>
<li>screen resolution</li>
<li>page zooming capability, scrolling</li>
<li>physical dimensions</li>
<li>number of colors</li>
<li>contrast</li>
</ol>
<p>In general, to avoid problems when running test across devices:</p>
<ol class='guidelines'>
<li>Keep your tests as simple as possible. Do not include any visible information that might obfuscate or get in the way of the result. </li>
<li>Avoid tests based on absolute dimensions.</li>
<li>Be as concise as possible to avoid scrolling.</li>
<li>When using colors to express the result of the test, consider conveying the result also with text.</li>
</ol>
<div class='figure'>
<p>Bad test:</p>
<p><img src='bad-test.png' alt='Image depcting how a test from the CSS1 test suite looks on a mobile browser. Due to the amount of visible meta data included, it is impossible to see the actual test result.' /></p>
</div>
<div class='figure'>
<p>Good test:</p>
<p><img src='good-test.png' alt='Image depicting a simple CSS test case consisting of a pass condition (“There should be a green block under this paragraph”) and a green block indicating that the test has passed.' /></p>
</div>
<h3>Memory limitations</h3>
<p>To avoid hitting memory limitations of the devices on which the technology under test run:</p>
<ol class='guidelines'>
<li>Keep the number of style sheets, images, objects or scripts to a minimum.</li>
<li>On markup based documents, keep the document object model (DOM) tree as small as possible.</li>
<li>Keep the number of data structure created dynamically (e.g. by scripts) to a minimum.</li>
</ol>
<p>Less is more.</p>
<h3>Network bandwidth, latency and cost</h3>
<p>The characteristics of network access across devices vary greatly, in particular in terms of bandwidth available, the latency induced by network requests, and the cost of transferring content over the network.</p>
<p>To cater for test environments where network is slow or costly:</p>
<ol class='guidelines'>
<li>Keep the number of external resources loaded along with the test case to a minimum; prefer including the content (e.g. using internal style sheets instead of external ones) rather than loading it, except if it can be cached and re-used across test cases.</li>
<li>Keep the number of network requests originated from scripts to a minimum to speed up data transfer.</li>
<li>Take care when triggering DOM operations that they will not require downloading DTDs.</li>
<li>Be aware that mobile networks are can be aggressively cached or transformed, therefore you might need to adjust originator HTTP headers depending on the nature of your test.</li>
<li>Eliminate any white space in your tests.</li>
</ol>
<h3>CPU power</h3>
<p>CPU-intensive operations on mobile devices can drain the battery, and are likely to be much slower than on larger hardware. As a result:</p>
<ol class='guidelines'>
<li>Avoid unneeded images processing operations, such as scaling up or down raster images.</li>
</ol>
<h3>Extension capabilities</h3>
<p>Most mobile devices have limited capabilities when it comes to plugins,
additional fonts, or software extensions in general. Therefore do not rely on
non-standard features such as specific fonts to be installed.</p>
<h3>Keyboard or pointing device access</h3>
<p>Many mobile devices don't provide a full keyboard, and thus require several key presses to enter a given character.</p>
<ol class="guidelines">
<li>Minimise interactive tests</li>
<li>Provide helpers to reach the test suites using short URLs, 2D-barcodes, etc.</li>
<li>Enable httpd directory listings to allow simple navigation between tests</li>
</ol>
<h2>References</h2>
<dl>
<dt id="acdi">ACDI</dt>
<dd>Authoring Challenges for Device Independence, Rhys Lewis, Editor, W3C Working Group Note, 1 September 2003 (See <a href="http://www.w3.org/TR/2003/NOTE-acdi-20030901/">http://www.w3.org/TR/2003/NOTE-acdi-20030901/</a>)</dd>
<dt id="CSSTCAG">CSSTCAG</dt>
<dd>CSS2.1 Test Case Authoring Guidelines, Tantek Çelik, Ian Hickson, Elika J. Etemad, Editors (See <a href="http://www.w3.org/Style/CSS/Test/guidelines.html">http://www.w3.org/Style/CSS/Test/guidelines.html</a>)</dd>
<dt id="MWTSSURVEY">MWTSSURVEY</dt>
<dd>Conformance Test Suites for mobile web technologies (See <a href="http://www.w3.org/2005/MWI/Tests/matrix">http://www.w3.org/2005/MWI/Tests/matrix</a>)</dd>
<dt id="MWBP">MWBP</dt>
<dd>Mobile Web Best Practices 1.0, Basic Guidelines, Jo Rabin, Charles McCathieNeville, Editors, W3C Recommendation, 29 July 2008 (See <a href="http://www.w3.org/TR/mobile-bp/">http://www.w3.org/TR/mobile-bp/</a>)</dd>
<dt id="svgts">SVGTS</dt>
<dd>SVG Test Suite Manual, Lofton Henderson, Editor, 1 September 2001 (See <a href="http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm#TestReviewGuidelines">http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm#TestReviewGuidelines</a>)</dd>
<dt id="testfaq">TESTFAQ</dt>
<dd>Test Development FAQ, Patrick Curran, Editor (See <a href="http://www.w3.org/QA/WG/2005/01/test-faq">http://www.w3.org/QA/WG/2005/01/test-faq</a>)</dd>
</dl>
</body>
</html>