Skip to content

Commit 247337f

Browse files
borsbors
committed
Auto merge of #82350 - ehuss:test-chapter, r=jyn514
Add a chapter on the test harness. There isn't really any online documentation on the test harness, so this adds a chapter to the rustc book which provides information on how the harness works and details on the command-line options.
2 parents 7c3a914 + 7d99d6d commit 247337f

File tree

4 files changed

+289
-9
lines changed

4 files changed

+289
-9
lines changed

‎library/test/src/cli.rs

+2-8
Original file line numberDiff line numberDiff line change
@@ -51,14 +51,8 @@ fn optgroups() -> getopts::Options {
5151
.optflag("", "test", "Run tests and not benchmarks")
5252
.optflag("", "bench", "Run benchmarks instead of tests")
5353
.optflag("", "list", "List all tests and benchmarks")
54-
.optflag("h", "help", "Display this message (longer with --help)")
55-
.optopt(
56-
"",
57-
"logfile",
58-
"Write logs to the specified file instead \
59-
of stdout",
60-
"PATH",
61-
)
54+
.optflag("h", "help", "Display this message")
55+
.optopt("", "logfile", "Write logs to the specified file", "PATH")
6256
.optflag(
6357
"",
6458
"nocapture",

‎src/doc/rustc/src/SUMMARY.md

+1
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
- [Deny-by-default lints](lints/listing/deny-by-default.md)
1212
- [Codegen options](codegen-options/index.md)
1313
- [JSON Output](json.md)
14+
- [Tests](tests/index.md)
1415
- [Platform Support](platform-support.md)
1516
- [Targets](targets/index.md)
1617
- [Built-in Targets](targets/built-in.md)

‎src/doc/rustc/src/command-line-arguments.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,8 @@ out a longer explanation of a given error.
204204
## `--test`: build a test harness
205205

206206
When compiling this crate, `rustc` will ignore your `main` function
207-
and instead produce a test harness.
207+
and instead produce a test harness. See the [Tests chapter](tests/index.md)
208+
for more information about tests.
208209

209210
<a id="option-target"></a>
210211
## `--target`: select a target triple to build

‎src/doc/rustc/src/tests/index.md

+284
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,284 @@
1+
# Tests
2+
3+
`rustc` has a built-in facility for building and running tests for a crate.
4+
More information about writing and running tests may be found in the [Testing
5+
Chapter] of the Rust Programming Language book.
6+
7+
Tests are written as free functions with the [`#[test]`
8+
attribute][attribute-test]. For example:
9+
10+
```rust
11+
#[test]
12+
fn it_works() {
13+
assert_eq!(2 + 2, 4);
14+
}
15+
```
16+
17+
Tests "pass" if they return without an error. They "fail" if they [panic], or
18+
return a [`Result`] with an error.
19+
20+
By passing the [`--test` option] to `rustc`, the compiler will build the crate
21+
in a special mode to construct an executable that will run the tests in the
22+
crate. The `--test` flag will make the following changes:
23+
24+
* The crate will be built as a `bin` [crate type], forcing it to be an
25+
executable.
26+
* Links the executable with [`libtest`], the test harness that is part of the
27+
standard library, which handles running the tests.
28+
* Synthesizes a [`main` function] which will process command-line arguments
29+
and run the tests. This new `main` function will replace any existing `main`
30+
function as the entry point of the executable, though the existing `main`
31+
will still be compiled.
32+
* Enables the [`test` cfg option], which allows your code to use conditional
33+
compilation to detect if it is being built as a test.
34+
* Enables building of functions annotated with the [`test`][attribute-test]
35+
and [`bench`](#benchmarks) attributes, which will be run by the test
36+
harness.
37+
38+
After the executable is created, you can run it to execute the tests and
39+
receive a report on what passes and fails. If you are using [Cargo] to manage
40+
your project, it has a built-in [`cargo test`] command which handles all of
41+
this automatically. An example of the output looks like this:
42+
43+
```text
44+
running 4 tests
45+
test it_works ... ok
46+
test check_valid_args ... ok
47+
test invalid_characters ... ok
48+
test walks_the_dog ... ok
49+
50+
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
51+
```
52+
53+
> **Note**: Tests must be built with the [`unwind` panic
54+
> strategy][panic-strategy]. This is because all tests run in the same
55+
> process, and they are intended to catch panics, which is not possible with
56+
> the `abort` strategy. See the unstable [`-Z panic-abort-tests`] option for
57+
> experimental support of the `abort` strategy by spawning tests in separate
58+
> processes.
59+
60+
## Test attributes
61+
62+
Tests are indicated using attributes on free functions. The following
63+
attributes are used for testing, see the linked documentation for more
64+
details:
65+
66+
* [`#[test]`][attribute-test] — Indicates a function is a test to be run.
67+
* `#[bench]` — Indicates a function is a benchmark to be
68+
run. Benchmarks are currently unstable and only available in the nightly
69+
channel, see the [unstable docs][bench-docs] for more details.
70+
* [`#[should_panic]`][attribute-should_panic] — Indicates that the test
71+
function will only pass if the function [panics][panic].
72+
* [`#[ignore]`][attribute-ignore] — Indicates that the test function will be
73+
compiled, but not run by default. See the [`--ignored`](#--ignored) and
74+
[`--include-ignored`](#--include-ignored) options to run these tests.
75+
76+
## CLI arguments
77+
78+
The libtest harness has several command-line arguments to control its
79+
behavior.
80+
81+
> Note: When running with [`cargo test`], the libtest CLI arguments must be
82+
> passed after the `--` argument to differentiate between flags for Cargo and
83+
> those for the harness. For example: `cargo test -- --nocapture`
84+
85+
### Filters
86+
87+
Positional arguments (those without a `-` prefix) are treated as filters which
88+
will only run tests whose name matches one of those strings. The filter will
89+
match any substring found in the full path of the test function. For example,
90+
if the test function `it_works` is located in the module
91+
`utils::paths::tests`, then any of the filters `works`, `path`, `utils::`, or
92+
`utils::paths::tests::it_works` will match that test.
93+
94+
See [Selection options](#selection-options) for more options to control which
95+
tests are run.
96+
97+
### Action options
98+
99+
The following options perform different actions other than running tests.
100+
101+
#### `--list`
102+
103+
Prints a list of all tests and benchmarks. Does not run any of the tests.
104+
[Filters](#filters) can be used to list only matching tests.
105+
106+
#### `-h`, `--help`
107+
108+
Displays usage information and command-line options.
109+
110+
### Selection options
111+
112+
The following options change how tests are selected.
113+
114+
#### `--test`
115+
116+
This is the default mode where all tests will be run as well as running all
117+
benchmarks with only a single iteration (to ensure the benchmark works,
118+
without taking the time to actually perform benchmarking). This can be
119+
combined with the `--bench` flag to run both tests and perform full
120+
benchmarking.
121+
122+
#### `--bench`
123+
124+
This runs in a mode where tests are ignored, and only runs benchmarks. This
125+
can be combined with `--test` to run both benchmarks and tests.
126+
127+
#### `--exact`
128+
129+
This forces [filters](#filters) to match the full path of the test exactly.
130+
For example, if the test `it_works` is in the module `utils::paths::tests`,
131+
then only the string `utils::paths::tests::it_works` will match that test.
132+
133+
#### `--skip` _FILTER_
134+
135+
Skips any tests whose name contains the given _FILTER_ string. This flag may
136+
be passed multiple times.
137+
138+
#### `--ignored`
139+
140+
Runs only tests that are marked with the [`ignore`
141+
attribute][attribute-ignore].
142+
143+
#### `--include-ignored`
144+
145+
Runs both [ignored](#--ignored) and non-ignored tests.
146+
147+
#### `--exclude-should-panic`
148+
149+
Excludes tests marked with the [`should_panic`
150+
attribute][attribute-should_panic].
151+
152+
⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z
153+
unstable-options` flag. See [tracking issue
154+
#82348](https://github.com/rust-lang/rust/issues/82348) for more information.
155+
156+
### Execution options
157+
158+
The following options affect how tests are executed.
159+
160+
#### `--test-threads` _NUM_THREADS_
161+
162+
Sets the number of threads to use for running tests in parallel. By default,
163+
uses the amount of concurrency available on the hardware as indicated by
164+
[`available_concurrency`].
165+
166+
This can also be specified with the `RUST_TEST_THREADS` environment variable.
167+
168+
#### `--force-run-in-process`
169+
170+
Forces the tests to run in a single process when using the [`abort` panic
171+
strategy][panic-strategy].
172+
173+
⚠️ 🚧 This only works with the unstable [`-Z panic-abort-tests`] option, and
174+
requires the `-Z unstable-options` flag. See [tracking issue
175+
#67650](https://github.com/rust-lang/rust/issues/67650) for more information.
176+
177+
#### `--ensure-time`
178+
179+
⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z
180+
unstable-options` flag. See [tracking issue
181+
#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable
182+
docs](../../unstable-book/compiler-flags/report-time.html) for more information.
183+
184+
### Output options
185+
186+
The following options affect the output behavior.
187+
188+
#### `-q`, `--quiet`
189+
190+
Displays one character per test instead of one line per test. This is an alias
191+
for [`--format=terse`](#--format-format).
192+
193+
#### `--nocapture`
194+
195+
Does not capture the stdout and stderr of the test, and allows tests to print
196+
to the console. Usually the output is captured, and only displayed if the test
197+
fails.
198+
199+
This may also be specified by setting the `RUST_TEST_NOCAPTURE` environment
200+
variable set to anything but `0`.
201+
202+
#### `--show-output`
203+
204+
Displays the stdout and stderr of successful tests after all tests have run.
205+
206+
Contrast this with [`--nocapture`](#--nocapture) which allows tests to print
207+
*while they are running*, which can cause interleaved output if there are
208+
multiple tests running in parallel, `--show-output` ensures the output is
209+
contiguous, but requires waiting for all tests to finish.
210+
211+
#### `--color` _COLOR_
212+
213+
Control when colored terminal output is used. Valid options:
214+
215+
* `auto`: Colorize if stdout is a tty and [`--nocapture`](#--nocapture) is not
216+
used. This is the default.
217+
* `always`: Always colorize the output.
218+
* `never`: Never colorize the output.
219+
220+
#### `--format` _FORMAT_
221+
222+
Controls the format of the output. Valid options:
223+
224+
* `pretty`: This is the default format, with one line per test.
225+
* `terse`: Displays only a single character per test. [`--quiet`](#-q---quiet)
226+
is an alias for this option.
227+
* `json`: Emits JSON objects, one per line. ⚠️ 🚧 This option is
228+
[unstable](#unstable-options), and requires the `-Z unstable-options` flag.
229+
See [tracking issue #49359](https://github.com/rust-lang/rust/issues/49359)
230+
for more information.
231+
232+
#### `--logfile` _PATH_
233+
234+
Writes the results of the tests to the given file.
235+
236+
#### `--report-time` _FORMAT_
237+
238+
⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z
239+
unstable-options` flag. See [tracking issue
240+
#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable
241+
docs](../../unstable-book/compiler-flags/report-time.html) for more information.
242+
243+
### Unstable options
244+
245+
Some CLI options are added in an "unstable" state, where they are intended for
246+
experimentation and testing to determine if the option works correctly, has
247+
the right design, and is useful. The option may not work correctly, break, or
248+
change at any time. To signal that you acknowledge that you are using an
249+
unstable option, they require passing the `-Z unstable-options` command-line
250+
flag.
251+
252+
## Benchmarks
253+
254+
The libtest harness supports running benchmarks for functions annotated with
255+
the `#[bench]` attribute. Benchmarks are currently unstable, and only
256+
available on the [nightly channel]. More information may be found in the
257+
[unstable book][bench-docs].
258+
259+
## Custom test frameworks
260+
261+
Experimental support for using custom test harnesses is available on the
262+
[nightly channel]. See [tracking issue
263+
#50297](https://github.com/rust-lang/rust/issues/50297) and the
264+
[custom_test_frameworks documentation] for more information.
265+
266+
[`--test` option]: ../command-line-arguments.md#option-test
267+
[`-Z panic-abort-tests`]: https://github.com/rust-lang/rust/issues/67650
268+
[`available_concurrency`]: ../../std/thread/fn.available_concurrency.html
269+
[`cargo test`]: ../../cargo/commands/cargo-test.html
270+
[`libtest`]: ../../test/index.html
271+
[`main` function]: ../../reference/crates-and-source-files.html#main-functions
272+
[`Result`]: ../../std/result/index.html
273+
[`test` cfg option]: ../../reference/conditional-compilation.html#test
274+
[attribute-ignore]: ../../reference/attributes/testing.html#the-ignore-attribute
275+
[attribute-should_panic]: ../../reference/attributes/testing.html#the-should_panic-attribute
276+
[attribute-test]: ../../reference/attributes/testing.html#the-test-attribute
277+
[bench-docs]: ../../unstable-book/library-features/test.html
278+
[Cargo]: ../../cargo/index.html
279+
[crate type]: ../../reference/linkage.html
280+
[custom_test_frameworks documentation]: ../../unstable-book/language-features/custom-test-frameworks.html
281+
[nightly channel]: ../../book/appendix-07-nightly-rust.html
282+
[panic-strategy]: ../../book/ch09-01-unrecoverable-errors-with-panic.html
283+
[panic]: ../../book/ch09-01-unrecoverable-errors-with-panic.html
284+
[Testing Chapter]: ../../book/ch11-00-testing.html

0 commit comments

Comments
 (0)