|
| 1 | +Tests |
| 2 | +===== |
| 3 | + |
| 4 | + |
| 5 | +Quick Start |
| 6 | +----------- |
| 7 | + |
| 8 | +To add a simple unit test for a new feature you developed, open or create a |
| 9 | +`test-data/unit/check-*.test` file with a name that roughly relates to the |
| 10 | +feature you added. |
| 11 | + |
| 12 | +Add the test in this format anywhere in the file: |
| 13 | + |
| 14 | + [case testNewSyntaxBasics] |
| 15 | + # flags: --python-version 3.6 |
| 16 | + x: int |
| 17 | + x = 5 |
| 18 | + y: int = 5 |
| 19 | + |
| 20 | + a: str |
| 21 | + a = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") |
| 22 | + b: str = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") |
| 23 | + |
| 24 | + zzz: int |
| 25 | + zzz: str # E: Name 'zzz' already defined |
| 26 | + |
| 27 | +- no code here is executed, just type checked |
| 28 | +- optional `# flags: ` indicates which flags to use for this unit test |
| 29 | +- `# E: abc...` indicates that this line should result in type check error |
| 30 | +with text "abc..." |
| 31 | +- note a space after `E:` and `flags:` |
| 32 | +- lines without `# E: ` should cause no type check errors |
| 33 | +- optional `[builtins fixtures/...]` tells the type checker to use |
| 34 | +stubs from the indicated file (see Fixtures section below) |
| 35 | +- optional `[out]` is an alternative to the "# E:" notation: it indicates that |
| 36 | +any text after it contains the expected type checking error messages. |
| 37 | +usually, "E: " is preferred because it makes it easier to associate the |
| 38 | +errors with the code generating them at a glance, and to change the code of |
| 39 | +the test without having to change line numbers in `[out]` |
| 40 | +- an empty `[out]` section has no effect |
| 41 | + |
| 42 | + |
| 43 | +Fixtures |
| 44 | +-------- |
| 45 | + |
| 46 | +The unit tests use minimal stubs for builtins, so a lot of operations are not |
| 47 | +possible. You should generally define any needed classes within the test case |
| 48 | +instead of relying on builtins, though clearly this is not always an option |
| 49 | +(see below for more about stubs in test cases). This way tests run much |
| 50 | +faster and don't break if the stubs change. If your test crashes mysteriously |
| 51 | +even though the code works when run manually, you should make sure you have |
| 52 | +all the stubs you need for your test case, including built-in classes such as |
| 53 | +`list` or `dict`, as these are not included by default. |
| 54 | + |
| 55 | + - The builtins used by default in unit tests live in |
| 56 | + `test-data/unit/lib-stub`. |
| 57 | + |
| 58 | + - Individual test cases can override the stubs by using `[builtins |
| 59 | + fixtures/foo.pyi]`; this targets files in `test-data/unit/fixtures`; feel |
| 60 | + free to modify existing files there or create new ones as you deem fit. |
| 61 | + |
| 62 | + - You are also free to add additional stubs to this directory, but |
| 63 | + generally don't update files in `lib-stub` without first discussing the |
| 64 | + addition with other mypy developers, as additions could slow down the test |
| 65 | + suite. |
| 66 | + |
| 67 | + |
| 68 | +Running tests and linting |
| 69 | +------------------------- |
| 70 | + |
| 71 | +First install any additional dependencies needed for testing: |
| 72 | + |
| 73 | + $ python3 -m pip install -U -r test-requirements.txt |
| 74 | + |
| 75 | +To run all tests, run the script `runtests.py` in the mypy repository: |
| 76 | + |
| 77 | + $ ./runtests.py |
| 78 | + |
| 79 | +Note that some tests will be disabled for older python versions. |
| 80 | + |
| 81 | +This will run all tests, including integration and regression tests, |
| 82 | +and will type check mypy and verify that all stubs are valid. |
| 83 | + |
| 84 | +You can run a subset of test suites by passing positive or negative |
| 85 | +filters: |
| 86 | + |
| 87 | + $ ./runtests.py lex parse -x lint -x stub |
| 88 | + |
| 89 | +For example, to run unit tests only, which run pretty quickly: |
| 90 | + |
| 91 | + $ ./runtests.py unit-test pytest |
| 92 | + |
| 93 | +The unit test suites are driven by a mixture of test frameworks: mypy's own |
| 94 | +`myunit` framework, and `pytest`, which we're in the process of migrating to. |
| 95 | +Test suites for individual components are in the files `mypy/test/test*.py`. |
| 96 | +You can run many of these individually by doing `runtests.py testfoobar`. For |
| 97 | +finer control over which unit tests are run and how, you can run `py.test` or |
| 98 | +`scripts/myunit` directly, or pass inferior arguments via `-a`: |
| 99 | + |
| 100 | + $ py.test mypy/test/testcheck.py -v -k MethodCall |
| 101 | + $ ./runtests.py -v 'pytest mypy/test/testcheck' -a -v -a -k -a MethodCall |
| 102 | + |
| 103 | + $ PYTHONPATH=$PWD scripts/myunit -m mypy.test.testlex -v '*backslash*' |
| 104 | + $ ./runtests.py mypy.test.testlex -a -v -a '*backslash*' |
| 105 | + |
| 106 | +You can also run the type checker for manual testing without |
| 107 | +installing anything by setting up the Python module search path |
| 108 | +suitably (the lib-typing/3.2 path entry is not needed for Python 3.5 |
| 109 | +or when you have manually installed the `typing` module): |
| 110 | + |
| 111 | + $ export PYTHONPATH=$PWD:$PWD/lib-typing/3.2 |
| 112 | + $ python<version> -m mypy PROGRAM.py |
| 113 | + |
| 114 | +You can add the entry scripts to PATH for a single python3 version: |
| 115 | + |
| 116 | + $ export PATH=$PWD/scripts |
| 117 | + $ mypy PROGRAM.py |
| 118 | + |
| 119 | +You can check a module or string instead of a file: |
| 120 | + |
| 121 | + $ mypy PROGRAM.py |
| 122 | + $ mypy -m MODULE |
| 123 | + $ mypy -c 'import MODULE' |
| 124 | + |
| 125 | +To run the linter: |
| 126 | + |
| 127 | + $ ./runtests.py lint |
| 128 | + |
| 129 | +Many test suites store test case descriptions in text files |
| 130 | +(`test-data/unit/*.test`). The module `mypy.test.data` parses these |
| 131 | +descriptions. The package `mypy.myunit` contains the test framework used for |
| 132 | +the non-checker test cases. |
| 133 | + |
| 134 | +Python evaluation test cases are a little different from unit tests |
| 135 | +(`mypy/test/testpythoneval.py`, `test-data/unit/pythoneval.test`). These |
| 136 | +type check programs and run them. Unlike the unit tests, these use the |
| 137 | +full builtins and library stubs instead of minimal ones. Run them using |
| 138 | +`runtests.py testpythoneval`. |
| 139 | + |
| 140 | +`runtests.py` by default runs tests in parallel using as many processes as |
| 141 | +there are logical cores the `runtests.py` process is allowed to use (on |
| 142 | +some platforms this information isn't available, so 2 processes are used by |
| 143 | +default). You can change the number of workers using `-j` option. |
| 144 | + |
| 145 | +All pytest tests run as a single test from the perspective of `runtests.py`, |
| 146 | +and so `-j` option has no effect on them. Instead, `pytest` itself determines |
| 147 | +the number of processes to use. The default (set in `./pytest.ini`) is the |
| 148 | +number of logical cores; this can be overridden using `-n` option. |
| 149 | + |
| 150 | +Note that running more processes than logical cores is likely to |
| 151 | +significantly decrease performance; the relevant count is the number of |
| 152 | +processes used by `runtests.py` plus those used by `pytest`. |
| 153 | + |
| 154 | + |
| 155 | +Coverage reports |
| 156 | +---------------- |
| 157 | + |
| 158 | +There is an experimental feature to generate coverage reports. To use |
| 159 | +this feature, you need to `pip install -U lxml`. This is an extension |
| 160 | +module and requires various library headers to install; on a |
| 161 | +Debian-derived system the command |
| 162 | + `apt-get install python3-dev libxml2-dev libxslt1-dev` |
| 163 | +may provide the necessary dependencies. |
| 164 | + |
| 165 | +To use the feature, pass e.g. `--txt-report "$(mktemp -d)"`. |
0 commit comments