[Docs] Added documentation on pytest target parametrization.

docs/dev/how_to.rst

@@ -29,3 +29,4 @@ various areas of the TVM stack.
    relay_add_pass
    relay_bring_your_own_codegen
    codebase_walkthrough
+   pytest_target_parametrization

docs/dev/pytest_target_parametrization.rst

@@ -0,0 +1,266 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+Python Target Parametrization
+=============================
+
+Summary
+-------
+
+A single python function in the test suite can expand to several
+parametrized unit tests, each of which tests a single target device.
+In order for a test to be run, all of the following must be true.
+
+- The test exists in a file or directory that has been passed to
+  `pytest`.
+
+- The pytest marks applied to the function, either explicitly or
+  through target parametrization, must be compatible with the
+  expression passed to pytest's `-m` argument.
+
+- For parametrized tests using the `target` fixture, the target must
+  appear in the environment variable `TVM_TEST_TARGETS`.
+
+- For parametrized tests using the `target` fixture, the build
+  configuration in `config.cmake` must enable the corresponding
+  runtime.
+
+Unit-Test File Contents
+-----------------------
+
+.. _pytest-marks: https://docs.pytest.org/en/6.2.x/mark.html
+
+Each test file has one or more test functions.  The test functions can
+be decorated with `pytest marks <pytest-marks>`_ to include metadata.
+The most frequently applied marks are as follows.
+
+- ``@pytest.mark.gpu`` - Tags a function as using GPU
+  capabilities. This has no effect on its own, but can be paired with
+  command-line arguments ``-m gpu`` or ``-m 'not gpu'`` to restrict
+  which tests pytest will executed.  This should be called on its own,
+  but is part of other marks used in unit-tests.
+
+- ``@tvm.testing.uses_gpu`` - Applies ``@pytest.mark.gpu``.  This
+  should be used to mark a unit tests that may use the GPU, if one is
+  present.  This decorator is only needed for tests that explicitly
+  loop over ``tvm.testing.enabled_targets()``, but that is no longer
+  the preferred style of writing unit tests (see below).  When using
+  ``tvm.testing.parametrize_targets()``, this decorator is implicit
+  for GPU targets, and does not need to be explicitly applied.
+
+- ``@tvm.testing.requires_gpu`` - Applies ``@tvm.testing.uses_gpu``,
+  and additionally marks that the test should be skipped
+  (``@pytest.mark.skipif``) entirely if no GPU is present.
+
+- ``@tvfm.testing.requires_RUNTIME`` - Several decorators
+  (e.g. ``@tvm.testing.requires_cuda``), each of which skips a test if
+  the specified runtime cannot be used. A runtime cannot be used if it
+  is disabled in the ``config.cmake``, or if a compatible device is
+  not present. For runtimes that use the GPU, this includes
+  ``@tvm.testing.requires_gpu``.
+
+A single test can loop over ``tvm.testing.enabled_targets()`` to
+validate all enabled targets, based on the environment variable
+``TVM_TEST_TARGETS``, the build configuration, and the physical
+hardware present.  Most current tests use this style, but it is not
+recommended for new tests.  The pytest output for this style silently
+skips runtimes that are disabled in ``config.cmake``, or do not have a
+device on which they can run.  In addition, the test halts on the
+first target to fail, which is ambiguous as to whether the error
+occurs on a particular target, or on every target.
+
+.. code-block:: python
+
+    # Old style, do not use.
+    def test_function():
+        for target,dev in tvm.testing.enabled_targets():
+            # Test code goes here
+
+The recommended method to run a test on multiple targets is by
+parametrizing the test.  This can be done explicitly for a fixed list
+of targets by decorating with
+``@tvm.testing.parametrize_targets('target_1', 'target_2', ...)``, and
+accepting ``target`` or ``dev`` as function arguments.  The function will
+be run once for each target listed, and the success/failure of each
+target is reported separately.  Each test run is decorated with the
+``@tvm.testing.requires_RUNTIME`` that corresponds to the target being
+used.  As a result, if a target is disabled in ``config.cmake`` or does
+not have appropriate hardware to run, it will be explicitly listed as
+skipped.
+
+.. code-block:: python
+
+    # Explicit listing of targets to use.
+    @tvm.testing.parametrize_target('llvm', 'cuda')
+    def test_function(target, dev):
+        # Test code goes here
+
+For tests that should run correctly on all targets, the decorator can
+be omitted.  Any test that accepts a ``target`` or ``dev`` argument
+will automatically be parametrized over all targets specified in
+``TVM_TEST_TARGETS``.  The parametrization provides the same
+pass/fail/skipped report for each target, while allowing the test
+suite to be easily extended to cover additional targets.
+
+.. code-block:: python
+
+    # Implicitly parametrized to run on all targets
+    # in environment variable TVM_TEST_TARGETS
+    def test_function(target, dev):
+        # Test code goes here
+
+The ``@tvm.testing.parametrize_targets`` can also be used as a bare
+decorator to explicitly draw attention to the parametrization, but has
+no additional effect.
+
+.. code-block:: python
+
+    # Explicitly parametrized to run on all targets
+    # in environment variable TVM_TEST_TARGETS
+    @tvm.testing.parametrize_targets
+    def test_function(target, dev):
+        # Test code goes here
+
+
+Specific targets can be excluded or marked as expected to fail using
+the ``@tvm.testing.exclude_targets`` or
+``@tvm.testing.known_failing_targets`` decorators.  For more
+information on their intended use cases, please see their docstrings.
+
+Lastly, in some cases it may be necessary to parametrize across
+multiple parameters.  For instance, there may be target-specific
+implementations that should be tested, where some targets have more
+than one implementation.  These can be done by explicitly
+parametrizing over tuples of arguments, such as shown below.  In these
+cases, only the explicitly listed targets will run, but they will
+still have the appropriate ``@tvm.testing.requires_RUNTIME`` mark
+applied to them.
+
+.. code-block:: python
+
+   @pytest.mark.parametrize('target,impl', [
+        ('llvm', cpu_implementation),
+        ('cuda', gpu_implementation_small_batch),
+        ('cuda', gpu_implementation_large_batch),
+    ])
+    def test_function(target, dev, impl):
+        # Test code goes here
+
+Running locally
+---------------
+
+To run the python unit-tests locally, use the command ``pytest`` in
+the ``${TVM_HOME}`` directory.
+
+- Environment variables
+    - ``TVM_TEST_TARGETS`` should be a semicolon-separated list of
+      targets to run. If unset, will default to the targets defined in
+      ``tvm.testing.DEFAULT_TEST_TARGETS``.
+
+      Note: If ``TVM_TEST_TARGETS`` does not contain any targets that
+      are both enabled, and have an accessible device of that type,
+      then the tests will fall back to running on the ``llvm`` target
+      only.
+
+    - ``TVM_LIBRARY_PATH`` should be a path to the ``libtvm.so``
+      library. This can be used, for example, to run tests using a
+      debug build. If unset, will search for ``libtvm.so`` relative to
+      the TVM source directory.
+
+- Command-line arguments
+
+    - Passing a path to a folder or file will run only the unit tests
+      in that folder or file. This can be useful, for example, to
+      avoid running tests located in ``tests/python/frontend`` on a
+      system without a specific frontend installed.
+
+    - The ``-m`` argument only runs unit tests that are tagged with a
+      specific pytest marker. The most frequent usage is to use ``m
+      gpu`` to run only tests that are marked with
+      ``@pytest.mark.gpu`` and use a GPU to run. It can also be used
+      to run only tests that do not use a GPU, by passing ``m 'not
+      gpu'``.
+
+      Note: This filtering takes place after the selection of targets
+      based on the ``TVM_TEST_TARGETS`` environment variable.  Even if
+      ``-m gpu`` is specified, if ``TVM_TEST_TARGETS`` does not
+      contain GPU targets, no GPU tests will be run.
+
+Running in local docker container
+---------------------------------
+
+.. _tlcpack: https://hub.docker.com/u/tlcpack
+
+The ``docker/bash.sh`` script can be used to run unit tests inside the
+same docker image as is used by the CI.  The first argument should
+specify which docker image to run (e.g. ``docker/bash.sh ci_gpu``).
+Allowed image names are defined at the top of the Jenkinsfile located
+in the TVM source directory, and map to images at `tlcpack`_.
+
+If no additional arguments are given, the docker image will be loaded
+with an interactive bash session.  If a script is passed as an
+optional argument (e.g. ``docker/bash.sh ci_gpu tests/scripts/task_python_unittest.sh``), then that script will be
+executed inside the docker image.
+
+Note: The docker images contain all system dependencies, but do not
+include the ``build/config.cmake`` configuration file for those
+systems.  The TVM source directory is used as the home directory of
+the docker image, and so this will default to using the same
+config/build directories as the local config.  One solution is to
+maintain separate ``build_local`` and ``build_docker`` directories,
+and make a symlink from ``build`` to the appropriate folder when
+entering/exiting docker.
+
+Running in CI
+-------------
+
+Everything in the CI starts from the task definitions present in the
+Jenkinsfile.  This includes defining which docker image gets used,
+what the compile-time configuration is, and which tests are included
+in which stages.
+
+- Docker images
+
+  Each task of the Jenkinsfile (e.g. 'BUILD: CPU') makes calls to
+  ``docker/bash.sh``.  The argument following the call to
+  docker/bash.sh defines the docker image in CI, just as it does
+  locally.
+
+- Compile-time configuration
+
+  The docker image does not have the ``config.cmake`` file built into
+  it, so this is the first step in each of the ``BUILD`` tasks.  This
+  is done using the ``tests/scripts/task_config_build_*.sh`` scripts.
+  Which script is used depends on the build being tested, and is
+  specified in the Jenkinsfile.
+
+  Each ``BUILD`` task concludes by packing a library for use in later
+  tests.
+
+- Which tests run
+
+  The ``Unit Test`` and ``Integration Test`` stages of the Jenkinsfile
+  determine how ``pytest`` is called.  Each task starts by unpacking a
+  compiled library that was previous compiled in the ``BUILD`` stage,
+  then runs a test script
+  (e.g. ``tests/script/task_python_unittest.sh``).  These scripts set
+  the files/folders and command-line options that are passed to
+  ``pytest``.
+
+  Several of these scripts include the ``-m gpu`` option, which
+  restricts the tests to only run tests that include the
+  ``@pytest.mark.gpu`` mark.