docs: all about tests (DataDog#6230)

This change puts all of the documentation about testing in its own file.
It also turns a bit of documentation into code via the `docker-compose`
file.

README.md

@@ -21,73 +21,4 @@ To get started as a contributor, see [the contributing docs](https://ddtrace.rea
 
 [setup docs]: https://docs.datadoghq.com/tracing/setup/python/
 [api docs]: https://ddtrace.readthedocs.io/
-[visualization docs]: https://docs.datadoghq.com/tracing/visualization/
-
-## Testing
-
-### Running Tests in docker
-
-The dd-trace-py testrunner docker image allows you to run tests in an environment that matches CI. This is especially useful
-if you are unable to install certain test dependencies on your dev machine's bare metal.
-
-Once your docker-compose environment is running, you can use the shell script to
-execute tests within a Docker image. You can start the container with a bash shell:
-
-    $ scripts/ddtest
-
-You can now run tests as you would do in your local environment. We use
-[riot][riot], a new tool that we developed for addressing
-our specific needs with an ever growing matrix of tests. You can list the tests
-managed by each:
-
-    $ riot list
-
-You can run multiple tests by using regular expressions:
-
-    $ riot run psycopg
-
-[riot]: https://github.com/DataDog/riot/
-
-### Running Tests locally
-
-1. Install riot: `pip install riot`.
-2. Create the base virtual environments: `riot -v generate`.
-3. You can list the available test suites with `riot list`.
-4. Certain tests might require running service containers in order to emulate
-   the necessary testing environment. You can spin up individual containers with
-   `docker-compose up -d <SERVICE_NAME>`, where `<SERVICE_NAME>` should match a
-   service specified in the `docker-compose.yml` file.
-5. Run a test suite: `riot -v run <RUN_FLAGS> <TEST_SUITE_NAME>`.
-
-You can use the `-s` and `-x` flags: `-s` prevents riot from reinstalling the dev package;
-`-x` forces an exit after the first failed test suite. To limit the tests to a particular
-version of Python, use the `-p` flag: `riot -v run -p <PYTHON_VERSION>`. You can also pass
-command line arguments to the underlying test runner (like pytest) with the `--` argument.
-For example, you can run a specific test under pytest with
-`riot -v run -s gunicorn -- -k test_no_known_errors_occur`
-
-The `run` command uses regex syntax, which in some cases will cause multiple
-test suites to run. Use the following syntax to ensure only an individual suite
-runs: `^<TEST_SUITE_NAME>$` where `^` signifies the start of a string and `$`
-signifies the end of a string. For example, use `riot -v run -s -x ^redis$` to
-run only the redis suite.
-
-### Use the APM Test Agent
-
-The APM test agent can emulate the APM endpoints of the Datadog agent. Spin up
-the `testagent` container along with any other service container:
-
-    $ docker-compose up -d testagent <SERVICE_CONTAINER>
-
-Run the test agent as a proxy in your tests:
-
-    $ DD_TRACE_AGENT_URL=http://localhost:9126/ riot -v run <RUN_FLAGS> --pass-env <TEST_SUITE_NAME>
-
-`--pass-env` injects the environment variables of the current shell session into
-the command. Here's an example command for running the redis test suite along
-with the test agent, limited to tests for Python 3.9:
-
-    $ DD_TRACE_AGENT_URL=http://localhost:9126/ riot -v run -p 3.9 -s -x --pass-env '^redis$'
-
-Read more about the APM test agent:
-https://github.com/datadog/dd-apm-test-agent#readme
+[visualization docs]: https://docs.datadoghq.com/tracing/visualization/

docker-compose.yml

@@ -161,6 +161,7 @@ services:
         environment:
             - TOX_SKIP_DIST=True
         network_mode: host
+        userns_mode: host
         working_dir: /root/project/
         volumes:
           - ddagent:/tmp/ddagent

docs/contributing-testing.rst

@@ -0,0 +1,95 @@
+Testing
+=======
+
+Imagine you're making a change to the library.
+
+If your change touches Python code, it should probably include at least one test.
+
+What kind of tests should I write?
+----------------------------------
+
+We use heuristics to decide when and what sort of tests to write. For example, a pull request implementing
+a new feature should include enough unit tests to cover the feature's "happy path" use cases in addition
+to any known likely edge cases. If the feature involves a new form of communication with another component
+(like the Datadog Agent or libddwaf), it should probably include at least one integration test exercising
+the end-to-end communication.
+
+If a pull request fixes a bug, it should include a test that, on the trunk branch, would replicate the bug.
+Seeing this test pass on the fix branch gives us confidence that the bug was actually fixed.
+
+Where do I put my tests?
+------------------------
+
+Put your code's tests in the appropriate subdirectory of the ``tests`` directory based on what they are testing.
+If your feature is substantially new, you may decide to create a new ``tests`` subdirectory in the interest
+of code organization.
+
+How do I run the test suite?
+----------------------------
+
+With `docker <https://www.docker.com/products/docker>`_ installed, run this.
+
+.. code-block:: bash
+
+    $ scripts/ddtest riot run -p 3.10
+
+This command runs the entire test suite, which is probably not what you want to do.
+
+How do I run only the tests I care about?
+-----------------------------------------
+
+1. Note the names of the tests you care about - these are the "test names".
+2. Find the ``Venv`` in the `riotfile <https://github.com/DataDog/dd-trace-py/blob/32b88eadc00e05cd0bc2aec587f565cc89f71229/riotfile.py#L426>`_
+   whose ``command`` contains the tests you're interested in. Note the ``Venv``'s ``name`` - this is the
+   "suite name".
+3. Find the directive in the `CI config <https://github.com/DataDog/dd-trace-py/blob/32b88eadc00e05cd0bc2aec587f565cc89f71229/.circleci/config.yml#L664>`_
+   whose ``pattern`` is equal to the suite name. Note the ``docker_services`` section of the directive, if present -
+   these are the "suite services".
+4. Start the suite services, if applicable, with ``$ docker-compose up -d service1 service2``.
+5. Start the test-runner Docker container with ``$ scripts/ddtest``.
+6. In the test-runner shell, run the tests with ``$ riot -v run --pass-env -s -p 3.10 <suite_name> -- -s -vv -k 'test_name1 or test_name2'``.
+
+Anatomy of a Riot Command
+-------------------------
+
+.. code-block:: bash
+
+    $ riot -v run -s -p 3.10 <suite_name> -- -s -vv -k 'test_name1 or test_name2'
+
+* ``-v``: Print verbose output
+* ``--pass-env``: Pass all environment variables in the current shell to the pytest invocation
+* ``-s``: Skip repetitive installation steps when possible
+* ``-p 3.10``: Run the tests using Python 3.10. You can change the version string if you want.
+* ``<suite_name>``: A regex matching the names of the Riot ``Venv`` instances to run
+* ``--``: Everything after this gets treated as a ``pytest`` argument
+* ``-s``: Make potential uses of ``pdb`` work properly
+* ``-vv``: Be loud about which tests are being run
+* ``-k 'test1 or test2'``: Test selection by `keyword expression <https://docs.pytest.org/en/7.1.x/how-to/usage.html#specifying-which-tests-to-run>`_
+
+Why are my tests failing with 404 errors?
+-----------------------------------------
+
+If your test relies on the ``testagent`` service, you might see it fail with a 404 error.
+To fix this:
+
+.. code-block:: bash
+
+    # outside of the testrunner shell
+    $ docker-compose up -d testagent
+
+    # inside the testrunner shell, started with scripts/ddtest
+    $ DD_AGENT_PORT=9126 riot -v run --pass-env ...
+
+Why is my CI run failing with a message about requirements files?
+-----------------------------------------------------------------
+
+``.riot/requirements`` contains requirements files generated with ``pip-compile`` for every environment specified
+by ``riotfile.py``. Riot uses these files to build its environments, and they do not get rebuilt automatically
+when the riotfile changes. Thus, if you make changes to the riotfile, you need to rebuild them.
+
+.. code-block:: bash
+
+  $ scripts/ddtest scripts/compile-and-prune-test-requirements
+
+You can commit and pull request the resulting changes to files in ``.riot/requirements`` alongside the
+changes you made to ``riotfile.py``.

docs/contributing.rst

@@ -98,37 +98,8 @@ opening an issue describing the limitations of the current design.
 Tests
 -----
 
-If your change touches Python code, it should probably include at least one test. We use heuristics to
-decide when and what sort of tests to write. For example, a pull request implementing a new feature
-should include enough unit tests to cover the feature's "happy path" use cases in addition to any known
-likely edge cases. If the feature involves a new form of communication with another component (like the
-Datadog Agent or libddwaf), it should probably include at least one integration test exercising the end-to-end
-communication.
-
-If a pull request fixes a bug, it should include a test that, on the trunk branch, would replicate the bug.
-Seeing this test pass on the fix branch gives us confidence that the bug was actually fixed.
-
-Put your code's tests in the appropriate subdirectory of the ``tests`` directory based on what they are testing.
-If your feature is substantially new, you may decide to create a new ``tests`` subdirectory in the interest
-of code organization.
-
-``.riot/requirements`` contains requirements files generated with ``pip-compile`` for every environment specified
-by ``riotfile.py``. Riot uses these files to build its environments, and they do not get rebuilt automatically
-when the riotfile changes. Thus, if you make changes to the riotfile, you need to run either
-``scripts/compile-and-prune-test-requirements`` or ``riot run -c <mytests>`` to regenerate the requirements
-files for the environments that changed. In order to run the script, you need to have all minor versions
-of Python that the tracer supports. The easiest way to accomplish this and generate the files is to simply
-spin up the testagent container, exec into it, and run the script from there.
-
-.. code-block:: bash
-
-  cd dd-trace-py && docker run --network host --userns=host --rm -w /root/project -v $PWD/:/root/project \
-    -it ghcr.io/datadog/dd-trace-py/testrunner \
-    bash -c "git config --global --add safe.directory /root/project && pip install riot && bash -i './scripts/compile-and-prune-test-requirements'"
-
-
-This can also be accomplished using pyenv and installing all of the Python versions before running the script.
-You can commit and pull request changes to files in ``.riot/requirements`` alongside the corresponding changes to ``riotfile.py``.
+If your change touches Python code, it should probably include at least one test. See the
+`testing guidelines <contributing-testing.rst>`_ for details.
 
 Documentation
 -------------
@@ -158,4 +129,5 @@ Keep the following in mind when writing logging code:
     :hidden:
 
     contributing-integrations
+    contributing-testing
     releasenotes

scripts/ddtest

@@ -33,6 +33,7 @@ else
     docker-compose run \
                    -e DD_TRACE_AGENT_URL \
                    --rm \
+                   -i \
                    testrunner \
-                   bash -c "$FULL_CMD"
+                   bash -c "git config --global --add safe.directory /root/project && $FULL_CMD"
 fi