Upload coverage to codecov (#1190)

* Upload coverage to codecov
* Update tests documentation
* Fix pytest markers

.circleci/config.yml

@@ -3,6 +3,7 @@ version: 2.1
 
 orbs:
   coverage-reporter: codacy/coverage-reporter@11.4.1
+  codecov: codecov/codecov@1.2.3
 
 commands:
   check_changes:
@@ -34,7 +35,7 @@ jobs:
             conda activate esmvaltool
             pip install .[test]
             pytest -n 2 -m "not installation and not sequential"
-            pytest -n 0 -m "sequential"
+            pytest -n 0 -m "sequential" --cov-append
       - save_cache:
           key: test-{{ .Branch }}
           paths:
@@ -44,6 +45,11 @@ jobs:
           path: test-reports/
       - store_artifacts:
           path: test-reports/
+      - run:
+          # Install curl for codecov upload
+          command: apt install -y curl
+      - codecov/upload:
+          file: 'test-reports/coverage.xml'
       - coverage-reporter/send_report:
           coverage-reports: 'test-reports/coverage.xml'
           project-token: $CODACY_PROJECT_TOKEN

README.md

@@ -4,7 +4,7 @@
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3387139.svg)](https://doi.org/10.5281/zenodo.3387139)
 [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/ESMValGroup?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 [![CircleCI](https://circleci.com/gh/ESMValGroup/ESMValCore/tree/main.svg?style=svg)](https://circleci.com/gh/ESMValGroup/ESMValCore/tree/main)
-[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/5d496dea9ef64ec68e448a6df5a65783)](https://www.codacy.com/gh/ESMValGroup/ESMValCore?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValCore&utm_campaign=Badge_Coverage)
+[![codecov](https://codecov.io/gh/ESMValGroup/ESMValCore/branch/main/graph/badge.svg?token=wQnDzguwq6)](https://codecov.io/gh/ESMValGroup/ESMValCore)
 [![Codacy Badge](https://app.codacy.com/project/badge/Grade/5d496dea9ef64ec68e448a6df5a65783)](https://www.codacy.com/gh/ESMValGroup/ESMValCore?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValCore&utm_campaign=Badge_Grade)
 [![Docker Build Status](https://img.shields.io/docker/cloud/build/esmvalgroup/esmvalcore)](https://hub.docker.com/r/esmvalgroup/esmvalcore/)
 [![Anaconda-Server Badge](https://anaconda.org/esmvalgroup/esmvalcore/badges/installer/conda.svg)](https://conda.anaconda.org/esmvalgroup)

doc/contributing.rst

@@ -395,27 +395,21 @@ Tests
 -----
 
 To check that the code works correctly, there tests available in the
-`tests directory <https://github.com/ESMValGroup/ESMValCore/tree/main/tests>`_.
+`tests <https://github.com/ESMValGroup/ESMValCore/tree/main/tests>`__ directory.
 We use `pytest <https://docs.pytest.org>`_ to write and run our tests.
 
-Contributions to ESMValCore should be covered by unit tests.
+Contributions to ESMValCore should be
+`covered by unit tests <https://the-turing-way.netlify.app/reproducible-research/testing/testing-guidance.html#aim-to-have-a-good-code-coverage>`_.
 Have a look at the existing tests in the ``tests`` directory for inspiration on
 how to write your own tests.
 If you do not know how to start with writing unit tests, ask the
 `@ESMValGroup/tech-reviewers`_ for help by commenting on the pull request and
 they will try to help you.
-To check which parts of your code are covered by tests, open the file
-``test-reports/coverage_html/index.html`` and browse to the relevant file.
-It is also possible to view code coverage on Codacy_ (click the Files tab)
-and CircleCI_ (open the ``tests`` job and click the ARTIFACTS tab).
+It is also recommended that you have a look at the pytest_ documentation at some
+point when you start writing your own tests.
 
-Whenever you make a pull request or push new commits to an existing pull
-request, the tests in the `tests directory`_ of the branch associated with the
-pull request will be run automatically on CircleCI_.
-The results appear at the bottom of the pull request.
-Click on 'Details' for more information on a specific test job.
-To see some of the results on CircleCI, you may need to log in.
-You can do so using your GitHub account.
+Running tests
+~~~~~~~~~~~~~
 
 To run the tests on your own computer, go to the directory where the repository
 is cloned and run the command
@@ -426,11 +420,58 @@ is cloned and run the command
 
 Optionally you can skip tests which require additional dependencies for
 supported diagnostic script languages by adding ``-m 'not installation'`` to the
-previous command.
+previous command. To only run tests from a single file, run the command
+
+.. code-block:: bash
+
+   pytest tests/unit/test_some_file.py
+
+If you would like to avoid loading the default pytest configuration from
+`setup.cfg <https://github.com/ESMValGroup/ESMValCore/blob/main/setup.cfg>`_
+because this can be a bit slow for running just a few tests, use
+
+.. code-block:: bash
+
+   pytest -c /dev/null tests/unit/test_some_file.py
+
+Use
+
+.. code-block:: bash
+
+    pytest --help
+
+for more information on the available commands.
+
+Whenever you make a pull request or push new commits to an existing pull
+request, the tests in the ``tests`` directory of the branch associated with the
+pull request will be run automatically on CircleCI_.
+The results appear at the bottom of the pull request.
+Click on 'Details' for more information on a specific test job.
 
 When reviewing a pull request, always check that all test jobs on CircleCI_ were
 successful.
 
+Test coverage
+~~~~~~~~~~~~~
+
+To check which parts of your code are `covered by unit tests`_, open the file
+``test-reports/coverage_html/index.html`` (available after running a ``pytest``
+command) and browse to the relevant file.
+
+CircleCI will upload the coverage results from running the tests to codecov and
+Codacy.
+`codecov <https://app.codecov.io/gh/ESMValGroup/ESMValCore/pulls>`_ is a service
+that will comment on pull requests with a summary of the test coverage.
+If codecov_ reports that the coverage has decreased, check the report and add
+additional tests.
+Alternatively, it is also possible to view code coverage on Codacy_ (click the
+Files tab) and CircleCI_ (open the ``tests`` job and click the ARTIFACTS tab).
+To see some of the results on CircleCI, Codacy, or codecov, you may need to log
+in; you can do so using your GitHub account.
+
+When reviewing a pull request, always check that new code is covered by unit
+tests and codecov_ reports an increased coverage.
+
 .. _sample_data_tests:
 
 Sample data
@@ -471,7 +512,7 @@ Automated testing
 ~~~~~~~~~~~~~~~~~
 
 Whenever you make a pull request or push new commits to an existing pull
-request, the tests in the `tests directory`_ of the branch associated with the
+request, the tests in the ``tests`` of the branch associated with the
 pull request will be run automatically on CircleCI_.
 
 Every night, more extensive tests are run to make sure that problems with the

setup.cfg

@@ -21,8 +21,9 @@ flake8-ignore =
     doc/conf.py ALL
 log_level = WARNING
 markers =
-    installation: test requires installation of dependencies
+    installation: Test requires installation of dependencies
     use_sample_data: Run functional tests using real data
+    sequential: Run these tests sequentially, see https://github.com/ESMValGroup/ESMValCore/issues/644
 
 [coverage:run]
 parallel = true