Meta-ticket: Adopt mainstream Python testing/linting infrastructure: tox, pytest, ..., describe in Developer's Guide

[mkoeppe]

This metaticket proposes to use mainstream Python testing and linting infrastructure for Sage -- by adding configuration files and documentation.

tox:

• Add src/tox.ini as a standard entrypoint for doctesting #28987 Add src/tox.ini: As an alternative interface to running the sage doctester.
• src/tox.ini: Check patchbot plugin patterns and common typo patterns #30467 src/tox.ini: Check patchbot plugin patterns
• Command "sage -tox" #30410 (Command "sage -tox")
• Make tox a standard package #30416 (Make tox a standard package)
• Document "sage -tox" #30453 Document "sage -tox"
• src/tox.ini: Add environment pyright #30411 (src/tox.ini: Add environment pyright)
• src/tox.ini: Add validation of docstrings using flake8-rst-docstrings #30448 (src/tox.ini: Add validation of rst files and docstrings via a quick docbuild)

pytest:

• Add minimal pytest configuration #31003 Add minimal pytest configuration
• sage.numerical.backends: Replace use of TestSuite by pytest #31103 sage.numerical.backends: Replace use of TestSuite by pytest
• Add symplectic structures #30362 Add symplectic structures
• Improve doctest interaction with pytest #32975 Improve doctest interaction with pytest
• Rename test_... functions to avoid the naming scheme reserved by pytest #33549 Rename test_... functions to avoid the naming scheme reserved by pytest
• sage -t: Do not run pytest on individual Python files unless they match the pytest file pattern #31924 sage -t: Do not run pytest on individual Python files unless they match the pytest file pattern
• sage -pytest #33572 sage -pytest
• pytest: ignore cython files #33560 pytest: ignore cython files
• Rename test_... classes to avoid the naming scheme reserved by pytest #33561 Rename test_... classes to avoid the naming scheme reserved by pytest
• Preparse sage files before passing them to pytest #33550 Preparse sage files before passing them to pytest
• Document pytest #31123 Document pytest
• doctesting with pytest fails on system install of sage #33521 doctesting with pytest fails on system install of sage
• pytest in sage is broken, please make it really optional #33531 pytest in sage is broken, please make it really optional
• Make pytest a standard package #31110 Make pytest a standard package
• Use pytest to run doctests #33546 Replace custom Sage doctest discovery by pytest (using pytest_collect_file)
• Replace custom Sage unit test discovery by pytest #30738 Replace custom Sage unit test discovery by pytest
• sage.geometry.polyhedron: Replace use of TestSuite by pytest #31104 sage.geometry.polyhedron: Replace use of TestSuite by pytest

Other tickets:

• "sage -startuptime" should accept file names too #30452 ("sage -startuptime" should accept file names too)

• Add pyright config and linting documentation #30361 (Add pyright config and linting documentation; dup: sage developer's guide should mention pyflakes #17992)

• sage -t : Write report in JUnitXML format #33232 (sage -t: Write report in JUnitXML format)

• Remove pyflakes? #30415 (Remove pyflakes?)

Documentation tickets:

• Update section "Sage Development Process" in Developer's Guide #29784 Update section "Sage Development Process" in Developer's Guide

• Mention .lgtm.yml in the Developer's Guide #29520: Mention .lgtm.yml in the Developer's Guide

Infrastructure tickets:

• Meta-ticket: Migration from Trac to GitHub #30363 Replace patchbot by github actions (part of https://trac.sagemath.org/ticket/)

• Use pre-commit https://pre-commit.com/

• Use pipenv in place of venv https://pipenv.pypa.io/en/latest/#

See also:

• Meta-ticket Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages #29060: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

• Generalize sage.numerical.backends.logging_backend.LoggingBackend to help generate _test methods

• Add script-type _spkg_check package that installs tox, mock, pytest etc. #28998: Add script-type check_dependencies package that installs tox, mock, pytest etc.

• Extensive description of different linters: https://github.com/vintasoftware/python-linters-and-code-analysis

CC: @jdemeyer @dimpase @embray @videlec @fchapoton @tscrim @jm58660 @tobiasdiez @kliem @nthiery @tornaria

Component: doctest framework

Issue created by migration from https://trac.sagemath.org/ticket/28936

[embray]

Replying to @mkoeppe:

tox seems to be the preferred interface for running tests - see for example pypa/setuptools#931 (deprecation of setup.py test).

I don't know what you mean by "preferred interface" here. There isn't really any such thing as a preferred interface. It's just being proposed here as a recommendation over setup.py test. Sage doesn't use setup.py test anyways so there's nothing that needs to be "replaced" with a different interface if/when setup.py goes away.

The main advantage to tox is the ability to run a package's tests across multiple runtime environments (esp. different Python versions). This isn't really possible currently with Sage; it would actually be very difficult to use tox in any sensible manner with Sage.

If someone wants to do this anyways I don't see the harm really, but I'm not sure what problem it would be solving.

[mkoeppe]

comment:4

A first step: #28987

[mkoeppe]

comment:5

Replying to @embray:

Replying to @mkoeppe:
The main advantage to tox is the ability to run a package's tests across multiple runtime environments (esp. different Python versions). This isn't really possible currently with Sage; it would actually be very difficult to use tox in any sensible manner with Sage.

FWIW, I managed to use tox on top of the sage installation, see https://github.com/mkoeppe/sage-numerical-backends-gurobi/blob/master/tox.ini: The trick is to run tox with sitepackages = true and an install_command that installs into the user scheme. With this trick, after all, no new plugin for tox is needed to handle the sage environment.

[mkoeppe]

comment:6

I have broadened the ticket description.

[mkoeppe]

comment:8

pushing these forward to 9.2

[tobiasdiez]

comment:31

What do you think about using pytest-cov for generating the test coverage report? The results can then be uploded to codecov as part of the build process (github action). This gives a nice overview of the actual test coverage (i.e. code covered by tests) and could replace the sage-coverage scripts in the future.

[mkoeppe]

comment:32

+1 on investigating this. Myself I have no experience with the available tools (including the sage-coverage scripts even though I have fixed them up at least once). As usual, it's best to build a prototype and then demonstrate to the sage-devel community that it is useful.