Trac #31924: sage -t: Do not run pytest on individual Python files un…

…less they match the pytest file pattern

`sage -t` is broken since #31003/#31103 because
- pytest is configured to look for methods prefixed with test_ and
treats them as pytest test functions/methods, which are then executed.
However, sage's code base contains some functions in its codebase that
match this pattern without being pytests.
- pytest (#31003) is configured to only discover `*_test.py` files; but
if one uses `sage -t` with file arguments, this will override it and
lead to many errors.

{{{
$ ./sage -t  src/sage/databases/*.py
too many failed tests, not using stored timings
Running doctests with ID 2021-06-07-11-04-14-3c1f8784.
Using --optional=4ti2,bliss,build,database_knotinfo,dochtml,e_antic,home
brew,jupymake,latte_int,lidia,lrslib,normaliz,pip,sage,sage_spkg
Doctesting 16 files.
sage -t --random-seed=0 src/sage/databases/__init__.py
    [0 tests, 0.00 s]
sage -t --random-seed=0 src/sage/databases/all.py
    [5 tests, 0.10 s]
sage -t --random-seed=0 src/sage/databases/conway.py
    [42 tests, 0.10 s]
sage -t --random-seed=0 src/sage/databases/cremona.py
    [133 tests, 0.31 s]
sage -t --random-seed=0 src/sage/databases/cunningham_tables.py
    [0 tests, 0.00 s]
sage -t --random-seed=0 src/sage/databases/db_class_polynomials.py
    [7 tests, 0.01 s]
sage -t --random-seed=0 src/sage/databases/db_modular_polynomials.py
    [13 tests, 0.01 s]
sage -t --random-seed=0 src/sage/databases/findstat.py
    [122 tests, 0.25 s]
sage -t --random-seed=0 src/sage/databases/jones.py
    [8 tests, 0.05 s]
sage -t --random-seed=0 src/sage/databases/knotinfo_db.py
    [97 tests, 1.80 s]
sage -t --random-seed=0 src/sage/databases/odlyzko.py
    [0 tests, 0.00 s]
sage -t --random-seed=0 src/sage/databases/oeis.py
    [134 tests, 0.91 s]
sage -t --random-seed=0 src/sage/databases/sloane.py
    [0 tests, 0.00 s]
sage -t --random-seed=0 src/sage/databases/sql_db.py
    [293 tests, 0.27 s]
sage -t --random-seed=0 src/sage/databases/stein_watkins.py
    [12 tests, 0.01 s]
sage -t --random-seed=0 src/sage/databases/symbolic_data.py
    [0 tests, 0.00 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 4.6 seconds
    cpu time: 3.9 seconds
    cumulative wall time: 3.8 seconds
========================================================================
====== test session starts =============================================
==================================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/src,
configfile: tox.ini
collected 0 items / 7 errors

========================================================================
============= ERRORS ===================================================
==================================
_____________________________________________________________________
ERROR collecting sage/databases/all.py
_____________________________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-
rebasing/worktree-gcc11/src/sage/databases/all.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
src/sage/databases/all.py:51: in <module>
    from .sql_db import SQLQuery, SQLDatabase
E   ImportError: attempted relative import with no known parent package
_____________________________________________________________________
ERROR collecting sage/databases/all.py
_____________________________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-
rebasing/worktree-gcc11/src/sage/databases/all.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
src/sage/databases/all.py:51: in <module>
    from .sql_db import SQLQuery, SQLDatabase
E   ImportError: attempted relative import with no known parent package
___________________________________________________________________
ERROR collecting sage/databases/cremona.py
___________________________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-
rebasing/worktree-gcc11/src/sage/databases/cremona.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
src/sage/databases/cremona.py:52: in <module>
    from .sql_db import SQLDatabase, verify_column
E   ImportError: attempted relative import with no known parent package
____________________________________________________________ ERROR
collecting sage/databases/db_class_polynomials.py
_____________________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-
rebasing/worktree-gcc11/src/sage/databases/db_class_polynomials.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
src/sage/databases/db_class_polynomials.py:14: in <module>
    from .db_modular_polynomials import _dbz_to_integers
E   ImportError: attempted relative import with no known parent package
__________________________________________________________________ ERROR
collecting sage/databases/findstat.py
___________________________________________________________________
src/sage/databases/findstat.py:329: in <module>
    class FindStat(UniqueRepresentation, SageObject):
sage/misc/nested_class.pyx:318: in
sage.misc.nested_class.NestedClassMetaclass.__init__
    ???
E   KeyError: 'findstat'
_________________________________________________________________ ERROR
collecting sage/databases/knotinfo_db.py
_________________________________________________________________
src/sage/databases/knotinfo_db.py:330: in <module>
    class KnotInfoDataBase(SageObject, UniqueRepresentation):
sage/misc/nested_class.pyx:318: in
sage.misc.nested_class.NestedClassMetaclass.__init__
    ???
E   KeyError: 'knotinfo_db'
____________________________________________________________________
ERROR collecting sage/databases/oeis.py
_____________________________________________________________________
src/sage/databases/oeis.py:651: in <module>
    class OEISSequence(SageObject, UniqueRepresentation):
sage/misc/nested_class.pyx:318: in
sage.misc.nested_class.NestedClassMetaclass.__init__
    ???
E   KeyError: 'oeis'
========================================================================
==== short test summary info ===========================================
==================================
ERROR src/sage/databases/all.py
ERROR src/sage/databases/all.py
ERROR src/sage/databases/cremona.py
ERROR src/sage/databases/db_class_polynomials.py
ERROR src/sage/databases/findstat.py - KeyError: 'findstat'
ERROR src/sage/databases/knotinfo_db.py - KeyError: 'knotinfo_db'
ERROR src/sage/databases/oeis.py - KeyError: 'oeis'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Interrupted: 7 errors during collection
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
}}}

Related report in https://groups.google.com/g/sage-
devel/c/SE_A2Jw5Kko/m/KQpA9GbjBQAJ:
{{{
=============================================================== ERRORS
================================================================
_________________________________________ ERROR collecting
sage/structure/sage_object_test.py
_________________________________________
ImportError while importing test module
'/home/john/sage/src/sage/structure/sage_object_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
src/sage/structure/sage_object_test.py:3: in <module>
from .sage_object import SageObject
E ImportError: attempted relative import with no known parent package
======================================================= short test
summary info =======================================================
ERROR src/sage/structure/sage_object_test.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error
during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
}}}

In this ticket, we fix it by passing names of regular files to `pytest`
only if they match the pytest pattern (ending with `_test.py`).

(The dependency #32975 renamed the files in the Sage sources that used
this naming scheme but were not pytest files.)

Example:
{{{
$ ./sage -tp src/sage/numerical/backends
src/sage/symbolic/expression.pyx
src/sage/manifolds/differentiable/symplectic_form_test.py
too many failed tests, not using stored timings
Running doctests with ID 2022-03-22-11-30-32-fd280468.
Using --optional=4ti2,buckygen,ccache,cryptominisat,debugpy,e_antic,gap_
packages,homebrew,igraph,jupymake,latte_int,libsemigroups,lidia,lrslib,m
eataxe,normaliz,pip,polytopes_db_4d,pynormaliz,sage,sage_spkg
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,
csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_j
ones_numfield,database_knotinfo,dvipng,graphviz,imagemagick,jupymake,ken
zo,latte_int,lrslib,mcqd,meataxe,nauty,palp,pandoc,pdf2svg,pdftocairo,pl
antri,polytopes_db,polytopes_db_4d,pynormaliz,python_igraph,rubiks,sage.
combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.plot,sage
.rings.number_field,sage.rings.padics,sage.rings.real_double,sage.symbol
ic,sage_numerical_backends_coin,sagemath_doc_html,sphinx,tdlib
Sorting sources by runtime so that slower doctests are run first....
Doctesting 27 files using 8 threads.
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/glpk_backend.pxd
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/interactivelp_backend.pxd
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/generic_backend.pxd
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/glpk_graph_backend.pxd
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/matrix_sdp_backend.pxd
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/cvxopt_sdp_backend.pyx
    [52 tests, 0.08 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/generic_sdp_backend.pxd
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/matrix_sdp_backend.pyx
    [87 tests, 0.14 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/glpk_graph_backend.pyx
    [193 tests, 0.17 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/ppl_backend.pyx
    [221 tests, 0.25 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/logging_backend.py
    [45 tests, 0.05 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/glpk_exact_backend.pxd
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/generic_sdp_backend.pyx
    [37 tests, 0.07 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/manifolds/differentiable/symplectic_form_test.py
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/interactivelp_backend_test.py
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/cvxopt_backend.pyx
    [25 tests, 0.05 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/glpk_exact_backend.pyx
    [24 tests, 0.03 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/cvxopt_backend_test.py
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/ppl_backend_test.py
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/glpk_backend_test.py
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/__init__.py
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/generic_backend_test.py
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/glpk_exact_backend_test.py
    [0 tests, 0.00 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/generic_backend.pyx
    [96 tests, 0.59 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/interactivelp_backend.pyx
    [266 tests, 3.09 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/numerical/backends/glpk_backend.pyx
    [592 tests, 3.08 s]
sage -t --random-seed=156872034187085869352407545599456758260
src/sage/symbolic/expression.pyx
    [3070 tests, 31.83 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 32.6 seconds
    cpu time: 39.3 seconds
    cumulative wall time: 39.4 seconds
Features detected for doctesting: sage.rings.number_field
========================================================================
==================================== test session starts ===============
========================================================================
=====================
platform darwin -- Python 3.7.8, pytest-7.1.1, pluggy-1.0.0
rootdir: /Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/src,
configfile: tox.ini
collected 32 items

src/sage/numerical/backends/cvxopt_backend_test.py ..
[  6%]
src/sage/numerical/backends/glpk_backend_test.py ..
[ 12%]
src/sage/numerical/backends/glpk_exact_backend_test.py ..
[ 18%]
src/sage/numerical/backends/interactivelp_backend_test.py ..
[ 25%]
src/sage/numerical/backends/ppl_backend_test.py ..
[ 31%]
src/sage/manifolds/differentiable/symplectic_form_test.py
......................
[100%]

========================================================================
===============
}}}

URL: https://trac.sagemath.org/31924
Reported by: mkoeppe
Ticket author(s): Matthias Koeppe
Reviewer(s): Sebastian Oehms

src/bin/sage-runtests

@@ -156,10 +156,17 @@ if __name__ == "__main__":
         pytest_options = ["--import-mode", "importlib"]
         if args.verbose:
             pytest_options.append("-v")
-        exit_code_pytest = pytest.main(pytest_options + args.filenames)
-        if exit_code_pytest == 5:
-            # Exit code 5 means there were no test files, pass in this case
-            exit_code_pytest = 0
+        # #31924: Do not run pytest on individual Python files unless
+        # they match the pytest file pattern.  However, pass names
+        # of directories. We use 'not os.path.isfile(f)' for this so that
+        # we do not silently hide typos.
+        filenames = [f for f in args.filenames
+                     if f.endswith("_test.py") or not os.path.isfile(f)]
+        if filenames or not args.filenames:
+            exit_code_pytest = pytest.main(pytest_options + filenames)
+            if exit_code_pytest == 5:
+                # Exit code 5 means there were no test files, pass in this case
+                exit_code_pytest = 0
 
     except ModuleNotFoundError:
         print("Pytest is not installed, skip checking tests that rely on it.")