using argparse in sage-runtests script

[fchapoton]

instead of the deprecated optparse

This was rather simple, but needs testing.

also full flake8 cleanup for the modified file

CC: @tscrim @slel @kliem @antonio-rojas

Component: scripts

Author: Frédéric Chapoton

Branch: c87148b

Reviewer: Travis Scrimshaw

Issue created by migration from https://trac.sagemath.org/ticket/32332

[fchapoton]

Branch: u/chapoton/32332

[fchapoton]

Commit: 32d10d1

[fchapoton]

New commits:

32d10d1

use argparse in sage-runtests

[sagetrac-git]

Changed commit from 32d10d1 to 8c92fc4

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

8c92fc4

simplify GCAction in sage-runtests

[tscrim]

Reviewer: Travis Scrimshaw

[tscrim]

comment:4

LGTM as far as I can tell. A good ticket for the initial 9.5.beta0 release.

[vbraun]

comment:5

make ptestlong
[...]
make[1]: Leaving directory '/home/release/Sage'
./sage -t -p --all --long --logfile=logs/ptestlong.log
usage: sage -t [options] filenames
sage-runtests: error: argument -p/--nthreads: expected one argument
make: *** [Makefile:201: ptestlong] Error 2

[sagetrac-git]

Changed commit from 8c92fc4 to 1d3dfc3

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

0d148cb	Merge branch 'u/chapoton/32332' in 9.5.b0
1d3dfc3	fix for argparse when using -p

[sagetrac-git]

Changed commit from 1d3dfc3 to 4aa95ed

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

4aa95ed

another fix for the logfile

[fchapoton]

comment:8

ok, should be better now. I fixed the behaviour of -p and --logfile.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

8a458aa

better handling for -p

[sagetrac-git]

Changed commit from 4aa95ed to 8a458aa

[tscrim]

comment:10

There are still issues:

$ ./sage -btp src/sage/combinat/partition.py
...
sage-runtests: error: argument -p/--nthreads: invalid int value: 'src/sage/combinat/partition.py'
$ ./sage -tp src/sage/combinat/partition.py 
usage: sage -t [options] filenames
sage-runtests: error: argument -p/--nthreads: invalid int value: 'src/sage/combinat/partition.py'

I at least use these shortcuts a lot, but I guess that is not a strict requirement to have them fixed. It would just mean a lot of relearning for me.

These both continue to not work (so they don't need to be fixed)

$ ./sage -btp8 src/sage/combinat/partition.py 
Error: unknown option: -btp8
$ ./sage -tp8 src/sage/combinat/partition.py 
Error: unknown option: -tp8

but this now works (before it didn't):

$ ./sage -t -p8 src/sage/combinat/partition.py

[tscrim]

comment:28

Thank you. We are getting close.

$ ./sage -tp --warn-long src/sage/rings/function_field/function_field.py

works, but this does not (although it used to)

$ ./sage -tp src/sage/rings/function_field/function_field.py --warn-long
too few successful tests, not using stored timings
Running doctests with ID 2021-09-13-14-49-55-02c83a90.
Git branch: u/chapoton/32332
Using --optional=bliss,build,debian,dochtml,dot2tex,e_antic,libnauty,ninja_build,normaliz,pip,sage,sage_spkg
Traceback (most recent call last):
  File "/home/uqtscrim/sage-build/src/bin/sage-runtests", line 149, in <module>
    err = DC.run()
  File "/home/uqtscrim/sage-build/local/lib/python3.9/site-packages/sage/doctest/control.py", line 1204, in run
    self.expand_files_into_sources()
  File "/home/uqtscrim/sage-build/local/lib/python3.9/site-packages/sage/doctest/control.py", line 788, in expand_files_into_sources
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/home/uqtscrim/sage-build/local/lib/python3.9/site-packages/sage/doctest/control.py", line 788, in <listcomp>
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/home/uqtscrim/sage-build/local/lib/python3.9/site-packages/sage/doctest/sources.py", line 527, in __init__
    raise ValueError("unknown file extension %r"%ext)
ValueError: unknown file extension ''

Although I could very easily argue that this is not supported behavior, so it might be more luck than anything it worked before. This is a very minor point and not something I would withhold a positive review for.

[fchapoton]

comment:29

I do not think that we should allow options after the filenames.

The usage is "options then filenames", and that's all.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

66083fd	Merge branch 'u/chapoton/32332' in 9.5.b1
04e40ff	fine tuning short and warn-long options

[sagetrac-git]

Changed commit from d057ee2 to 04e40ff

[tscrim]

comment:31

Okay, then let's not (unofficially) support that input format anymore.

Shouldn't --short take floating points? Granted, I find it unlikely that someone will want to do something in 321.0123 seconds, but perhaps for one file they don't want whole seconds? The output for --short has also become a lot more verbose. (Not that that is a bad thing, it is just a change in behavior.)

Anyways, this should hopefully be the last of it. The sooner we can get this merged and out for people to test it more in the wild, the better we can pick up other changes in behavior (that people care about).

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

84471a3

fix for --short

[sagetrac-git]

Changed commit from 04e40ff to 84471a3

[fchapoton]

comment:33

ok, short was indeed broken by the change of default value. Repaired now.

Short was taking integers and will still take integers. I would prefer not to change that.

[tscrim]

comment:34

Okay, thank you. Let's get this tested by the rest of the developers.

[vbraun]

comment:35

sage -t --long --warn-long 44.9 --random-seed=0 src/sage/doctest/test.py  # 21 doctests failed
sage -t --long --warn-long 44.9 --random-seed=0 src/sage/doctest/forker.py  # 11 doctests failed
sage -t --long --warn-long 44.9 --random-seed=0 src/sage/doctest/control.py  # Bad exit: 1 after testing finished

[sagetrac-git]

Changed commit from 84471a3 to c87148b

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

c87148b

fixing logfile and warn-long handling

[fchapoton]

comment:37

damn. I have fixed a few things again.

[tscrim]

comment:38

Green bot and passes locally.

[vbraun]

Changed branch from u/chapoton/32332 to c87148b

[antonio-rojas]

Changed commit from c87148b to none

[antonio-rojas]

comment:40

This breaks running sage -t file --optional=foo for me, regardless of the optional tags I specify

> sage -t /usr/lib/python3.9/site-packages/sage/sets/integer_range.py --optional=sage
too many failed tests, not using stored timings
Running doctests with ID 2021-10-12-12-41-27-6d41dc7c.
Using --optional=dochtml,pip,sage
Traceback (most recent call last):
  File "/usr/bin/sage-runtests", line 151, in <module>
    err = DC.run()
  File "/usr/lib/python3.9/site-packages/sage/doctest/control.py", line 1210, in run
    self.expand_files_into_sources()
  File "/usr/lib/python3.9/site-packages/sage/doctest/control.py", line 794, in expand_files_into_sources
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/usr/lib/python3.9/site-packages/sage/doctest/control.py", line 794, in <listcomp>
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/usr/lib/python3.9/site-packages/sage/doctest/sources.py", line 527, in __init__
    raise ValueError("unknown file extension %r"%ext)
ValueError: unknown file extension ''

[antonio-rojas]

comment:42

Nevermind, I see this is intentional (re comment 29)