Allow regrtest to receive a file with test (and subtests) to ignore

[pablogsal]

BPO	37957
Nosy	@vstinner, @serhiy-storchaka, @pablogsal
PRs	bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore #16989

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-08-27.00:03:54.483>
labels = ['type-feature', 'tests', '3.9']
title = 'Allow regrtest to receive a file with test (and subtests) to ignore'
updated_at = <Date 2020-04-14.23:05:24.243>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2020-04-14.23:05:24.243>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2019-08-27.00:03:54.483>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 37957
keywords = ['patch']
message_count = 11.0
messages = ['350584', '350601', '350606', '350607', '350617', '350626', '357006', '357015', '357016', '366411', '366460']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'pablogsal']
pr_nums = ['16989']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37957'
versions = ['Python 3.9']

[pablogsal]

When building Python in some uncommon platforms (I am looking at you Solaris and AIX) there are some known tests that will fail. Right now, regrtest has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter.

The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept.

The problem with -x is that is not easy to ignore just a subtests that fail and the whole tests needs to be ignored.

So I suggest to add a new command line option similar to --matchfile but the other way around. Another possibility is allowing to reverse the meaning of the matchfile argument, but I find that a bit more confusing.

[vstinner]

I'm fine with adding an --excludefile=filename option to regrtest. Do you want to try to implement it? A test can be added in test_regrtest.

[serhiy-storchaka]

Tests that fail on some platforms can be marked to skip in code.

For example:

@unittest.skipIf(sys.platform.startswith('aix'),
                 'bpo-29972: broken test on AIX')
def test_strcoll_with_diacritic(self):
    self.assertLess(locale.strcoll('à', 'b'), 0)

[vstinner]

Tests that fail on some platforms can be marked to skip in code.

We provide best-effort support for AIX, but I consider that Solaris is no longer supported, and so I would prefer to avoid code specific to Soliars in the Python source code. Except if Pablo wants to provide again best-effort support for Solaris?
https://pythondev.readthedocs.io/platforms.html

[pablogsal]

Tests that fail on some platforms can be marked to skip in code.

One of the use cases of this feature is for people to package Python at distributions or companies and want to run the test suite without modifying the code or applying patches.

[vstinner]

FYI in the past, Fedora and RHEL packages skipped multiple tests using downstream patches. Hopefully, we managed to reenable almost all tests.

Currently in Fedora, the python3 package use regrtest -x to skip some tests:

# Run the upstream test suite
# test_gdb skipped on s390x:
# https://bugzilla.redhat.com/show_bug.cgi?id=1678277
# test_gdb skipped everywhere:
# https://bugzilla.redhat.com/show_bug.cgi?id=1734327
# test_distutils
# distutils.tests.test_bdist_rpm tests fail when bootstraping the Python
# package: rpmbuild requires /usr/bin/pythonX.Y to be installed
LD_LIBRARY_PATH=$ConfDir $ConfDir/python -m test.regrtest \
-wW --slowest -j0 \
%if %{with bootstrap}
-x test_distutils \
%endif
-x test_gdb \
%ifarch %{mips64}
-x test_ctypes \
%endif

https://src.fedoraproject.org/rpms/python3/blob/master/f/python3.spec

We could only skip some test methods, but it's simpler for us to skip a whole test file.

[pablogsal]

New changeset e0cd8aa by Pablo Galindo in branch 'master':
bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)
e0cd8aa

[vstinner]

I'm trying to keep regrtest in sync between 3.7, 3.8 and master branches. Maybe backport this change to 3.7 and 3.8 branches once buildbots validated the change? I'm keeping them in sync mostly to make bugfixes easier, but also to make my life easier when I debug issues in 3.7 or 3.8 branch.

[pablogsal]

Ok, I reopen until the backports