Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance Python regrtest test runner (test.libregrtest) #109276

Closed
vstinner opened this issue Sep 11, 2023 · 2 comments
Closed

Enhance Python regrtest test runner (test.libregrtest) #109276

vstinner opened this issue Sep 11, 2023 · 2 comments

Comments

@vstinner
Copy link
Member

vstinner commented Sep 11, 2023 •
edited by bedevere-app bot
Loading

In issue gh-109162, I refactored deeply libregrtest code to easy its maintenance. While refactoring libregrtest, I saw many things that I would like to enhance. So I create this meta ticket to track enhancements :-)

Linked PRs
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: use a separated file for JSON

Verified

This commit was signed with the committer’s verified signature.
kornelski Kornel
SSH Key Fingerprint: N6wZz6VB+UM3nZwtlH5DYLaDNnXME2ALnASwGhBDdNk
Verified
Learn about vigilant mode
44f7909 
libregrtest now uses a separated file descriptor to write test result
into JSON. Previously, if a test wrote debug messages around the
JSON, the main test process failed to parse JSON.

Rename TestResult.write_json() to TestResult.write_json_into().

worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: use separated file for JSON
fcdd93f 
libregrtest now uses a separated file descriptor to write test result
into JSON. Previously, if a test wrote debug messages around the
JSON, the main test process failed to parse JSON.

Rename TestResult.write_json() to TestResult.write_json_into().

worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: use separated file for JSON
477d7ce 
libregrtest now uses a separated file descriptor to write test result
as JSON.  Previously, if a test wrote debug messages late around the
JSON, the main test process failed to parse JSON.

Rename TestResult.write_json() to TestResult.write_json_into().

worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.
@bedevere-app bedevere-app bot mentioned this issue Sep 11, 2023
Merged
@vstinner
Copy link
Member Author

Recent libregrtest enhancements:

  • Worker processes no longer import test.libregrtest.main: the python -m test.libregrtest.worker command is used instead. Change: commit a939b65.
  • Worker processes are now run in a different working directory: in the temporary directory created by the main test process. Change: commit a939b65.
  • --failfast now fails faster (!): RegressionTestResult.addFailure() now calls self.stop is support.failfast is true. Change: commit 31c2945.

vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest only checks saved_test_environment() once
2ac5bd7 
There is no need to check for environment changes twice.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest only checks saved_test_environment() once
c1e9e8f 
There is no need to check for environment changes twice.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest only checks saved_test_environment() once
a0da31a 
There is no need to check for environment changes twice.
@bedevere-app bedevere-app bot mentioned this issue Sep 11, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: use separated file for JSON

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
4715098 
libregrtest now uses a separated file descriptor to write test result
as JSON.  Previously, if a test wrote debug messages late around the
JSON, the main test process failed to parse JSON.

Rename TestResult.write_json() to TestResult.write_json_into().

worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.

create_worker_process() now always spawn the process with
close_fds=True.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest calls random.seed() before each test
3c37f4b 
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once before running all
test files and it was not called in worker processes.

* Convert some f-strings to regular strings in test_regrtest when
  f-string is not needed.
* Remove unused all_methods variables from test_regrtest.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest calls random.seed() before each test
11de79d 
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.

* Convert some f-strings to regular strings in test_regrtest when
  f-string is not needed.
* Remove unused all_methods variables from test_regrtest.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest calls random.seed() before each test

Verified

This commit was signed with the committer’s verified signature.
bjorn3
SSH Key Fingerprint: 9p7UbsyTHW8iK9OhC2DaZMlEbAAKfX1yaRsZ/AgyljE
Verified
Learn about vigilant mode
75651e2 
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.

* Convert some f-strings to regular strings in test_regrtest when
  f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest calls random.seed() before each test

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
138b87a 
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.

* Convert some f-strings to regular strings in test_regrtest when
  f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest calls random.seed() before each test

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
b29982d 
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.

* Convert some f-strings to regular strings in test_regrtest when
  f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
* Add RunTests members are now mandatory.
@bedevere-app bedevere-app bot mentioned this issue Sep 11, 2023
Merged
vstinner added a commit that referenced this issue Sep 11, 2023
@vstinner
gh-109276: libregrtest: use separated file for JSON (#109277)

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
de5f8f7 
libregrtest now uses a separated file descriptor to write test result
as JSON.  Previously, if a test wrote debug messages late around the
JSON, the main test process failed to parse JSON.

Rename TestResult.write_json() to TestResult.write_json_into().

worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.

create_worker_process() now always spawn the process with
close_fds=True.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest calls random.seed() before each test

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
4506d1c 
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.

* Convert some f-strings to regular strings in test_regrtest when
  f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
* Add RunTests members are now mandatory.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: limit number workers

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
1bfd583 
Don't spawn more threads than the maximum number of jobs: these
worker threads will not get anything to do.

Add RunTests.get_jobs() method.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: limit number workers

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
5cc6ae9 
Don't spawn more threads than the number of jobs: these worker
threads would never get anything to do.

Add RunTests.get_jobs() method.
@bedevere-app bedevere-app bot mentioned this issue Sep 11, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: add RunTests.work_dir

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
ed8ab0f 
* WorkerThread now always creates a temporary directory, even on
  Emscripten and WASI: it's used as the working directory of the test
  worker process.
* Fix 'wasm32-emscripten node' and 'wasm32-wasi' buildbot workers:
  start the test worker process in the Python source code directory,
  since these buildbot workers need to get their 'python.js' and
  'python.wasm' file. Then worker_process() goes to the temporary
  directory created to run tests.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: add RunTests.work_dir

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
867ef99 
* WorkerThread now always creates a temporary directory, even on
  Emscripten and WASI: it's used as the working directory of the test
  worker process.
* Fix Emscripten and WASI: start the test worker process in the
  Python source code directory, where 'python.js' and 'python.wasm'
  can be found. Then worker_process() goes to the temporary directory
  created to run tests.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: add RunTests.work_dir

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
8a62de0 
* WorkerThread now always creates a temporary directory, even on
  Emscripten and WASI: it's used as the working directory of the test
  worker process.
* Fix Emscripten and WASI: start the test worker process in the
  Python source code directory, where 'python.js' and 'python.wasm'
  can be found. Then worker_process() goes to the temporary directory
  created to run tests.
@bedevere-app bedevere-app bot mentioned this issue Sep 11, 2023
Closed
vstinner added a commit to vstinner/cpython that referenced this issue Sep 11, 2023
@vstinner
pythongh-109276: libregrtest: add RunTests.work_dir
3169629 
* WorkerThread now always creates a temporary directory, even on
  Emscripten and WASI: it's used as the working directory of the test
  worker process.
* Fix Emscripten and WASI: start the test worker process in the
  Python source code directory, where 'python.js' and 'python.wasm'
  can be found. Then worker_process() goes to the temporary directory
  created to run tests.
* --cleanup now also removes "temporary" directories of test worker
  processes (in /tmp).
vstinner added a commit to vstinner/cpython that referenced this issue Sep 12, 2023
@vstinner
pythongh-109276: libregrtest: add RunTests.work_dir
b7dec3c 
* WorkerThread now always creates a temporary directory, even on
  Emscripten and WASI: it's used as the working directory of the test
  worker process.
* Fix Emscripten and WASI: start the test worker process in the
  Python source code directory, where 'python.js' and 'python.wasm'
  can be found. Then worker_process() goes to the temporary directory
  created to run tests.
* --cleanup now also removes "temporary" directories of test worker
  processes (in /tmp).
* test.pythoninfo now also logs sysconfig 'abs_builddir' and
  'srcdir', and sysconfig.is_python_build().
vstinner added a commit to vstinner/cpython that referenced this issue Sep 12, 2023
@vstinner
pythongh-109276: libregrtest: add RunTests.work_dir
0507687 
* WorkerThread now always creates a temporary directory, even on
  Emscripten and WASI: it's used as the working directory of the test
  worker process.
* Fix Emscripten and WASI: start the test worker process in the
  Python source code directory, where 'python.js' and 'python.wasm'
  can be found. Then worker_process() goes to the temporary directory
  created to run tests.
* --cleanup now also removes "temporary" directories of test worker
  processes (in /tmp).
* test.pythoninfo now also logs:

  * sysconfig 'abs_builddir' and 'srcdir'
  * sysconfig.is_python_build()
  * tempfile.gettempdir()
vstinner added a commit to vstinner/cpython that referenced this issue Sep 12, 2023
@vstinner
pythongh-109276: Complete test.pythoninfo
acbb67c 
* Enable collect_sysconfig() on Windows.
* Add sysconfig 'abs_builddir' and 'srcdir'
* Add sysconfig.is_python_build()
* Add tempfile.gettempdir()
* Remove compatiblity with Python 2.7 (print_function).
@bedevere-app bedevere-app bot mentioned this issue Sep 12, 2023
Merged
@bedevere-app bedevere-app bot mentioned this issue Sep 25, 2023
Merged
vstinner added a commit that referenced this issue Sep 25, 2023
@vstinner
gh-109276: Enhance libregrtest results (#109828)
faebed4 
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
vstinner added a commit that referenced this issue Sep 25, 2023
@vstinner
gh-109276: regrtest re-runs "env changed" tests (#109831)
72fb39c 
When a test fails with "env changed" and --rerun option is used, the
test is now re-run in verbose mode in a fresh process.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109276, pythongh-109508: Fix libregrtest stdout
805258a 
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109276, pythongh-109508: Fix libregrtest stdout
f2d228d 
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109276, pythongh-109508: Fix libregrtest stdout
dd77210 
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
@bedevere-app bedevere-app bot mentioned this issue Sep 26, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109276, pythongh-109508: Fix libregrtest stdout
f3118a2 
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109276, pythongh-109508: Fix libregrtest stdout
a36b87c 
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109276, pythongh-109508: Fix libregrtest stdout
b5c891b 
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
vstinner added a commit that referenced this issue Sep 26, 2023
@vstinner
gh-109276, gh-109508: Fix libregrtest stdout (#109903)
b1e4f6e 
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023
@vstinner @csm10495
pythongh-109276: Enhance libregrtest results (python#109828)
9f395fc 
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023
@vstinner @csm10495
pythongh-109276: regrtest re-runs "env changed" tests (python#109831)
82ab866 
When a test fails with "env changed" and --rerun option is used, the
test is now re-run in verbose mode in a fresh process.
csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023
@vstinner @csm10495
pythongh-109276, pythongh-109508: Fix libregrtest stdout (python#109903)
5c99caf 
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-109276: regrtest: add WORKER_FAILED state
a9c4660 
WORKER_FAILED does not stop the manager, whereas WORKER_BUG does.

Change also TestResults.display_result() order: display failed tests
at the end, the important important information.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-109276: regrtest: add WORKER_FAILED state
be9510a 
WORKER_FAILED does not stop the manager, whereas WORKER_BUG does.

Change also TestResults.display_result() order: display failed tests
at the end, the important important information.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-109276: regrtest: add WORKER_FAILED state
8328f37 
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.

Change also TestResults.display_result() order: display failed tests
at the end, the important important information.
@bedevere-app bedevere-app bot mentioned this issue Sep 30, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-109276: regrtest: add WORKER_FAILED state
2a3e1a2 
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.

Change also TestResults.display_result() order: display failed tests
at the end, the important important information.

WorkerThread now tries to get the signal name for negative exit code.
vstinner added a commit that referenced this issue Sep 30, 2023
@vstinner
gh-109276: regrtest: add WORKER_FAILED state (#110148)
2c23419 
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.

Change also TestResults.display_result() order: display failed tests
at the end, the important important information.

WorkerThread now tries to get the signal name for negative exit code.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 4, 2023
@vstinner
pythongh-109276: regrtest: shorter list of resources
1376d46
@bedevere-app bedevere-app bot mentioned this issue Oct 4, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Oct 4, 2023
@vstinner
pythongh-109276: regrtest: shorter list of resources
73ad79e
vstinner added a commit to vstinner/cpython that referenced this issue Oct 4, 2023
@vstinner
pythongh-109276: regrtest: shorter list of resources
68f2ade
vstinner added a commit that referenced this issue Oct 4, 2023
@vstinner
gh-109276: regrtest: shorter list of resources (#110326)
efd8c7a
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109276: Enhance libregrtest results (python#109828)
4f78dff 
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109276: regrtest re-runs "env changed" tests (python#109831)
0e8f38d 
When a test fails with "env changed" and --rerun option is used, the
test is now re-run in verbose mode in a fresh process.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109276, pythongh-109508: Fix libregrtest stdout (python#109903)
35a5968 
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109276: regrtest: add WORKER_FAILED state (python#110148)
11ec62f 
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.

Change also TestResults.display_result() order: display failed tests
at the end, the important important information.

WorkerThread now tries to get the signal name for negative exit code.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109276: regrtest: shorter list of resources (python#110326)
5a3e395
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vstinner