Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update slow tests marks and add script to automate #1164

Merged
merged 7 commits into from
Sep 24, 2024

Conversation

matt-graham
Copy link
Collaborator

@matt-graham matt-graham commented Oct 16, 2023

Fixes #1145

Adds a new script src/scripts/automation/mark_slow_tests.py which uses a JSON report of Pytest results generated using pytest-json-report plugin to identify slow running tests and update the decorators used to mark the tests in the source code. The script uses RedBaron to parse the modules' source code in to a lossless abstract source tree, and adds or removes the slow mark decorators to the relevant test functions identified from the test report. To avoid too much switching back and forth for tests with duration close to the threshold used to determine if a test is 'slow' (currently ~10s), a basic form of hysteresis is used with slow marks only added if measured test duration is above a upper threshold (default 11s) and only removed if test duration is below a slower threshold (default 9s). A tox environment which install additional dependencies (pytest-json-report and redbaron), runs pytest with JSON report output and then runs mark_slow_tests.py script to update the tests is also added.

Updates to the slow marks based on a local run of the tests and script are also included in this PR - we could possibly separate this out in to a different PR. It would probably also be worth exploring automating running the script as an Actions workflow, potentially doing something like running on a weekly schedule and opening a PR for any changes to slow marks.

@matt-graham matt-graham requested a review from tamuri October 16, 2023 09:34
@matt-graham matt-graham marked this pull request as draft October 16, 2023 09:38
@matt-graham
Copy link
Collaborator Author

matt-graham commented Oct 16, 2023

Check failing due to not removing pytest import on modules for which all slow marks have been removed and no other pytest usages.

  • Add check if there are any remaining pytest usages after removing slow marks and remove pytest import if so

@matt-graham matt-graham marked this pull request as ready for review October 16, 2023 13:02
@matt-graham matt-graham force-pushed the mmg/mark-slow-tests branch from d3f2dee to 7c9a462 Compare August 1, 2024 14:37
@matt-graham
Copy link
Collaborator Author

In the interests of reviving this PR and keep the changes more minimal and so easier to review, I've remove the commits which ran this script to update the existing slow marks in the tests so this PR now only adds the script for automatically updating slow marks plus corresponding tox environment. We can look at actually applying to update the slow tests marks in a separate PR or series of PRs.

@matt-graham matt-graham merged commit 66f5028 into master Sep 24, 2024
59 checks passed
@matt-graham matt-graham deleted the mmg/mark-slow-tests branch September 24, 2024 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Mark new slow running tests
2 participants