-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: add --repeat option to tools/test.py #6700
Conversation
Concept sgtm but I'm not following that Python at all, what's with the new loop that just appends |
@thefourtheye you're a Python person too right? |
just fyi, the new loop same as this code
|
for v in VARIANT_FLAGS: | ||
tests = self.GetConfiguration(context).ListTests(current_path, path, | ||
arch, mode) | ||
for t in tests: t.variant_flags = v | ||
result += tests | ||
|
||
for rep in range(0, repeat): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
0 is not needed as it is the default. Moreover xrange is preferred over range in 2.x
Updated to address all comments now |
@@ -780,7 +779,7 @@ def GetTestStatus(self, context, sections, defs): | |||
class Context(object): | |||
|
|||
def __init__(self, workspace, buildspace, verbose, vm, args, expect_fail, | |||
timeout, processor, suppress_dialogs, store_unexpected_output): | |||
timeout, processor, suppress_dialogs, store_unexpected_output, repeat): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hate to be that guy but... long line. :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps, we can refactor this with dict unpacking.
LGTM with style nit. |
LGTM |
I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. �
Updated for last nit |
CI looks green except for pre-existing issue: #6716 and windows build that seem to be stuck on previous build |
Plan to land tomorrow morning unless there are any objections before then |
@@ -1324,6 +1325,9 @@ def BuildOptions(): | |||
default="") | |||
result.add_option('--temp-dir', | |||
help='Optional path to change directory used for tests', default=False) | |||
result.add_option('--repeat', | |||
help='how many times to repeat each test', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this be written as "Number of times to repeat given tests"
LGTM with the text update @thefourtheye suggests (or similar). |
Made sure that |
I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. PR-URL: #6700 Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com> Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com> Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Ok incorporated updated help message from thefourtheye and landed as ccbb00e |
I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. PR-URL: #6700 Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com> Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com> Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
@mhdawson this will need a manual backport to v4.x |
I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. PR-URL: #6700 Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com> Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com> Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
ping @mhdawson |
Still on my list, will try to do it this week. |
Manual backport from master to 4.x stream, original commit message follows. I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. PR-URL: nodejs#6700 Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com> Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com> Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Ok PR against v4.x-staging to backport #7740 FYI @gibfahn, @thealphanerd |
Manual backport from master to 4.x stream, original commit message follows. I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. PR-URL: #6700 Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com> Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com> Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Manual backport from master to 4.x stream, original commit message follows. I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. PR-URL: #6700 Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com> Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com> Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Manual backport from master to 4.x stream, original commit message follows. I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. PR-URL: #6700 Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com> Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com> Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Manual backport from master to 4.x stream, original commit message follows. I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. PR-URL: #6700 Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com> Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com> Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Manual backport from master to 4.x stream, original commit message follows. I often want to run a test many times to see if a failure can be recreated and I believe this is a common use case. We even have this job in the CI https://ci.nodejs.org/job/node-stress-single-test/configure but often you want to run it on a specific machine. This patch adds the --repeat option so that you can repeat the selected set of tests a number of times. Given existing options in test.py this will allow you to run one or more tests for the number of repeats specified. For example: tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv runs the test-process-exec-argv test 1000 times, running 8 copies in parallel tools/test.py --repeat 2 would run the entire test suite twice. PR-URL: #6700 Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Fedor Indutny <fedor@indutny.com> Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com> Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Checklist
Affected core subsystem(s)
test
Description of change
I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case. We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.
This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:
tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv
runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel
tools/test.py --repeat 2
would run the entire test suite twice
�