Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: add --repeat option to tools/test.py #6700

Closed
wants to merge 4 commits into from

Conversation

mhdawson
Copy link
Member

Checklist
  • tests and code linting passes
  • the commit message follows commit guidelines
Affected core subsystem(s)

test

Description of change

I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case. We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice

@mhdawson mhdawson added the test Issues and PRs related to the tests. label May 11, 2016
@mhdawson mhdawson self-assigned this May 11, 2016
@rvagg
Copy link
Member

rvagg commented May 12, 2016

Concept sgtm but I'm not following that Python at all, what's with the new loop that just appends tests? why aren't the arguments to the ListTests() matching up with the arguments in the function definition? Suspecting I'm not at all qualified to sign off on this cause I don't see how it would work.

@jbergstroem? @bnoordhuis? @chrisdickinson?

@rvagg
Copy link
Member

rvagg commented May 12, 2016

@thefourtheye you're a Python person too right?

@S-YOU
Copy link

S-YOU commented May 12, 2016

just fyi, the new loop same as this code

result += tests * repeat

>>> [] * 10
[]
>>> [1] * 10
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
>>> [1,2,3] * 10
[1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]

for v in VARIANT_FLAGS:
tests = self.GetConfiguration(context).ListTests(current_path, path,
arch, mode)
for t in tests: t.variant_flags = v
result += tests

for rep in range(0, repeat):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 is not needed as it is the default. Moreover xrange is preferred over range in 2.x

@mhdawson
Copy link
Member Author

Updated to address all comments now

@@ -780,7 +779,7 @@ def GetTestStatus(self, context, sections, defs):
class Context(object):

def __init__(self, workspace, buildspace, verbose, vm, args, expect_fail,
timeout, processor, suppress_dialogs, store_unexpected_output):
timeout, processor, suppress_dialogs, store_unexpected_output, repeat):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hate to be that guy but... long line. :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, we can refactor this with dict unpacking.

@bnoordhuis
Copy link
Member

LGTM with style nit.

@thefourtheye
Copy link
Contributor

LGTM

I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

�
@mhdawson
Copy link
Member Author

Updated for last nit

@mhdawson
Copy link
Member Author

@mhdawson
Copy link
Member Author

CI looks green except for pre-existing issue: #6716 and windows build that seem to be stuck on previous build

@mhdawson
Copy link
Member Author

Plan to land tomorrow morning unless there are any objections before then

@@ -1324,6 +1325,9 @@ def BuildOptions():
default="")
result.add_option('--temp-dir',
help='Optional path to change directory used for tests', default=False)
result.add_option('--repeat',
help='how many times to repeat each test',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be written as "Number of times to repeat given tests"

@jbergstroem
Copy link
Member

LGTM with the text update @thefourtheye suggests (or similar).

@joaocgreis
Copy link
Member

Made sure that --run still works, LGTM.

mhdawson added a commit that referenced this pull request May 13, 2016
I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: #6700
Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
@mhdawson
Copy link
Member Author

Ok incorporated updated help message from thefourtheye and landed as ccbb00e

@mhdawson mhdawson closed this May 13, 2016
Fishrock123 pushed a commit that referenced this pull request May 23, 2016
I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: #6700
Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
@MylesBorins
Copy link
Contributor

@mhdawson this will need a manual backport to v4.x

rvagg pushed a commit that referenced this pull request Jun 2, 2016
I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: #6700
Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
@MylesBorins
Copy link
Contributor

ping @mhdawson

@mhdawson
Copy link
Member Author

Still on my list, will try to do it this week.

mhdawson added a commit to mhdawson/io.js that referenced this pull request Jul 14, 2016
Manual backport from master to 4.x stream, original
commit message follows.

I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: nodejs#6700
Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
@mhdawson
Copy link
Member Author

Ok PR against v4.x-staging to backport #7740

FYI @gibfahn, @thealphanerd

MylesBorins pushed a commit that referenced this pull request Jul 27, 2016
Manual backport from master to 4.x stream, original
commit message follows.

I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: #6700
Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
MylesBorins pushed a commit that referenced this pull request Aug 16, 2016
Manual backport from master to 4.x stream, original
commit message follows.

I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: #6700
Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
MylesBorins pushed a commit that referenced this pull request Sep 28, 2016
Manual backport from master to 4.x stream, original
commit message follows.

I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: #6700
Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
rvagg pushed a commit that referenced this pull request Oct 18, 2016
Manual backport from master to 4.x stream, original
commit message follows.

I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: #6700
Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
MylesBorins pushed a commit that referenced this pull request Oct 26, 2016
Manual backport from master to 4.x stream, original
commit message follows.

I often want to run a test many times to see if a failure
can be recreated and I believe this is a common
use case.  We even have this job in the CI
https://ci.nodejs.org/job/node-stress-single-test/configure
but often you want to run it on a specific machine.

This patch adds the --repeat option so that
you can repeat the selected set of tests a
number of times. Given existing options
in test.py this will allow you to run
one or more tests for the number of
repeats specified. For example:

tools/test.py -j8 --repeat 1000 parallel/test-process-exec-argv

runs the test-process-exec-argv test 1000 times,
running 8 copies in parallel

tools/test.py --repeat 2

would run the entire test suite twice.

PR-URL: #6700
Reviewed-By: Ben Noorhduis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: thefourtheye - Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: joaocgreis - João Reis <reis@janeasystems.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
@MylesBorins MylesBorins mentioned this pull request Oct 26, 2016
@mhdawson mhdawson deleted the stress branch March 15, 2017 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants