Skip to content

Commit

Permalink
test: move benchmark tests out of main test suite
Browse files Browse the repository at this point in the history
Move benchmark tests (which are slow) out of the main test suite. We can
hopefully add them to node-daily-master so that they are still run daily
on CI.

Backport-PR-URL: #25049
PR-URL: #24265
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
  • Loading branch information
Trott authored and MylesBorins committed Dec 26, 2018
1 parent 17b55bf commit d04c3c2
Show file tree
Hide file tree
Showing 38 changed files with 38 additions and 33 deletions.
11 changes: 7 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ v8:
tools/make-v8.sh $(V8_ARCH).$(BUILDTYPE_LOWER) $(V8_BUILD_OPTIONS)

.PHONY: jstest
jstest: build-addons build-addons-napi bench-addons-build ## Runs addon tests and JS tests
jstest: build-addons build-addons-napi ## Runs addon tests and JS tests
$(PYTHON) tools/test.py $(PARALLEL_ARGS) --mode=$(BUILDTYPE_LOWER) \
--skip-tests=$(CI_SKIP_TESTS) \
$(CI_JS_SUITES) \
Expand Down Expand Up @@ -414,7 +414,7 @@ clear-stalled:
echo $${PS_OUT} | xargs kill -9; \
fi

test-build: | all build-addons build-addons-napi bench-addons-build
test-build: | all build-addons build-addons-napi

test-build-addons-napi: all build-addons-napi

Expand Down Expand Up @@ -455,7 +455,7 @@ test-ci-js: | clear-stalled
.PHONY: test-ci
# Related CI jobs: most CI tests, excluding node-test-commit-arm-fanned
test-ci: LOGLEVEL := info
test-ci: | clear-stalled build-addons build-addons-napi doc-only bench-addons-build
test-ci: | clear-stalled build-addons build-addons-napi doc-only
out/Release/cctest --gtest_output=tap:cctest.tap
$(PYTHON) tools/test.py $(PARALLEL_ARGS) -p tap --logfile test.tap \
--mode=$(BUILDTYPE_LOWER) --flaky-tests=$(FLAKY_TESTS) \
Expand Down Expand Up @@ -496,7 +496,7 @@ test-debug: test-build
test-message: test-build
$(PYTHON) tools/test.py $(PARALLEL_ARGS) message

test-simple: | cctest bench-addons-build # Depends on 'all'.
test-simple: | cctest # Depends on 'all'.
$(PYTHON) tools/test.py $(PARALLEL_ARGS) parallel sequential

test-pummel: all
Expand All @@ -509,6 +509,9 @@ test-node-inspect: $(NODE_EXE)
USE_EMBEDDED_NODE_INSPECT=1 $(NODE) tools/test-npm-package \
--install deps/node-inspect test

test-benchmark: | bench-addons-build
$(PYTHON) tools/test.py $(PARALLEL_ARGS) benchmark

test-tick-processor: all
$(PYTHON) tools/test.py $(PARALLEL_ARGS) tick-processor

Expand Down
1 change: 1 addition & 0 deletions test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ GitHub with the `autocrlf` git config flag set to true.
| `addons` | Yes | Tests for [addon](https://nodejs.org/api/addons.html) functionality along with some tests that require an addon to function properly. |
| `addons-napi` | Yes | Tests for [n-api](https://nodejs.org/api/n-api.html) functionality. |
| `async-hooks` | Yes | Tests for [async_hooks](https://nodejs.org/api/async_hooks.html) functionality. |
| `benchmark` | No | Test minimal functionality of benchmarks. |
| `cctest` | Yes | C++ tests that are run as part of the build process. |
| `code-cache` | No | Tests for a Node.js binary compiled with V8 code cache. |
| `common` | | Common modules shared among many tests. [Documentation](./common/README.md) |
Expand Down
21 changes: 21 additions & 0 deletions test/benchmark/benchmark.status
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
prefix benchmark

# To mark a test as flaky, list the test name in the appropriate section
# below, without ".js", followed by ": PASS,FLAKY". Example:
# sample-test : PASS,FLAKY

[true] # This section applies to all platforms

[$system==win32]

[$system==linux]

[$system==macos]

[$system==solaris] # Also applies to SmartOS

[$system==freebsd]

[$system==aix]

[$arch==arm]
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
6 changes: 6 additions & 0 deletions test/benchmark/testcfg.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import sys, os
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
import testpy

def GetConfiguration(context, root):
return testpy.SimpleTestConfiguration(context, root, 'benchmark')
28 changes: 0 additions & 28 deletions test/root.status
Original file line number Diff line number Diff line change
Expand Up @@ -9,27 +9,6 @@ async-hooks/test-tlswrap: SLOW
async-hooks/test-tlswrap: SLOW
message/eval_messages: SLOW
message/stdin_messages: SLOW
parallel/test-benchmark-assert: SLOW
parallel/test-benchmark-cluster: SLOW
parallel/test-benchmark-crypto: SLOW
parallel/test-benchmark-dns: SLOW
parallel/test-benchmark-domain: SLOW
parallel/test-benchmark-es: SLOW
parallel/test-benchmark-events: SLOW
parallel/test-benchmark-fs: SLOW
parallel/test-benchmark-misc: SLOW
parallel/test-benchmark-module: SLOW
parallel/test-benchmark-os: SLOW
parallel/test-benchmark-process: SLOW
parallel/test-benchmark-querystring: SLOW
parallel/test-benchmark-streams: SLOW
parallel/test-benchmark-string_decoder: SLOW
parallel/test-benchmark-timers: SLOW
parallel/test-benchmark-url: SLOW
parallel/test-benchmark-util: SLOW
parallel/test-benchmark-v8: SLOW
parallel/test-benchmark-vm: SLOW
parallel/test-benchmark-zlib: SLOW
parallel/test-buffer-constructor-node-modules-paths: SLOW
parallel/test-buffer-indexof: SLOW
parallel/test-child-process-spawnsync-input: SLOW
Expand Down Expand Up @@ -165,13 +144,6 @@ parallel/test-worker-unsupported-things: SLOW
parallel/test-worker-workerdata-sharedarraybuffer: SLOW
parallel/test-zlib-bytes-read: SLOW
parallel/test-zlib-convenience-methods: SLOW
sequential/test-benchmark-buffer: SLOW
sequential/test-benchmark-child-process: SLOW
sequential/test-benchmark-dgram: SLOW
sequential/test-benchmark-http: SLOW
sequential/test-benchmark-net: SLOW
sequential/test-benchmark-path: SLOW
sequential/test-benchmark-tls: SLOW
sequential/test-child-process-execsync: SLOW
sequential/test-child-process-exit: SLOW
sequential/test-child-process-pass-fd: SLOW
Expand Down
1 change: 1 addition & 0 deletions tools/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -1498,6 +1498,7 @@ def PrintCrashed(code):
IGNORED_SUITES = [
'addons',
'addons-napi',
'benchmark',
'doctool',
'internet',
'pummel',
Expand Down
3 changes: 2 additions & 1 deletion vcbuild.bat
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ if /i "%1"=="build-addons" set build_addons=1&goto arg-ok
if /i "%1"=="build-addons-napi" set build_addons_napi=1&goto arg-ok
if /i "%1"=="test-addons" set test_args=%test_args% addons&set build_addons=1&goto arg-ok
if /i "%1"=="test-addons-napi" set test_args=%test_args% addons-napi&set build_addons_napi=1&goto arg-ok
if /i "%1"=="test-benchmark" set test_args=%test_args% benchmark&goto arg-ok
if /i "%1"=="test-simple" set test_args=%test_args% sequential parallel -J&goto arg-ok
if /i "%1"=="test-message" set test_args=%test_args% message&goto arg-ok
if /i "%1"=="test-tick-processor" set test_args=%test_args% tick-processor&goto arg-ok
Expand Down Expand Up @@ -632,7 +633,7 @@ del .used_configure_flags
goto exit

:help
echo vcbuild.bat [debug/release] [msi] [doc] [test/test-ci/test-all/test-addons/test-addons-napi/test-internet/test-pummel/test-simple/test-message/test-tick-processor/test-known-issues/test-node-inspect/test-check-deopts/test-npm/test-async-hooks/test-v8/test-v8-intl/test-v8-benchmarks/test-v8-all] [ignore-flaky] [static/dll] [noprojgen] [projgen] [small-icu/full-icu/without-intl] [nobuild] [nosnapshot] [noetw] [noperfctr] [ltcg] [nopch] [licensetf] [sign] [ia32/x86/x64] [vs2017] [download-all] [enable-vtune] [lint/lint-ci/lint-js/lint-js-ci/lint-md] [lint-md-build] [package] [build-release] [upload] [no-NODE-OPTIONS] [link-module path-to-module] [debug-http2] [debug-nghttp2] [clean] [no-cctest] [openssl-no-asm]
echo vcbuild.bat [debug/release] [msi] [doc] [test/test-ci/test-all/test-addons/test-addons-napi/test-benchmark/test-internet/test-pummel/test-simple/test-message/test-tick-processor/test-known-issues/test-node-inspect/test-check-deopts/test-npm/test-async-hooks/test-v8/test-v8-intl/test-v8-benchmarks/test-v8-all] [ignore-flaky] [static/dll] [noprojgen] [projgen] [small-icu/full-icu/without-intl] [nobuild] [nosnapshot] [noetw] [noperfctr] [ltcg] [nopch] [licensetf] [sign] [ia32/x86/x64] [vs2017] [download-all] [enable-vtune] [lint/lint-ci/lint-js/lint-js-ci/lint-md] [lint-md-build] [package] [build-release] [upload] [no-NODE-OPTIONS] [link-module path-to-module] [debug-http2] [debug-nghttp2] [clean] [no-cctest] [openssl-no-asm]
echo Examples:
echo vcbuild.bat : builds release build
echo vcbuild.bat debug : builds debug build
Expand Down

0 comments on commit d04c3c2

Please sign in to comment.