From ded5d205f82776cc5c2a8b47ac71d33c18c2800a Mon Sep 17 00:00:00 2001 From: Joyee Cheung Date: Tue, 7 Feb 2017 23:56:10 +0800 Subject: [PATCH 1/7] doc: move benchmark/README.md to doc/guides --- .../guides}/doc_img/compare-boxplot.png | Bin {benchmark => doc/guides}/doc_img/scatter-plot.png | Bin .../guides/writing-and-running-benchmarks.md | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename {benchmark => doc/guides}/doc_img/compare-boxplot.png (100%) rename {benchmark => doc/guides}/doc_img/scatter-plot.png (100%) rename benchmark/README.md => doc/guides/writing-and-running-benchmarks.md (100%) diff --git a/benchmark/doc_img/compare-boxplot.png b/doc/guides/doc_img/compare-boxplot.png similarity index 100% rename from benchmark/doc_img/compare-boxplot.png rename to doc/guides/doc_img/compare-boxplot.png diff --git a/benchmark/doc_img/scatter-plot.png b/doc/guides/doc_img/scatter-plot.png similarity index 100% rename from benchmark/doc_img/scatter-plot.png rename to doc/guides/doc_img/scatter-plot.png diff --git a/benchmark/README.md b/doc/guides/writing-and-running-benchmarks.md similarity index 100% rename from benchmark/README.md rename to doc/guides/writing-and-running-benchmarks.md From 9ae1f0d2a44b79315ed0b79f1e761ccd2798d9cb Mon Sep 17 00:00:00 2001 From: Joyee Cheung Date: Wed, 8 Feb 2017 01:10:09 +0800 Subject: [PATCH 2/7] doc: add benchmark/README.md and fix guide --- benchmark/README.md | 243 +++++++++++++++++++ doc/guides/writing-and-running-benchmarks.md | 6 +- 2 files changed, 245 insertions(+), 4 deletions(-) create mode 100644 benchmark/README.md diff --git a/benchmark/README.md b/benchmark/README.md new file mode 100644 index 00000000000000..15b9883419641e --- /dev/null +++ b/benchmark/README.md @@ -0,0 +1,243 @@ +# Node.js Core Benchmarks + +This folder contains code and data used to measure performance +of different Node.js implementations and different ways of +writing JavaScript run by the built-in JavaScript engine. + +For a detailed guide on how to write and run benchmarks in this +directory, see [the guide on benchmarks](../doc/guides/writing-and-running-benchmarks.md). + +## Table of Contents + +* [Benchmark directories](#benchmark-directories) +* [Common API](#common-api) + +## Benchmark Directories + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DirectoryPurpose
arrays + Benchmarks for various operations on array-like objects, + including Array, Buffer, and typed arrays. +
assert + Benchmarks for the assert subsystem. +
buffers + Benchmarks for the buffer subsystem. +
child_process + Benchmarks for the child_process subsystem. +
crypto + Benchmarks for the crypto subsystem. +
dgram + Benchmarks for the dgram subsystem. +
domain + Benchmarks for the domain subsystem. +
es + Benchmarks for various new ECMAScript features and their + pre-ES2015 counterparts. +
events + Benchmarks for the events subsystem. +
fixtures + Benchmarks fixtures used in various benchmarks throughout + the benchmark suite. +
fs + Benchmarks for the fs subsystem. +
http + Benchmarks for the http subsystem. +
misc + Miscellaneous benchmarks and benchmarks for shared + internal modules. +
module + Benchmarks for the module subsystem. +
net + Benchmarks for the net subsystem. +
path + Benchmarks for the path subsystem. +
process + Benchmarks for the process subsystem. +
querystring + Benchmarks for the querystring subsystem. +
streams + Benchmarks for the streams subsystem. +
string_decoder + Benchmarks for the string_decoder subsystem. +
timers + Benchmarks for the timers subsystem, including + setTimeout, setInterval, .etc. +
tls + Benchmarks for the tls subsystem. +
url + Benchmarks for the url subsystem, including the legacy + url implementation and the WHATWG URL implementation. +
util + Benchmarks for the util subsystem. +
vm + Benchmarks for the vm subsystem. +
+ +### Other Top-level files + +The top-level files include common dependencies of the benchmarks +and the tools for launching benchmarks and visualizing their output. +The actual benchmark scripts should be placed in their corresponding +directories. + +* `_benchmark_progress.js`: implements the progress bar displayed + when running `compare.js` +* `_cli.js`: parses the command line arguments passed to `compare.js`, + `run.js` and `scatter.js` +* `_cli.R`: parses the command line arguments passed to `compare.R` +* `_http-benchmarkers.js`: selects and runs external tools for benchmarking + the `http` subsystem. +* `common.js`: see [Common API](#common-api). +* `compare.js`: command line tool for comparing performance between different + Node.js binaries. +* `compare.R`: R script for statistically analyzing the output of + `compare.js` +* `run.js`: command line tool for running individual benchmark suite(s). +* `scatter.js`: command line tool for comparing the performance + between different parameters in benchmark configurations, + for example to analyze the time complexity. +* `scatter.R`: R script for visualizing the output of `scatter.js` with + scatter plots. + +## Common API + +The common.js module is used by benchmarks for consistency across repeated +tasks. It has a number of helpful functions and properties to help with +writing benchmarks. + +### createBenchmark(fn, configs[, options]) + +See [the guide on writing benchmarks](../doc/guides/writing-and-running-benchmarks.md#creating-a-benchmark). + +### default\_http\_benchmarker + +The default benchmarker used to run HTTP benchmarks. + +### PORT + +The default port used to run HTTP benchmarks. + +### sendResult(data) + +Used in special benchmarks that can't use `createBenchmark` and the object +it returns to accomplish what they need. This function reports timing +data to the parent process (usually created by running `compare.js`, `run.js` or +`scatter.js`). + +### v8ForceOptimization(method) + +Force V8 to optimize the function with the native function +`%OptimizeFunctionOnNextCall()` and return the optimization status +after that. + +It is used to prevent the benchmark from getting disrupted +by the optimizer kicking in halfway through. However, this could result in a +less effective optimization, so it should be used with caution. diff --git a/doc/guides/writing-and-running-benchmarks.md b/doc/guides/writing-and-running-benchmarks.md index d1233470757f20..50072f273de112 100644 --- a/doc/guides/writing-and-running-benchmarks.md +++ b/doc/guides/writing-and-running-benchmarks.md @@ -1,8 +1,6 @@ -# Node.js core benchmark +# How to Write and Run Benchmarks in Node.js Core -This folder contains benchmarks to measure the performance of the Node.js APIs. - -## Table of Content +## Table of Contents * [Prerequisites](#prerequisites) * [Running benchmarks](#running-benchmarks) From 09f96f7572f0a7b2a374a2a0c816fdea7dbbb5fd Mon Sep 17 00:00:00 2001 From: Joyee Cheung Date: Wed, 8 Feb 2017 01:15:03 +0800 Subject: [PATCH 3/7] doc: add descriptions and links to test/README.md --- test/README.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/test/README.md b/test/README.md index 5ed028a19631d6..13b3d2e7683f57 100644 --- a/test/README.md +++ b/test/README.md @@ -1,4 +1,15 @@ -# Table of Contents +# Node.js Core Tests + +This folder contains code and data used to test the Node.js implementation. + +For a detailed guide on how to write tests in this +directory, see [the guide on writing tests](../doc/guides/writing-tests.md). + +On how to run tests in this direcotry, see +[the contributing guide](../CONTRIBUTING.md#step-5-test). + +## Table of Contents + * [Test directories](#test-directories) * [Common module API](#common-module-api) From fd331c18da2913a6a017b3b99d00542dd8a5a652 Mon Sep 17 00:00:00 2001 From: Joyee Cheung Date: Sun, 12 Feb 2017 16:41:58 +0800 Subject: [PATCH 4/7] fix benchmark guide links --- benchmark/README.md | 5 ++- doc/guides/writing-and-running-benchmarks.md | 39 ++++++++++++-------- 2 files changed, 28 insertions(+), 16 deletions(-) diff --git a/benchmark/README.md b/benchmark/README.md index 15b9883419641e..9da8adee5ad42e 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -215,15 +215,18 @@ writing benchmarks. ### createBenchmark(fn, configs[, options]) -See [the guide on writing benchmarks](../doc/guides/writing-and-running-benchmarks.md#creating-a-benchmark). +See [the guide on writing benchmarks](../doc/guides/writing-and-running-benchmarks.md#basics-of-a-benchmark). ### default\_http\_benchmarker The default benchmarker used to run HTTP benchmarks. +See [the guide on writing HTTP benchmarks](../doc/guides/writing-and-running-benchmarks.md#creating-an-http-benchmark). + ### PORT The default port used to run HTTP benchmarks. +See [the guide on writing HTTP benchmarks](../doc/guides/writing-and-running-benchmarks.md#creating-an-http-benchmark). ### sendResult(data) diff --git a/doc/guides/writing-and-running-benchmarks.md b/doc/guides/writing-and-running-benchmarks.md index 50072f273de112..cba6a3009116f9 100644 --- a/doc/guides/writing-and-running-benchmarks.md +++ b/doc/guides/writing-and-running-benchmarks.md @@ -3,22 +3,32 @@ ## Table of Contents * [Prerequisites](#prerequisites) + * [HTTP Benchmark Requirements](#http-benchmark-requirements) + * [Benchmark Analysis Requirements](#benchmark-analysis-requirements) * [Running benchmarks](#running-benchmarks) - * [Running individual benchmarks](#running-individual-benchmarks) - * [Running all benchmarks](#running-all-benchmarks) - * [Comparing node versions](#comparing-node-versions) - * [Comparing parameters](#comparing-parameters) + * [Running individual benchmarks](#running-individual-benchmarks) + * [Running all benchmarks](#running-all-benchmarks) + * [Comparing Node.js versions](#comparing-nodejs-versions) + * [Comparing parameters](#comparing-parameters) * [Creating a benchmark](#creating-a-benchmark) + * [Basics of a benchmark](#basics-of-a-benchmark) + * [Creating an HTTP benchmark](#creating-an-http-benchmark) ## Prerequisites +Basic Unix tools are required for some benchmarks. +[Git for Windows][git-for-windows] includes Git Bash and the necessary tools, +which need to be included in the global Windows `PATH`. + +### HTTP Benchmark Requirements + Most of the HTTP benchmarks require a benchmarker to be installed, this can be either [`wrk`][wrk] or [`autocannon`][autocannon]. -`Autocannon` is a Node script that can be installed using -`npm install -g autocannon`. It will use the Node executable that is in the +`Autocannon` is a Node.js script that can be installed using +`npm install -g autocannon`. It will use the Node.js executable that is in the path, hence if you want to compare two HTTP benchmark runs make sure that the -Node version in the path is not altered. +Node.js version in the path is not altered. `wrk` may be available through your preferred package manager. If not, you can easily build it [from source][wrk] via `make`. @@ -32,9 +42,7 @@ benchmarker to be used by providing it as an argument, e. g.: `node benchmark/http/simple.js benchmarker=autocannon` -Basic Unix tools are required for some benchmarks. -[Git for Windows][git-for-windows] includes Git Bash and the necessary tools, -which need to be included in the global Windows `PATH`. +### Benchmark Analysis Requirements To analyze the results `R` should be installed. Check you package manager or download it from https://www.r-project.org/. @@ -48,7 +56,6 @@ install.packages("ggplot2") install.packages("plyr") ``` -### CRAN Mirror Issues In the event you get a message that you need to select a CRAN mirror first. You can specify a mirror by adding in the repo parameter. @@ -133,9 +140,9 @@ It is possible to execute more groups by adding extra process arguments. $ node benchmark/run.js arrays buffers ``` -### Comparing node versions +### Comparing Node.js versions -To compare the effect of a new node version use the `compare.js` tool. This +To compare the effect of a new Node.js version use the `compare.js` tool. This will run each benchmark multiple times, making it possible to calculate statistics on the performance measures. @@ -144,7 +151,7 @@ As an example on how to check for a possible performance improvement, the an example. This pull request _claims_ to improve the performance of the `string_decoder` module. -First build two versions of node, one from the master branch (here called +First build two versions of Node.js, one from the master branch (here called `./node-master`) and another with the pull request applied (here called `./node-pr-5135`). @@ -284,6 +291,8 @@ chunk encoding mean confidence.interval ## Creating a benchmark +### Basics of a benchmark + All benchmarks use the `require('../common.js')` module. This contains the `createBenchmark(main, configs[, options])` method which will setup your benchmark. @@ -367,7 +376,7 @@ function main(conf) { } ``` -## Creating HTTP benchmark +### Creating an HTTP benchmark The `bench` object returned by `createBenchmark` implements `http(options, callback)` method. It can be used to run external tool to From 409e2fc7dc5eededbef3bd1106dc8c78ccc6211e Mon Sep 17 00:00:00 2001 From: Joyee Cheung Date: Sun, 12 Feb 2017 17:05:13 +0800 Subject: [PATCH 5/7] use tables for test/README.md --- test/README.md | 272 ++++++++++++++++++++++++------------------------- 1 file changed, 134 insertions(+), 138 deletions(-) diff --git a/test/README.md b/test/README.md index 13b3d2e7683f57..8635dea3140e31 100644 --- a/test/README.md +++ b/test/README.md @@ -15,144 +15,140 @@ On how to run tests in this direcotry, see ## Test Directories -### abort - -Tests for when the `--abort-on-uncaught-exception` flag is used. - -| Runs on CI | -|:----------:| -| No | - -### addons - -Tests for [addon](https://nodejs.org/api/addons.html) functionality along with -some tests that require an addon to function properly. - - -| Runs on CI | -|:----------:| -| Yes | - -### cctest - -C++ test that is run as part of the build process. - -| Runs on CI | -|:----------:| -| Yes | - -### debugger - -Tests for [debugger](https://nodejs.org/api/debugger.html) functionality. - -| Runs on CI | -|:----------:| -| No | - -### disabled - -Tests that have been disabled from running for various reasons. - -| Runs on CI | -|:----------:| -| No | - -### fixtures - -Test fixtures used in various tests throughout the test suite. - -### gc - -Tests for garbage collection related functionality. - -| Runs on CI | -|:----------:| -| No | - - -### inspector - -Tests for the V8 inspector integration. - -| Runs on CI | -|:----------:| -| Yes | - -### internet - -Tests that make real outbound connections (mainly networking related modules). -Tests for networking related modules may also be present in other directories, -but those tests do not make outbound connections. - -| Runs on CI | -|:----------:| -| No | - -### known_issues - -Tests reproducing known issues within the system. - -| Runs on CI | -|:----------:| -| No | - -### message - -Tests for messages that are output for various conditions (`console.log`, -error messages etc.) - -| Runs on CI | -|:----------:| -| Yes | - -### parallel - -Various tests that are able to be run in parallel. - -| Runs on CI | -|:----------:| -| Yes | - -### pummel - -Various tests for various modules / system functionality operating under load. - -| Runs on CI | -|:----------:| -| No | - -### sequential - -Various tests that are run sequentially. - -| Runs on CI | -|:----------:| -| Yes | - -### testpy - -Test configuration utility used by various test suites. - -### tick-processor - -Tests for the V8 tick processor integration. The tests are for the logic in -`lib/internal/v8_prof_processor.js` and `lib/internal/v8_prof_polyfill.js`. The -tests confirm that the profile processor packages the correct set of scripts -from V8 and introduces the correct platform specific logic. - -| Runs on CI | -|:----------:| -| No | - -### timers - -Tests for [timing utilities](https://nodejs.org/api/timers.html) (`setTimeout` -and `setInterval`). - -| Runs on CI | -|:----------:| -| No | - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DirectoryRuns on CIPurpose
abortNo + Tests for when the --abort-on-uncaught-exception + flag is used. +
addonsYes + Tests for addon + functionality along with some tests that require an addon to function + properly. +
cctestYes + C++ test that is run as part of the build process. +
debuggerNo + Tests for debugger + functionality along with some tests that require an addon to function + properly. +
disabledNo + Tests that have been disabled from running for various reasons. +
fixturesTest fixtures used in various tests throughout the test suite.
gcNoTests for garbage collection related functionality.
inspectorYesTests for the V8 inspector integration.
internetNo + Tests that make real outbound connections (mainly networking related + modules). Tests for networking related modules may also be present in + other directories, but those tests do not make outbound connections. +
known_issuesNoTests reproducing known issues within the system.
messageYes + Tests for messages that are output for various conditions + (console.log, error messages etc.)
parallelYesVarious tests that are able to be run in parallel.
pummelNo + Various tests for various modules / system functionality operating + under load. +
sequentialYes + Various tests that are run sequentially. +
testpy + Test configuration utility used by various test suites. +
tick-processorNo + Tests for the V8 tick processor integration. The tests are for the + logic in lib/internal/v8_prof_processor.js and + lib/internal/v8_prof_polyfill.js. The tests confirm that + the profile processor packages the correct set of scripts from V8 and + introduces the correct platform specific logic. +
timersNo + Tests for + timing utilities + (setTimeout and setInterval). +
## Common module API From b88d113874d70993eb2c0a333041118879e7aa84 Mon Sep 17 00:00:00 2001 From: Joyee Cheung Date: Sun, 12 Feb 2017 17:46:28 +0800 Subject: [PATCH 6/7] update v8ForceOptimization docs --- benchmark/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/benchmark/README.md b/benchmark/README.md index 9da8adee5ad42e..6fd9a97bdfb3bb 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -235,12 +235,12 @@ it returns to accomplish what they need. This function reports timing data to the parent process (usually created by running `compare.js`, `run.js` or `scatter.js`). -### v8ForceOptimization(method) +### v8ForceOptimization(method[, ...args]) -Force V8 to optimize the function with the native function +Force V8 to mark the `method` for optimization with the native function `%OptimizeFunctionOnNextCall()` and return the optimization status after that. -It is used to prevent the benchmark from getting disrupted -by the optimizer kicking in halfway through. However, this could result in a -less effective optimization, so it should be used with caution. +It can be used to prevent the benchmark from getting disrupted by the optimizer +kicking in halfway through. However, this could result in a less effective +optimization. In general, only use it if you know what it actually does. From d5302a1c42903878fd6e508f28351a5a9603582a Mon Sep 17 00:00:00 2001 From: Joyee Cheung Date: Sun, 12 Feb 2017 19:42:03 +0800 Subject: [PATCH 7/7] hint about how to use benchmark tools --- doc/guides/writing-and-running-benchmarks.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/guides/writing-and-running-benchmarks.md b/doc/guides/writing-and-running-benchmarks.md index cba6a3009116f9..a20f321b7c2408 100644 --- a/doc/guides/writing-and-running-benchmarks.md +++ b/doc/guides/writing-and-running-benchmarks.md @@ -113,7 +113,8 @@ buffers/buffer-tostring.js n=10000000 len=1024 arg=false: 3783071.1678948295 ### Running all benchmarks Similar to running individual benchmarks, a group of benchmarks can be executed -by using the `run.js` tool. Again this does not provide the statistical +by using the `run.js` tool. To see how to use this script, +run `node benchmark/run.js`. Again this does not provide the statistical information to make any conclusions. ```console @@ -144,7 +145,8 @@ $ node benchmark/run.js arrays buffers To compare the effect of a new Node.js version use the `compare.js` tool. This will run each benchmark multiple times, making it possible to calculate -statistics on the performance measures. +statistics on the performance measures. To see how to use this script, +run `node benchmark/compare.js`. As an example on how to check for a possible performance improvement, the [#5134](https://github.com/nodejs/node/pull/5134) pull request will be used as @@ -224,7 +226,8 @@ It can be useful to compare the performance for different parameters, for example to analyze the time complexity. To do this use the `scatter.js` tool, this will run a benchmark multiple times -and generate a csv with the results. +and generate a csv with the results. To see how to use this script, +run `node benchmark/scatter.js`. ```console $ node benchmark/scatter.js benchmark/string_decoder/string-decoder.js > scatter.csv