Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tools: improve node:test output when running in github actions #49129

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ test/fixtures
test/message/esm_display_syntax_error.mjs
tools/icu
tools/lint-md/lint-md.mjs
tools/github_reporter
benchmark/tmp
benchmark/fixtures
doc/**/*.js
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build-tarball.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,4 @@ jobs:
- name: Test
run: |
cd $TAR_DIR
make run-ci -j2 V=1 TEST_CI_ARGS="-p dots --measure-flakiness 9"
make run-ci -j2 V=1 TEST_CI_ARGS="-p dots --node-args='--test-reporter=spec' --measure-flakiness 9"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an improvement in terms of the amount of output we see from failed tests, which should help with debugging.

There is also the issue of the asan CI job truncating output. Since node:test is never used as the main test runner, a possible next step to address that issue could be to create a custom reporter in tools/ that only outputs the test failures and diagnostic messages. It could basically be a slimmed down version of the spec reporter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can do that in a follow-up. perhaps only in the asan job

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the CI ran fine so I guess there is no issue, but are we sure this won't impact any of the tests that expect different reporters?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no issue. tests that look at reporters (e.g snapshot tests, exit code tests etc) always spawn a child and control the env vars and flags

2 changes: 1 addition & 1 deletion .github/workflows/coverage-linux-without-intl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
# TODO(bcoe): fix the couple tests that fail with the inspector enabled.
# The cause is most likely coverage's use of the inspector.
- name: Test
run: NODE_V8_COVERAGE=coverage/tmp make test-cov -j2 V=1 TEST_CI_ARGS="-p dots --measure-flakiness 9" || exit 0
run: NODE_V8_COVERAGE=coverage/tmp make test-cov -j2 V=1 TEST_CI_ARGS="-p dots --node-args='--test-reporter=spec' --measure-flakiness 9" || exit 0
- name: Report JS
run: npx c8 report --check-coverage
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/coverage-linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
# TODO(bcoe): fix the couple tests that fail with the inspector enabled.
# The cause is most likely coverage's use of the inspector.
- name: Test
run: NODE_V8_COVERAGE=coverage/tmp make test-cov -j2 V=1 TEST_CI_ARGS="-p dots --measure-flakiness 9" || exit 0
run: NODE_V8_COVERAGE=coverage/tmp make test-cov -j2 V=1 TEST_CI_ARGS="-p dots --node-args='--test-reporter=spec' --measure-flakiness 9" || exit 0
- name: Report JS
run: npx c8 report --check-coverage
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/doc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,4 @@ jobs:
name: docs
path: out/doc
- name: Test
run: NODE=$(command -v node) make test-doc-ci TEST_CI_ARGS="-p actions --measure-flakiness 9"
run: NODE=$(command -v node) make test-doc-ci TEST_CI_ARGS="-p actions --node-args='--test-reporter=spec' --node-args='--test-reporter-destination=stdout' --measure-flakiness 9"
2 changes: 1 addition & 1 deletion .github/workflows/test-asan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,4 @@ jobs:
- name: Build
run: make build-ci -j2 V=1
- name: Test
run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions -t 300 --measure-flakiness 9"
run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions --node-args='--test-reporter=spec' --node-args='--test-reporter-destination=stdout' -t 300 --measure-flakiness 9"
2 changes: 1 addition & 1 deletion .github/workflows/test-linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,4 @@ jobs:
- name: Build
run: make build-ci -j2 V=1 CONFIG_FLAGS="--error-on-warn"
- name: Test
run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions --measure-flakiness 9"
run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions --node-args='--test-reporter=spec' --node-args='--test-reporter-destination=stdout' --measure-flakiness 9"
2 changes: 1 addition & 1 deletion .github/workflows/test-macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,4 +60,4 @@ jobs:
- name: Build
run: make build-ci -j$(getconf _NPROCESSORS_ONLN) V=1 CONFIG_FLAGS="--error-on-warn"
- name: Test
run: make run-ci -j$(getconf _NPROCESSORS_ONLN) V=1 TEST_CI_ARGS="-p actions --measure-flakiness 9"
run: make run-ci -j$(getconf _NPROCESSORS_ONLN) V=1 TEST_CI_ARGS="-p actions --node-args='--test-reporter=spec' --node-args='--test-reporter-destination=stdout' --measure-flakiness 9"
8 changes: 8 additions & 0 deletions .github/workflows/tools.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,14 @@ jobs:
cat temp-output
tail -n1 temp-output | grep "NEW_VERSION=" >> "$GITHUB_ENV" || true
rm temp-output
- id: github_reporter
subsystem: tools
label: tools
run: |
./tools/dep_updaters/update-github-reporter.sh > temp-output
cat temp-output
tail -n1 temp-output | grep "NEW_VERSION=" >> "$GITHUB_ENV" || true
rm temp-output
- id: googletest
subsystem: deps
label: dependencies, test
Expand Down
44 changes: 44 additions & 0 deletions tools/dep_updaters/update-github-reporter.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#!/bin/sh

# Shell script to update @reporters/github in the source tree to the latest release.

# This script must be in the tools directory when it runs because it uses the
# script source file path to determine directories to work in.

set -ex

ROOT=$(cd "$(dirname "$0")/../.." && pwd)
[ -z "$NODE" ] && NODE="$ROOT/out/Release/node"
[ -x "$NODE" ] || NODE=$(command -v node)
NPM="$ROOT/deps/npm/bin/npm-cli.js"

# shellcheck disable=SC1091
. "$ROOT/tools/dep_updaters/utils.sh"

NEW_VERSION=$("$NODE" "$NPM" view @reporters/github dist-tags.latest)
CURRENT_VERSION=$("$NODE" -p "require('./tools/github_reporter/package.json').version")

# This function exit with 0 if new version and current version are the same
compare_dependency_version "@reporters/github" "$NEW_VERSION" "$CURRENT_VERSION"

cd "$( dirname "$0" )/../.." || exit
rm -rf tools/github_reporter

(
rm -rf github-reporter-tmp
mkdir github-reporter-tmp
cd github-reporter-tmp || exit

"$NODE" "$NPM" init --yes
"$NODE" "$NPM" install --global-style --no-bin-links --ignore-scripts "@reporters/github@$NEW_VERSION"
"$NODE" "$NPM" exec --package=esbuild@0.17.15 --yes -- esbuild ./node_modules/@reporters/github --bundle --platform=node --outfile=bundle.js
)

mkdir tools/github_reporter
mv github-reporter-tmp/bundle.js tools/github_reporter/index.js
mv github-reporter-tmp/node_modules/@reporters/github/package.json tools/github_reporter/package.json
rm -rf github-reporter-tmp/

# The last line of the script should always print the new version,
# as we need to add it to $GITHUB_ENV variable.
echo "NEW_VERSION=$NEW_VERSION"
Loading