Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: update test workflow and expand test matrix #1086

Merged
merged 1 commit into from
Jun 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 110 additions & 77 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -1,99 +1,132 @@
step-restore-cache: &step-restore-cache
restore_cache:
keys:
- v1-dependencies-{{ arch }}-{{ checksum "yarn.lock" }}
version: 2.1

steps-test: &steps-test
steps:
- checkout
- *step-restore-cache
- run:
name: Install Node
command: |
case "$(uname -s)" in
Darwin)
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm install 12
nvm alias default 12
echo 'export NVM_DIR=${HOME}/.nvm' >> $BASH_ENV
echo "[ -s '${NVM_DIR}/nvm.sh' ] && . '${NVM_DIR}/nvm.sh'" >> $BASH_ENV
;;
Windows*|CYGWIN*|MINGW*|MSYS*)
nvm install 12.22.4
nvm use 12.22.4
;;
esac
- run: yarn install --frozen-lockfile --ignore-engines
- save_cache:
paths:
- node_modules
key: v1-dependencies-{{ arch }}-{{ checksum "yarn.lock" }}
- run: yarn run lint
- run:
name: Tests with code coverage
command: yarn run coverage
environment:
DEBUG: electron-rebuild
- run: yarn run codecov
orbs:
cfa: continuousauth/npm@1.0.2
node: circleci/node@5.1.0
win: circleci/windows@5.0.0

commands:
install:
parameters:
node-version:
description: Node.js version to install
type: string
steps:
- run: git config --global core.autocrlf input
- node/install:
node-version: << parameters.node-version >>
- run: nvm use << parameters.node-version >>
# Can't get yarn installed on Windows with the circleci/node orb, so use npx yarn for commands
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ arch }}-{{ checksum "yarn.lock" }}
- v1-dependencies-{{ arch }}
- run: npx yarn install --frozen-lockfile --ignore-engines
- save_cache:
paths:
- node_modules
key: v1-dependencies-{{ arch }}-{{ checksum "yarn.lock" }}
test:
steps:
- run: npx yarn run lint
- run:
name: Tests with code coverage
command: npx yarn run coverage
environment:
DEBUG: electron-rebuild
- run: npx yarn run codecov

version: 2.1
orbs:
win: circleci/windows@1.0.0
jobs:
test-linux-12:
test-linux:
docker:
- image: cimg/node:12.22
<<: *steps-test
test-linux-14:
docker:
- image: cimg/node:14.20
<<: *steps-test
test-linux-16:
docker:
- image: cimg/node:16.18
<<: *steps-test
- image: cimg/base:stable
parameters:
node-version:
description: Node.js version to install
type: string
steps:
- install:
node-version: << parameters.node-version >>
- test
test-mac:
macos:
xcode: "13.3.0"
<<: *steps-test
resource_class: macos.x86.medium.gen2
parameters:
node-version:
description: Node.js version to install
type: string
steps:
- install:
node-version: << parameters.node-version >>
- test
test-windows:
executor:
name: win/vs2019
name: win/server-2022
shell: bash.exe
environment:
GYP_MSVS_VERSION: '2019'
<<: *steps-test

release:
docker:
- image: cimg/node:14.17
GYP_MSVS_VERSION: '2022'
parameters:
node-version:
description: Node.js version to install
type: string
steps:
- checkout
- *step-restore-cache
- run: yarn install --frozen-lockfile --ignore-engines
- run: npx @continuous-auth/circleci-oidc-github-auth@1.0.4
- run: npx semantic-release@17.4.5
- run:
command: New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force
shell: powershell.exe
- install:
node-version: << parameters.node-version >>
- test

workflows:
version: 2
test_and_release:
# Run the test jobs first, then the release only when all the test jobs are successful
jobs:
- test-linux-16
- test-linux-12
- test-linux-14
- test-mac
- test-windows
- release:
- test-linux:
matrix:
parameters:
node-version:
- 20.2.0
- 18.16.0
- 16.20.0
- 14.21.3
- 12.22.12
Comment on lines +93 to +94
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why continue to support and encourage the use of end-of-life versions of Node.js? If users want support then they can try to reproduce their problems on a supported version. If folks are paying for long-term support for an OS that has an EOL version of Node.js then their OS provider can support them. If folks are using an OS that is beyond the standard support and are not paying for LTS, then their problems should be demonstrated on supported versions. Building on secure platforms and being able to use new language optimizations and features helps everyone's safety and velocity so let's encourage security and performance upgrades.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This package has an engine version that supports Node.js v12 and above, so dropping support would be a breaking change (and require a major version bump on this package). Any major version bump causes fragmentation with the user base, as can be seen in the npm downloads stats for older versions of this package (it used to be called just electron-rebuild) where there are still ~6k downloads/week on the final 2.y.z version and ~7.5k downloads/week on the final 1.y.z version. We want to avoid furthering that fragmentation, so we don't want to bump majors more often than we need to (which every Node.js EOL would require). These packages also don't have a ton of maintainer time to put into them, so it takes some time for things to be merged - if we bump the major version we might make the safety issue worse by putting new fixes behind a new major version where they won't be picked up by some subset of the user base. For ease of maintenance these packages aren't setup to do additional releases to old majors, so once we bump majors there are no more fixes being backported to old majors (and we would need the CI to be testing the old Node.js versions those majors support to be able to do so).

So the general policy of @~electron/wg-ecosystem is to continue to test and not explicitly drop older Node.js versions unless necessary. We're not encouraging the usage of EOL versions of Node.js, more avoiding breaking them when feasible. In most cases we won't go out of our way to fix bugs which only affect those EOL versions, and if a future change requires dropping support for older EOL versions of Node.js we will explicitly drop support then, if it's deemed necessary.

Hope this makes sense. 🙂

Copy link
Contributor

@cclauss cclauss Jun 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Users of legacy software create a lot of noise in support channels...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will that noise really go away if this package stops testing those EOL Node.js versions, though? Even if we drop support and bump the major, some users will stick on the current major (as seen in the npm download stats listed above).

If you tweak the GitHub search to include created:>2022-01-01, it only returns 16 issues in the past 18 months, and without going beyond flipping through the descriptions shown on the search page, at least 3 of them were reporting supported Node.js versions.

It's also worth noting that node-gyp itself is still showing Node.js v12 and v14 as supported engines according to package.json: https://github.com/nodejs/node-gyp/blob/33391db3a0008eff8408890da6ab232f2f90fcab/package.json#L37-L39

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will that noise really go away if this package stops testing those EOL Node.js versions?

Rephrase: Will that noise decrease if this package stops encouraging / supporting EOL Node.js versions?

Yes, it will. Developers are improving software and ignoring their efforts is suboptimal on many different levels. Solutions that used to work in older versions of npm, node, and gyp no longer work today. Supporting both legacy and current is complex, frustrating, and error-prone. Doing so for the sake of statistics is a shame.

- test-mac:
matrix:
parameters:
node-version:
- 20.2.0
- 18.16.0
- 16.20.0
- 14.21.3
- 12.22.12
- test-windows:
matrix:
parameters:
node-version:
- 20.2.0
- 18.16.0
- 16.20.0
- 14.21.3
- 12.22.12
- cfa/release:
requires:
- test-linux-16
- test-linux-12
- test-linux-14
- test-mac
- test-windows
- test-linux-20.2.0
- test-linux-18.16.0
- test-linux-16.20.0
- test-linux-14.21.3
- test-linux-12.22.12
- test-mac-20.2.0
- test-mac-18.16.0
- test-mac-16.20.0
- test-mac-14.21.3
- test-mac-12.22.12
- test-windows-20.2.0
- test-windows-18.16.0
- test-windows-16.20.0
- test-windows-14.21.3
- test-windows-12.22.12
filters:
branches:
only:
Expand Down
2 changes: 1 addition & 1 deletion test/rebuild-napibuildversion.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ describe('rebuild with napi_build_versions in binary config', async function ()
before(async () => {
await resetTestModule(testModulePath, true, 'napi-build-version')
// Forcing `msvs_version` needed in order for `arm64` `win32` binary to be built
process.env.GYP_MSVS_VERSION = "2019"
process.env.GYP_MSVS_VERSION = process.env.GYP_MSVS_VERSION ?? "2019"
});
after(() => cleanupTestModule(testModulePath));

Expand Down