OSX tests are stuck #3887
Labels
Comments
|
Alpine might be facing a similar problem. It's stuck on CITGM: https://ci.nodejs.org/job/citgm-smoker/3472/ |
It's not stuck -- https://ci.nodejs.org/job/node-test-commit-osx/60982/ hasn't started yet (it's been waiting for a osx11-x64 machine for 12 hours). https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx11-x64/ is incredibly backlogged -- looks like we only have one machine serving jobs. |
|
Running the cleanup script on test-orka-macos11-x64-2. |
|
test-orka-macos11-x64-2 is back online and processing jobs. Backlog is currently 16 queued jobs (+1 citgm job) for osx11-x64. |
|
Getting e.g.: https://ci.nodejs.org/job/node-test-commit-osx/61013/nodes=osx11-x64/console |
|
Ran the cleanup script on test-orka-macos11-x64-1. |
|
Disk space is too low again on test-orka-macos11-x64-2. What's happening? |
|
Now both nodes are offline. We can't keep cleaning them manually every day. |
|
Have the macOS Node.js builds got considerably larger? 18 GB (#3878 (comment)) sounds really high -- builds on Linux are only ~3 GB. |
|
According to my local folders: I don't have a v18.x build. |
|
I've run the cleanup script on test-orka-macos11-x64-2 and rebooted the machine. This is now reporting ~21GB of free space -- a node-test-commit-osx run has started and it is currently consuming 1.8 GB of that (and would be expected to grow to 18 GB). |
|
I suppose one other question -- where is the tmp dir on the macOS machines? |
|
|
🤷 |
|
Ah in https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx11-x64/61152/injectedEnvVars/
|
So this is now: |
And now that the build completed (and a new one started): 12 MB left behind in the tmp dir: |
This was referenced Sep 6, 2024
|
I ran the cleanup script again on test-orka-macos11-x64-1. FWIW The temp dir was 389MB: I've added a small bit of cleanup to node-test-commit-osx: |
|
It looks like both runners are offline (again). Should we disable OSX on |
Yes please. This is getting to be a real problem. |
cjihrig
added a commit
to cjihrig/node
that referenced
this issue
Sep 9, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887
cjihrig
added a commit
to cjihrig/node
that referenced
this issue
Sep 9, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887
What in those folders is taking so much space? The Node.js build in MacPorts is failing for the same reason. |
$ find out/Release -type f -exec du -h {} \; | sort -rh | head -n 50 |
|
As a comparison, here are the numbers for a V8 build following the official documentation on the same machine: $ find out/arm64.release -type f -exec du -h {} \; | sort -rh | head -n 50 |
|
I disabled
|
nodejs-github-bot
pushed a commit
to nodejs/node
that referenced
this issue
Sep 11, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887 PR-URL: #54856 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com>
aduh95
pushed a commit
to nodejs/node
that referenced
this issue
Sep 12, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887 PR-URL: #54856 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com>
aduh95
pushed a commit
to nodejs/node
that referenced
this issue
Sep 13, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887 PR-URL: #54856 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com>
aduh95
pushed a commit
to nodejs/node
that referenced
this issue
Sep 13, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887 PR-URL: #54856 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com>
targos
pushed a commit
to nodejs/node
that referenced
this issue
Sep 22, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887 PR-URL: #54856 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com>
targos
pushed a commit
to nodejs/node
that referenced
this issue
Sep 26, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887 PR-URL: #54856 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com>
|
This was a symptom of having long lived OSX runners, which will soon be fixed by our transition to ephemeral Orka macos runners. In the meantime, the disk was filling because macos spotlight indexing was indexing the builds , and creating new, unique uuid to filename mappings, which was filling up /private/var/db/uuidtext I've disabled spotlight, and removed the spotlight databases. We've now got 31GB free with a workspace on And 52 GB free on
both of the orka-macos10.15-x64 machines have over 50GB available as well. We should be able to re-enable these now, and they should last until they are replaced, shortly. |
targos
pushed a commit
to nodejs/node
that referenced
this issue
Oct 2, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887 PR-URL: #54856 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com>
targos
pushed a commit
to nodejs/node
that referenced
this issue
Oct 2, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887 PR-URL: #54856 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com>
|
All of the legacy OSX machines have been running successfully now for about 36 hours. The build results are very flappy between green and yellow status, and almost every yellow status is one particular test that keeps flagging as flaky: Im not sure if that test is specifically flaky on OSX or if thats a problematic test in general, but its the vast majority of flaky results for the current OSX builds. |
louwers
pushed a commit
to louwers/node
that referenced
this issue
Nov 2, 2024
The test runner's code coverage leaves old coverage data in the temp directory. This commit updates the cleanup logic to: - Stop code collection. Otherwise V8 would write collection data again when the process exits. - Remove the temp directory containing the coverage data. - Attempt to clean up the coverage data even if parsing the data resulted in an error. With this change, I no longer see any coverage data left behind in the system temp directory. Refs: nodejs/build#3864 Refs: nodejs/build#3887 PR-URL: nodejs#54856 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey folks,
I'm trying to finish the CI for nodejs/node#54560 (target is today) and osx test is taking too long (it seems stuck) >12h~
The text was updated successfully, but these errors were encountered: