test-rackspace-win2022_vs2022-x64-6 systematic build timeout for win-vs2022-arm64

[targos]

See https://ci.nodejs.org/computer/test-rackspace-win2022_vs2022-x64-6/builds

All win-vs2022-arm64 systematically fail with:

00:56:43 Cross-compilation to ARM64 detected. We'll use the x64 Node executable for license2rtf.
01:56:43 Build timed out (after 60 minutes). Marking the build as failed.
01:56:44 Build was aborted

Examples:
https://ci.nodejs.org/job/node-compile-windows/nodes=win-vs2022-arm64/52608/console
https://ci.nodejs.org/job/node-compile-windows/nodes=win-vs2022-arm64/52601/console
https://ci.nodejs.org/job/node-compile-windows/nodes=win-vs2022-arm64/52597/console

Other machines don't have this problem so I removed the win-vs2022-arm64 label from that one in https://ci.nodejs.org/computer/test-rackspace-win2022_vs2022-x64-6/configure to prevent CI failures.

[joyeecheung]

nodejs/reliability#676 indicates that this happens on test-rackspace-win2022_vs2022-x64-1 and test-rackspace-win2022_vs2022-x64-2 too, maybe less frequent though

[richardlau]

00:56:43 Cross-compilation to ARM64 detected. We'll use the x64 Node executable for license2rtf.

This message is immediately before an attempt is made to download a x64 Node.js Windows binary (to convert the license to RTF). #3477 (comment) says that on the test CI that there should be a caching mechanism to prevent the builds getting stuck here.

[targos]

Two win2019 machines are also affected:
https://ci.nodejs.org/computer/test-rackspace-win2019_vs2019-x64-1/builds
https://ci.nodejs.org/computer/test-rackspace-win2019_vs2019-x64-2/builds

[StefanStojanovic]

I took a look. There is indeed a cache there but it can get corrupted (that's what happened here). I'll improve the logic to prohibit this. There is also an occasional ongoing problem with release CI where nodes get stuck trying to download the x64 node.exe for days (no caching there). I'll try to make a standardized approach that will work in both CI environments.

[targos]

Label added back to https://ci.nodejs.org/computer/test-rackspace-win2022_vs2022-x64-6/configure

[targos]

@StefanStojanovic It seems the bug is still here: https://ci.nodejs.org/job/node-compile-windows/53041/nodes=win-vs2022-arm64/console

[StefanStojanovic]

Thanks for letting me know @targos! This is a different bug, but it represents the same. Caching only works well if it can download the x64 binary, in runs after you reenabled the machine I've seen this when downloading node.exe:

curl: (18) transfer closed with 69583820 bytes remaining to read

The machine didn't have a cached node.exe and in one run compilation failed. The new caching mechanism improves on the old one in a way that it doesn't overwrite a valid binary with an invalid one. Since this machine didn't have a valid binary to begin with, failure was a possibility. Now I've set it up correctly and checked other machines for invalid cache. Since now everything is as expected, the new caching mechanism should make sure it stays like that by never invalidating node.exe.

I'll be monitoring this in the following weeks and hopefully, this will not happen again.

[targos]

Removed win-vs2022-arm64 label from https://ci.nodejs.org/computer/test%2Drackspace%2Dwin2022%5Fvs2022%2Dx64%2D1/configure

[targos]

Happened with a release machine: https://ci-release.nodejs.org/job/iojs+release/10022/nodes=vs2022-arm64/console

[targos]

https://ci-release.nodejs.org/job/iojs+release/10201/nodes=vs2022-arm64/console

[targos]

https://ci-release.nodejs.org/job/iojs+release/10275/nodes=vs2022-arm64/console

[richardlau]

https://ci-release.nodejs.org/job/iojs+release/nodes=vs2022-arm64/10284/console

[StefanStojanovic]

I recall that release CI doesn't use node.exe caching for signing ARM64 binaries. I even opened an issue to implement it some time ago #3527. This was based on an improvement I had for caching (https://github.com/nodejs/build/pull/3514/files), but it broke test CI because of SETLOCAL EnableDelayedExpansion so I had to revert it and I haven't touched it since then.

In short, in release CI we need the download from https://nodejs.org/dist/latest/win-x64/node.exe to go smoothly and that is not always the case as sometimes it hangs. Luckily that doesn't happen too often.

[StefanStojanovic]

After landing the PR I ran windows fanned job, and node compile debug job, that was previously failing (with first PR that needed to be reverted) now passed. Will monitor CI runs closely in the next few days to make sure everything is OK.

[targos]

What about https://ci-release.nodejs.org/job/iojs+release/10380/nodes=vs2022-arm64/console ?

[StefanStojanovic]

Release CI doesn't use the compile.cmd, so the change I landed doesn't affect machines there. There is another issue I opened for this: #3527

[StefanStojanovic]

However, looking at everything now, it seems very likely that this somehow causes the error described here, which is very weird since the compile.cmd script is not used in test jobs at all. Additionally, my changes do not change the compilation process in any way as they are done as a preparation step, so I do not expect Node.js binaries produced to be any different, but the ones compiled for ARM64 most likely are. The last 2 runs of my test job are the first occurrences of failures that have been bringing havoc in the Windows CI today. At that time this seemed as a coincidence, but from what I've seen today, it is not. I will revert the changes I landed here. After that, I expect everything to go back to normal.