Windows release builds stuck

[richardlau]

Looks like we have executor starvation for the vs2019-arm64 machines on ci-release. The three current non-disconnected nodes look stuck:

• https://ci-release.nodejs.org/job/iojs+release/nodes=vs2019-arm64/9599/ (running for a day)
• https://ci-release.nodejs.org/job/iojs+release/nodes=vs2019-arm64/9595/ (running for three days)
• https://ci-release.nodejs.org/job/iojs+release/nodes=vs2019-arm64/9585/ (running for seven days)

All three appear to be attempting to download the x64 node binary to run license2rtf:
e.g. https://ci-release.nodejs.org/job/iojs+release/nodes=vs2019-arm64/9599/console

13:16:23 Cross-compilation to ARM64 detected. We'll use the x64 Node executable for license2rtf.
13:16:23 Downloading x64 node.exe...

Since this tries to download from https://nodejs.org/dist this is probably related to the slow downloads/500 errors due to the server that hosts the downloads being overwhelmed.

I haven't canceled any of the stuck jobs in case somebody wants to investigate further.

cc @StefanStojanovic

[StefanStojanovic]

I've taken care of it. It was indeed a problem with downloading x64 node.exe for signing. On test CI we have a caching mechanism to ensure things like this do not happen. The same thing, or something similar, should be made on the release CI as well in the future. This is fixed for now.

[targos]

It still happens occasionally:
https://ci-release.nodejs.org/job/iojs+release/10569/nodes=vs2022-arm64/
https://ci-release.nodejs.org/job/iojs+release/10559/nodes=vs2022-arm64/

[targos]

https://ci-release.nodejs.org/job/iojs+release/10581/nodes=vs2022-arm64/

[StefanStojanovic]

This is annoying and happens from time to time. It can also happen in Test CI, but rarely. I tried 2 fixes for it before (in the Test CI), but both broke some other part of the run, so they had to be reverted.

This time I'm aiming for a solution that works rather than a pretty solution, so I made a fix in Release CI which I will likely port to Test CI. The fix is simple. If there's node.exe in a certain location, copy it where vcbuild.bat will expect it once license generation starts. I've downloaded the latest LTS node.exe in that location on release machines so it will not need downloading anymore. Even if we don't update those node.exe files it'll be OK as it is only used for generating license, but I plan to update them from time to time.

After some time I'll apply this to test CI as well as there we have a script to update node.exe weekly, which is not really necessary.

I made a test job while working on this iojs+release-mefi-arm64