Adding ClangCL to the CI

[StefanStojanovic]

Plan

Since Node.js can now be compiled with ClangCL on Windows, I'll start adapting the CI to start using that. Here is the plan in short and we'll work out the details as we go forward:

1. Start compiling with ClangCL - Initially, we can add only x64, and later include arm64 too. The idea is to compile for some time to gain confidence. This phase should not be too long unless we encounter some issues.
2. Start testing the ClangCL-produced binaries - This is a second phase, that will be prepared during the first one (eg. detecting flaky tests, fixing them or marking them flaky, etc).
3. Start releasing ClangCL-produced binaries - This will happen after we gain enough confidence from testing. At this point, we can start thinking about dropping MSVC support in the long run or at least making it a lower-tier compiler on Windows.

Preparation

This part is done, as a part of it I did the following:

1. Configured compilation machines to be able to compile with ClangCL. I've also made ansible changes so that the machines we create in the future will be ready to do it out of the box
2. Added new labels for both compiling and testing the ClangCL setup.
3. Created temp jobs where I tested all of this before moving to the main Windows fanned job.

Other work

Besides the main plan, there are some things to take care of as well (this list might get new entries later on):

1. Enable compilation caching for ClangCL - ccache we are using for MSVC doesn't work here, I'm currently trying out ccache, but it is also not working out of the box because of the build process (V8 is generating some files in the obj directory which interferes with PCH).

[StefanStojanovic]

An update on this. Both x64 and arm64 architectures are compiled with ClangCL now, and thanks to ccache it is after than MSVC. Since the debug compilation job uses the release configuration, it will start building with CLang once we enable it in the release CI.

Next step is enabling the tests. I've already prepared everything, but there is a question of bandwidth as we will run double the set of tests now. I will probably start by enabling 1-2 configurations to begin with. VS2022 configurations need to be enabled last as those machines are already under a bigger load due to the increased number of compilations per run.

Also just want to note that I've already tested both debug compilation and testing in my temporary job and all passed fine so I'm not expecting any big issues with it other than the bandwidth.

[targos]

Thanks for the update. Is the goal for Node.js 24 to remove MSVC support? Asking since I still got no answer about the Windows issues on nodejs/node#57114, where Clang builds pass, but MSVC builds fail.

[StefanStojanovic]

Thanks for the update. Is the goal for Node.js 24 to remove MSVC support? Asking since I still got no answer about the Windows issues on nodejs/node#57114, where Clang builds pass, but MSVC builds fail.

Yes that is the idea. Next week I can try and resolve the issues with MSVC, in case ClangCL doesn't get ready for v24, but I would like to see v24 release happen with ClangCL. That is also the reason why I still haven't got to that as I focused on Clang CI changes.

By the way, should I start some kind of discussion with TSC (or someone else) regarding the transition to Clang in v24? It is still not ready (will be ready in time I think), but maybe it'll be too late to open that question later?

[targos]

I guess it doesn't hurt to ping @nodejs/tsc since the change will be semver-major.

[joyeecheung]

In general transitioning to Clang in v24 SGTM. Though I wonder if there are any problems with addons? I recall that @nodejs/node-api did a lot of investigation in the ecosystem during the development of N-API. Is there some kind of list of popular projects using addons that we can look into and double check?

[targos]

You mean that addons built with MSVC might not be compatible with Node.js built with Clang?

[joyeecheung]

Yes, for example the ones that might be using MSVC options in the configurations for Windows. I think some breakage should be expected, but it would be better to provide something for them to workaround/adapt to Clang in 24 (e.g. do we expose the clang == 1 variable to them to create some conditions?). Theoratically it should be possible to link an MSVC-compiled addon to Clang build still, but I am not sure if the options we use for clang + the options from common.gypi for MSVC etc. can produce ABI compatible builds? If not they might need some kind of indicator to switch to a different toolchain in their script when being built against v24 too and do some adjustment in their config for v24.

[legendecas]

Node-apis are just C APIs and it might be a different case from C++ Addon APIs. From https://clang.llvm.org/docs/MSVCCompatibility.html, it reads to me that Clang provides ABI compatibility guarantees.

For common.gypi, node-api addons can be more easily compiled without gyp configs as not depending on V8 headers. However, projects like https://github.com/cmake-js/cmake-js can be used for both C node-api and C++ API. cmake-js only supports clang on UNIX systems as the document of --prefer-clang suggests.

[joyeecheung]

I meant that the N-API effort was initially a migration effort so the team may have an idea about what tricky things the ecosystem is doing with their addon build config that might be relevant :)

[mhdawson]

From the discussion in the node-api team today, this is one of the issues were we looked at popular modules with native code, including how we identified them - nodejs/abi-stable-node#22

[mhdawson]

This directory also includes some historical information about the assessment done in the early days of node-api - https://github.com/nodejs/abi-stable-node/tree/doc/data

[StefanStojanovic]

About the discussion here: do you know if there is anything to do about it now? Also, are you onboard with switching to ClangCL binaries for v24? If yes, I'll start working on enabling ClangCL in the release CI, as it is currently only supported in the test CI.

[StefanStojanovic]

Since there were no objections, I made the first step - made changes to the VersionSelectorScript.groovy in JaneaSystems@ecd67f6 and a temporary job to test it in the release CI. The initial run has passed, so as far as I can tell, we are ready to release ClangCL-produced binaries starting from Node.js v24.

If that happens, I'd like to stop compiling with MSVC in test CI in v24 and above because double compilations are currently bottlenecking our Windows 2022 machines in the CI. It is manageable, but waiting for it can get frustrating. Another idea is to add 1-2 Windows 2022 machines just for testing (they can have weaker hardware), but ideally, we'd go with the first option.

[StefanStojanovic]

I'm pinging people for visibility. Are we good to go with ClangCL on Windows for the v24 release? Is there something else that should be done before moving forward with it (e.g., checking something, etc.)? Once I get a green light, I'll open a PR in build and change the release CI build job.

cc @nodejs/tsc @targos @joyeecheung @legendecas @mhdawson

Since there were no objections, I made the first step - made changes to the VersionSelectorScript.groovy in JaneaSystems@ecd67f6 and a temporary job to test it in the release CI. The initial run has passed, so as far as I can tell, we are ready to release ClangCL-produced binaries starting from Node.js v24.

If that happens, I'd like to stop compiling with MSVC in test CI in v24 and above because double compilations are currently bottlenecking our Windows 2022 machines in the CI. It is manageable, but waiting for it can get frustrating. Another idea is to add 1-2 Windows 2022 machines just for testing (they can have weaker hardware), but ideally, we'd go with the first option.

[joyeecheung]

Oops, sorry, meant to reply but I got distracted. I feel that it might be worth it to check one or more representative addons (maybe from the list in nodejs/abi-stable-node#22) and see if they work okay with the ClangCL build. Just so to be sure that it still works. For example serialport, bcrypt and microtime may be good targets, because they are covered by https://github.com/nodejs/citgm (and so, we'd have to check that they still work before the 24 release anyway).

[joyeecheung]

Oh actually, it seems we don't have the ClangCL build in https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/ ? If we are dropping MSVC it seems necessary to add ClangCL to this job, or otherwise we'd just no longer have CITGM coverage on Windows? Then we can run it to see whether the addons included in CITGM still works.

[StefanStojanovic]

OK, I'll look into it tomorrow. Thanks for the feedback.

[StefanStojanovic]

I've set up the CITGM job to add ClangCL for x64 and arm64. I ran it once. ARM64 failures are the same regardless of the compiler. x64 compilations failed because of free space on one VM, which is critically low (investigating it now). Once all machines have enough free space I'll rerun x64 CITGM jobs to compare them.

[StefanStojanovic]

In both x64 and arm64 CITGM runs, I see the same results (failures) with both MSVC and ClangCL. Next, I'll go through this list and see if there are any differences.

[joyeecheung]

By the way it's worth noting that CITGM in general isn't always green. So as long as it doesn't produce more failures than the MSVC one it should be good enough.

[StefanStojanovic]

By the way it's worth noting that CITGM in general isn't always green. So as long as it doesn't produce more failures than the MSVC one it should be good enough.

Yes, that was my plan too, not necessarily to have it all green, but to make sure ClangCL is not breaking something MSVC is not.

[StefanStojanovic]

I also went through the list mentioned above, and again, the same behavior with MSVC and ClangCL. Can I take this (together with the CITGM results) as a green light to proceed with opening a PR in build and change the release CI build job to use ClangCL? In addition, I'd also disable compiling Node.js v24+ with MSVC in test CI, as that is a bottleneck (doubling the number of the longest jobs in the CI).

cc @nodejs/tsc @targos @joyeecheung @legendecas @mhdawson

[targos]

Did some of your tests involve running an MSVC-compiled addon with the ClangCL-compiled node.exe ?
If yes, I think we're good to go!

[joyeecheung]

@StefanStojanovic Do you still have the links to the CITGM runs?

[StefanStojanovic]

Did some of your tests involve running an MSVC-compiled addon with the ClangCL-compiled node.exe ? If yes, I think we're good to go!

I think so, but I'm not sure (I've worked on this on and off for some time). I'll try that tomorrow to double-check. Do you have a specific addon example in mind, or is a simple hello world addon enough?

@StefanStojanovic Do you still have the links to the CITGM runs?

Here they are: x64 and arm64

[joyeecheung]

I took a brief look - not sure if I missed any that was actually working, but it seems the addons have all been failing to build with MSVC on v24.x already?

[StefanStojanovic]

I took a brief look - not sure if I missed any that was actually working, but it seems the addons have all been failing to build with MSVC on v24.x already?

Yes. Looking as far as Jenkins allows (20 March at this time), I cannot see any run that didn't have all configurations red.

[StefanStojanovic]

Did some of your tests involve running an MSVC-compiled addon with the ClangCL-compiled node.exe ? If yes, I think we're good to go!

I've just finished testing this again. Everything works well for me. I tested a few scenarios with passing and/or returning strings and numbers. Saw no problems with it.

[StefanStojanovic]

I was preparing to open the PR, but noticed some issues, as test and release CI labels are connected in a way, but are actually used differently. Currently trying to find the easiest way to fix it. Might need to change some labels in the test CI. Should have it all ready on Monday.