Compilation broken on Windows in the CI

[StefanStojanovic]

node-compile-windows and node-compile-windows-debug have been constantly failing since the weekend began. I'm currently investigating it and the reason seems to be the latest update of VS 2022 to 17.10.0 (everything was fine with 17.9.7). While I'm working on it, those jobs/machines will be down for some time, so just letting everyone know in advance. I'll write update here once I have something.

[targos]

I tried to compile locally out of curiosity. This is what I get:

  maglev.cc
C:\Users\mzasso\git\node\deps\v8\src\maglev\maglev-ir.cc(3855): fatal  error C1001: Internal compiler error. [C:\Users\
mzasso\git\node\tools\v8_gypfiles\v8_base_without_compiler.vcxproj]
  (compiler file 'D:\a\_work\1\s\src\vctools\Compiler\Utc\src\p2\main.c', line 242)
   To work around this problem, try simplifying or changing the program near the locations listed above.
  If possible please provide a repro here: https://developercommunity.visualstudio.com
  Please choose the Technical Support command on the Visual C++
   Help menu, or open the Technical Support help file for more information
    CL!RaiseException()+0x54
    CL!DllGetObjHandler()+0x31a0c
    CL!DllGetObjHandler()+0x448
    CL!DllGetObjHandler()+0x5c2690
    CL!DllGetObjHandler()+0x6cf368
    CL!DllGetObjHandler()+0x7c7a74
    CL!DllGetObjHandler()+0x7bce74
    CL!DllGetObjHandler()+0x6cd960
    CL!DllGetObjHandler()+0x394fbc
    CL!DllGetObjHandler()+0x114b0
    CL!DllGetObjHandler()+0x85d0
    CL!DllGetObjHandler()+0x18c88
    CL!DllGetObjHandler()+0x3fb918
    CL!strtol()+0xac
    CL!BaseThreadInitThunk()+0x30
    CL!RtlUserThreadStart()+0x3c

[StefanStojanovic]

I've managed to fix the compilation jobs. The problem was as I suspected the update to VS 17.10.0. Luckily, the VS community edition that we use offers rollback, so I took it back to 17.9.7 and pinned that version. There is also another possibility: to install VS2019 build tools with the exact version and then use VS only for the editor and add workloads, components, etc. from the build tools installation.

vcbuild.bat and vswhere_usability_wrapper.cmd are written so they only accept the latest version of build tools, so if you have VS 17.10.0, Node will not let you compile with eg v17.9.7 compiler. Since ClangCL is coming (hopefully) soon, I do not think this is worth fixing to support various versions of MSVC, but I'll keep it on my radar just in case.

The new problem I see now is that node-test-binary-windows-native-suites are constantly failing since yesterday (they were not running 1-2 days before that because of the compilation issue), so I'll dig into that.

[StefanStojanovic]

The fix was applied yesterday and everything seems to be back to normal, Some PRs will need to be rebased to pass the CI, but other than that, everything else is working normally.

I will keep this issue open for a few more days, in case something goes wrong. Planning to close it on Friday.

[targos]

What's the long term solution ? Is Microsoft aware of the MSVC bug?

[VXACDev]

Broken all over the place... The only "solution" with the current MSVC release is to #pragma optimize off/on around the problem function. REALLY not good-- this is impact MANY projects (including Unreal Engine)

[swaroop-sridhar]

Hi @targos @StefanStojanovic:
I work for the MSVC optimizer team, and we want to look into the above failure and fix it.

We'd really appreciate if you can provide a repro of the failure (ex: a pre-processed Cpp file and cl command line used) so that we can investigate the compiler error. Please file an issue with the repro at: https://developercommunity.visualstudio.com/cpp/report. Thanks.
This will help to resolve the issue and unblock the VS-toolset update.

Thanks.

[VXACDev]

I pasted how to repro.. it’s a public repository “Vahalla Game” on but.. I just got an msvc update about 2 hrs ago, and it’s working properly again, so looks like someone fixed it… but double check me on that. The issue was in “RenderSky.cpp” in the draw() function. Mind you this isn’t my code, I use it for a demo/unit test for compatibility. Happy to provide more info if you need it. Sent from [Proton Mail](https://proton.me/mail/home) for iOS

…

On Wed, May 29, 2024 at 4:16 PM, Swaroop Sridhar ***@***.***(mailto:On Wed, May 29, 2024 at 4:16 PM, Swaroop Sridhar <<a href=)> wrote: Hi ***@***.***(https://github.com/targos) ***@***.***(https://github.com/StefanStojanovic): I work for the MSVC optimizer team, and we want to look into the above failure and fix it. We'd really appreciate if you can provide a repro of the failure (ex: a pre-processed Cpp file and cl command line used) so that we can investigate the compiler error. Please file an issue with the repro at: https://developercommunity.visualstudio.com/cpp/report. Thanks. This will help to resolve the issue and unblock the VS-toolset update. Thanks. — Reply to this email directly, [view it on GitHub](#3739 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/BEZUO2RFKSLJOWKJQML4QTLZEZATPAVCNFSM6AAAAABIJUKI2SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZYGI3TOOBWGY). You are receiving this because you commented.Message ID: ***@***.***>

[StefanStojanovic]

What's the long term solution ? Is Microsoft aware of the MSVC bug?

I've brought this up in the conversation with my contact at Microsoft. He told me, he'll look into it and get back to me. I've also noticed there is a new version of VS v17.10.1 (still not in Chocolatey, but will be there in a few days probably). I'll try it locally to see if it compiles correctly.

[swaroop-sridhar]

Hi @StefanStojanovic it is still useful if you can file a developer community ticket with the repro at: https://developercommunity.visualstudio.com/cpp/report
This way, the issue will be officially tracked and resolved.

Thanks.

[targos]

I tried the new version and it fails as well. I'll need some time to find a way to generate a preprocessed file to submit to the community forums

[targos]

I don't really know what to do. As soon as I add /P to the MSVC arguments, compilation fails saying that it cannot find v8_pch.h.

[VXACDev]

There's a 2nd problem with these fixes [Compiler Version Mismatch:: There's a side-effect to the last "Fix" that sort-of fixed the optimization having an internal error-- Now, libraries built won't link on older (as in 1 week older) VS 2022, meaning you will need to deploy LIBs (if you deploy libs to customer) for 2022 prior to Mid-May 2022, AND Libs AFTER Mid-May 2022. The unit-test is quite easy.. just build any project with MSVC Prior to the last 2 updates, and create and link to a LIB made in the latest MSVC, and you'll get an error saying the LIB and OBJs have a compiler-version mismatch. Forcing customers to "upgrade" their existing compiler is not acceptable either (production machines simply won't do that on a whim). Microsoft Visual Studio Professional 2022 Version 17.10.1 VisualStudio.17.Release/17.10.1+34928.147 Microsoft .NET Framework Version 4.8.09032 Installed Version: Professional Visual C++ 2022 00483-10050-07952-AA543 Microsoft Visual C++ 2022 ASP.NET and Web Tools 17.10.338.1105 ASP.NET and Web Tools Azure App Service Tools v3.0.0 17.10.338.1105 Azure App Service Tools v3.0.0 C# Tools 4.10.0-3.24270.2+e8f775c1d8a73dee7ad02408712d714251e708ea C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used. Extensibility Message Bus 1.4.39 ***@***.***) Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration. GitHub Copilot 0.2.889.30432 GitHub Copilot is an AI pair programmer that helps you write code faster and with less work. Linux Core Dump Debugging 1.0.9.34902 Enables debugging of Linux core dumps. Microsoft JVM Debugger 1.0 Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines NuGet Package Manager 6.10.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/ Razor (ASP.NET Core) 17.10.3.2427201+4f57d1de251e654812adde201c0265a8ca7ca31d Provides languages services for ASP.NET Core Razor. Test Adapter for Boost.Test 1.0 Enables Visual Studio's testing tools with unit tests written for Boost.Test. The use terms and Third Party Notices are available in the extension installation directory. Test Adapter for Google Test 1.0 Enables Visual Studio's testing tools with unit tests written for Google Test. The use terms and Third Party Notices are available in the extension installation directory. TypeScript Tools 17.0.30327.2001 TypeScript Tools for Microsoft Visual Studio Visual Basic Tools 4.10.0-3.24270.2+e8f775c1d8a73dee7ad02408712d714251e708ea Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used. Visual C++ for Linux Development 1.0.9.34902 Visual C++ for Linux Development Visual F# Tools 17.10.0-beta.24228.1+dd749058c91585e9b5dae62b0f8df892429ee28f Microsoft Visual F# Tools Visual Studio IntelliCode 2.2 AI-assisted development for Visual Studio. VisualStudio.DeviceLog 1.0 Information about my package VisualStudio.Mac 1.0 Mac Extension for Visual Studio VSPackage Extension 1.0 VSPackage Visual Studio Extension Detailed Info Xamarin 17.10.0.110 ***@***.***) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android. Xamarin Designer 17.10.3.10 ***@***.***) Visual Studio extension to enable Xamarin Designer tools in Visual Studio. Xamarin.Android SDK 13.2.2.0 (d17-5/45b0e14) Xamarin.Android Reference Assemblies and MSBuild support. Mono: d9a6e87 Java.Interop: ***@***.*** SQLite: ***@***.*** Xamarin.Android Tools: ***@***.*** Sent with [Proton Mail](https://proton.me/) secure email.

…

On Friday, May 31st, 2024 at 3:57 AM, Michaël Zasso ***@***.***> wrote: I don't really know what to do. As soon as I add /P to the MSVC arguments, compilation fails saying that it cannot find v8_pch.h. — Reply to this email directly, [view it on GitHub](#3739 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/BEZUO2VIJXBTDBAAARQKQMDZFA3QHAVCNFSM6AAAAABIJUKI2SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBRGUZTQOBTGE). You are receiving this because you commented.Message ID: ***@***.***>

[swaroop-sridhar]

Hi @targos, the failure about a missing header usually indicates a missing include path in the compilation step. Do you get the same error with /E build? If you cannot generate the pre-compiled source, can you please share the instructions to build from the repro? (ex: After cloning nodejs repo, the build steps enough to follow, and the cl invocation that produces the compiler error)? I'll take a look using the same steps. Thanks.

[targos]

From the main branch of the nodejs/node repo:

.\vcbuild.bat debug

After some time, it errors.

Then:

cd tools\v8_gypfiles
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\CL.exe" /c /I..\..\deps\v8 /I..\..\deps\v8\include /I"..\..\out\Debug\obj\global_intermediate\generate-bytecode-output-root" /I..\..\out\Debug\obj\global_intermediate /I"..\..\deps\icu-small\source\i18n" /I"..\..\deps\icu-small\source\common" /I"..\..\deps\v8\third_party\abseil-cpp" /Z7 /nologo /W3 /WX- /diagnostics:column /MP /O2 /Ob2 /Oi /Oy- /D _GLIBCXX_USE_CXX11_ABI=1 /D NODE_OPENSSL_CONF_NAME=nodejs_conf /D NODE_OPENSSL_HAS_QUIC /D ICU_NO_USER_DATA_OVERRIDE /D V8_GYP_BUILD /D V8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64 /D V8_ENABLE_CHECKS /D WIN32 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _HAS_EXCEPTIONS=0 /D BUILDING_V8_SHARED=1 /D BUILDING_UV_SHARED=1 /D NOMINMAX /D OPENSSL_NO_PINSHARED /D OPENSSL_THREADS /D V8_TARGET_ARCH_X64 /D _WIN32_WINNT=0x0602 /D _SILENCE_ALL_CXX20_DEPRECATION_WARNINGS /D V8_HAVE_TARGET_OS /D V8_TARGET_OS_WIN /D "V8_EMBEDDER_STRING=\"-node.13\"" /D ENABLE_DISASSEMBLER /D V8_PROMISE_INTERNAL_FIELD_COUNT=1 /D V8_SHORT_BUILTIN_CALLS /D OBJECT_PRINT /D V8_INTL_SUPPORT /D V8_ATOMIC_OBJECT_FIELD_WRITES /D V8_ENABLE_LAZY_SOURCE_POSITIONS /D V8_USE_SIPHASH /D V8_SHARED_RO_HEAP /D NDEBUG /D V8_WIN64_UNWINDING_INFO /D V8_ENABLE_REGEXP_INTERPRETER_THREADED_DISPATCH /D V8_USE_ZLIB /D V8_ENABLE_SPARKPLUG /D V8_ENABLE_MAGLEV /D V8_ENABLE_TURBOFAN /D V8_ENABLE_SYSTEM_INSTRUMENTATION /D V8_ENABLE_ETW_STACK_WALKING /D V8_ENABLE_WEBASSEMBLY /D V8_ENABLE_JAVASCRIPT_PROMISE_HOOKS /D V8_ENABLE_CONTINUATION_PRESERVED_EMBEDDER_DATA /D V8_ALLOCATION_FOLDING /D V8_ALLOCATION_SITE_TRACKING /D V8_ADVANCED_BIGINT_ALGORITHMS /D UCONFIG_NO_SERVICE=1 /D U_ENABLE_DYLOAD=0 /D U_STATIC_IMPLEMENTATION=1 /D U_HAVE_STD_STRING=1 /D UCONFIG_NO_BREAK_ITERATION=0 /D DEBUG /D _DEBUG /D V8_TRACE_MAPS /D V8_ENABLE_ALLOCATION_TIMEOUT /D V8_ENABLE_FORCE_SLOW_PATH /D ENABLE_HANDLE_ZAPPING /D _UNICODE /D UNICODE /GF /Gm- /MTd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Yu"v8_pch.h" /Fp"..\..\out\Debug\obj\v8_compiler\v8_compiler.pch" /Fo"..\..\out\Debug\obj\v8_compiler\\deps\v8\src\compiler\backend\instruction-selector.obj" /Fd"..\..\out\Debug\v8_compiler.pdb" /external:W3 /Gd /TP /wd4351 /wd4355 /wd4800 /wd4251 /wd4275 /wd4244 /wd4267 /wd4129 /wd4245 /wd4324 /wd4506 /wd4661 /wd4701 /wd4702 /wd4703 /wd4709 /wd4714 /wd4715 /wd4718 /wd4723 /wd4724 /FIv8_pch.h /FC /errorReport:prompt /utf-8 /Zc:__cplusplus -std:c++20 /Zm2000 "..\..\deps\v8\src\compiler\backend\instruction-selector.cc"

Error:

instruction-selector.cc
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,75): error C2100: vous ne pouvez pas déréférencer un opérande de type 'v8::internal::compiler::turboshaft::WordRepresentation'
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,75): note: le contexte d’instanciation du modèle (le plus ancien) est
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,20): note: voir la référence à l'instanciation alias modèle 'v8::internal::compiler::turboshaft::Opmask::MaskBuilder<v8::internal::compiler::turboshaft::WordBinopOp,v8::internal::compiler::turboshaft::Opmask::OpMaskField<v8::internal::compiler::turboshaft::WordBinopOp::Kind,4>,v8::internal::compiler::turboshaft::Opmask::OpMaskField<v8::internal::compiler::turboshaft::Opmask::UnwrapRepresentation<v8::internal::compiler::turboshaft::WordRepresentation>::type,5>>::For<v8::internal::compiler::turboshaft::WordBinopOp::Kind::kAdd,v8::internal::compiler::turboshaft::WordRepresentation{v8::internal::compiler::turboshaft::RegisterRepresentation{v8::internal::compiler::turboshaft::MaybeRegisterRepresentation{v8::internal::compiler::turboshaft::MaybeRegisterRepresentation::Enum:v8::internal::compiler::turboshaft::MaybeRegisterRepresentation::Enum::kWord32}}}>' en cours de compilation
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(136,15): fatal error C1907: impossible de récupérer à partir des erreurs précédentes ; arrêt de la compilation
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\representations.h(457,49): note: durant l'évaluation de la fonction constexpr 'v8::internal::compiler::turboshaft::WordRepresentation::value'
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,75): note: durant l'évaluation de la fonction constexpr 'v8::internal::compiler::turboshaft::WordRepresentation::operator v8::internal::compiler::turboshaft::WordRepresentation::Enum'
ERREUR INTERNE DU COMPILATEUR dans 'C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\cl.exe'
    Choisissez la commande Support technique du menu ? (Aide) de Visual C++
  ou ouvrez le fichier d'aide du Support technique pour plus d'informations

• If I slightly modify the command to remove the arguments referencing v8_pch.h, it still errors.
• If I add /P or /E to the modified command, it doesn't error anymore.

[targos]

Ok, I was able to make a standalone repro using the output from /E: repro.zip

Opened https://developercommunity.visualstudio.com/t/Internal-compiler-error-with-MSVC-1440/10673166

[targos]

The report I made is now Fixed - Pending Release.

[targos]

VS 17.10.2 was released, but it doesn't include an updated MSVC.

[liudonghua123]

I encountered the similar problems when build node v22.3.0 recently, see the detailed ci logs here.

And at the same time I build v20.15.0, it is ok, the ci environment is the same (windows-latest in github actions).

The github action workflow file is windows-latest node-build.yml.

[targos]

Still an issue with VS 17.10.3.

[StefanStojanovic]

I encountered the similar problems when build node v22.3.0 recently, see the detailed ci logs here.

And at the same time I build v20.15.0, it is ok, the ci environment is the same (windows-latest in github actions).

The github action workflow file is windows-latest node-build.yml.

This is most likely due to the V8 version we have in those 2 versions.

[cristianadam]

Qt WebEngine has a fix at https://codereview.qt-project.org/c/qt/qtwebengine-chromium/+/573472

[StefanStojanovic]

Maybe some modification need to make on the upstream v8 project to fix it.

V8 is dropping MSVC soon and they'll most likely not accept any MSVC specific patch at this time.

[davidair]

Are there multiple issues? I tried compiling Node using VS 17.11 Preview 8 and also VS 17.8 but I am getting errors which might be different but can be resolved by making changes similar to the Qt WebEngine patch mentioned above. More specifically, I am seeing this:

Error (active) E0413 no suitable conversion function from "v8::internal::compiler::turboshaft::WordRepresentation" to "v8::internal::compiler::turboshaft::Opmask::OpMaskField<v8::internal::compiler::turboshaft::Opmask::UnwrapRepresentation<v8::internal::compiler::turboshaft::WordBinopOp::Kind>::type, 4Ui64>::type" (aka "v8::internal::compiler::turboshaft::WordBinopOp::Kind") exists torque_generated_definitions C:\open\node\deps\v8\src\compiler\turboshaft\opmasks.h 150

Is anyone seeing this? I get this when I open node.sln in VS and hit build. I think it used to work for me before.

[ShenHongFei]

It looks like another msvc compiler issue. Can you @davidair report a new problem? (https://developercommunity.visualstudio.com/cpp/report)

[davidair]

I reported the E0413 issue to Microsoft: https://developercommunity.visualstudio.com/t/E0413-no-suitable-conversion-function-fr/10709858

[targos]

Good news: Visual Studio 17.11.0 was just released and the build works again (at least on my PC)!

[StefanStojanovic]

Good news: Visual Studio 17.11.0 was just released and the build works again (at least on my PC)!

I tested it as well and it works for me too. We can wait a week or two, for safety and then unblock VS upgrade in the CI and eventually reenable GH action for building Windows.

[liudonghua123]

For the vision of Visual Studio used in windows-latest/window-2022 environment, you can check it on https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md#visual-studio-enterprise-2022.

And from the commit history, the update frequence seems to be a week.

[richardlau]

For the vision of Visual Studio used in windows-latest/window-2022 environment, you can check it on https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md#visual-studio-enterprise-2022.

actions/runner-images#10448

[targos]

It was deployed on GitHub Actions: actions/runner-images#10448 (comment)

Here's a run on my fork: https://github.com/targos/node/actions/runs/10643236538

[targos]

It passed so I enabled https://github.com/nodejs/node/actions/workflows/build-windows.yml and https://github.com/nodejs/node/actions/workflows/coverage-windows.yml.

[targos]

GH Actions runs are green. @StefanStojanovic I let you close this as soon as Jenkins hosts are updated.

[StefanStojanovic]

Thanks for enabling that @targos that's great! However, updating Jenkins CI won't be as straightforward as removing the Chocoaltey pin for VS and upgrading. While working on nodejs/node#54655 I did a lot of builds for all of the combinations (MSVC/ClangCL + x64/ARM64). I noticed that VS 17.11 MSVC + ARM64 doesn't compile (on VS 17.9 all works fine). The error I get is this:

E:\work\node\deps\ngtcp2\nghttp3\lib\nghttp3_ringbuf.c(38,14): error C2169: '__popcnt': intrinsic function, cannot be defined [E:\work\node\deps\ngtcp2\nghttp3.vcxproj]
E:\work\node\deps\ngtcp2\ngtcp2\lib\ngtcp2_ringbuf.c(36,21): error C2169: '__popcnt': intrinsic function, cannot be defined [E:\work\node\deps\ngtcp2\ngtcp2.vcxproj]

I'm OOF this week, so when I'm back I'll open an issue for this, but we'll probably need to submit a fix upstream since this problem is with the dependency.