`node_mksnapshot.exe` fails with `SyntaxError: Invalid or unexpected token`

[tniessen]

I (luckily) haven't had to build Node.js on Windows in a long time, but I (unfortunately) need to now. With the main branch checked out at d396a04 and no unstaged changes, I see the following error at the end of the build process:

> vcbuild
Looking for Python
Python found in C:\Program Files\Python310\\python.exe
Python 3.10.11
Looking for Visual Studio 2022
Looking for Visual Studio 2019
Found MSVS version 16.0

...

  node_mksnapshot.vcxproj -> out\Release\\node_mksnapshot.exe
  node_mksnapshot
  out\Release\node_mksnapshot.exe: node:internal/timers:687



  SyntaxError: Invalid or unexpected token
      at compileForInternalLoader (node:internal/bootstrap/realm:389:18)
      at requireBuiltin (node:internal/bootstrap/realm:423:14)
      at node:internal/bootstrap/node:69:24

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5): error MSB8066

The file lib/internal/timers.js is unmodified. Its SHA-1 checksum is 716eccf542993957462b7ad0b9c13db91c046cec.

[joyeecheung]

It's just a guess, but does removing the NODE_JS2C_USE_STRING_LITERALS and NODE_MKSNAPSHOT_USE_STRING_LITERALS flags in node.gyp make a difference? (Could be some compiler issue regarding UTF16 encoding, and lib/internal/timers.js is one of the few files that contain two-byte characters)

[joyeecheung]

Hmm, wait, we don't turn these on on Windows anyway, so it's probably not related to the use of string literals. It could still be related to UTF16 encoding though, because it fails in lib/internal/timers.js (the last time I built Node.js on Windows was probably a month ago, with Visual Studio 2022, and I didn't notice any issue, but I could check again this week)

[tniessen]

@joyeecheung Genius! Replacing [^\x00-\x7F] with an ASCII character in lib/internal/timers.js and deps/undici/undici.js appears to be a workaround for now. I believe I am using a rather typical Windows setup, but if nobody else is facing these problems, I'd be happy to close this issue.

[joyeecheung]

I verified that I could still build the main branch with Visual Studio 2022. I think one possible cause might be that the check out on your machine is not in UTF8? We assume the JS files on disk are encoded in UTF8 and transcode them to UTF16 if they contain 2 byte characters (if they only contain ASCII characters, they are copied verbatim).

[tniessen]

@joyeecheung The SHA-1 sum of the file is the same on Windows and on my usual Ubuntu setup, and it renders correctly when I open it as a UTF-8 text file on Windows.

[joyeecheung]

@tniessen Can you comment out the NODE_JS2C_USE_STRING_LITERALS part in node.gyp and build on different machines, then compare the output of out/Release/gen/node_javascript.cc? (in particular, the internal_timers_raw array?)

[RafaelGSS]

While doing the GHC a few folks faced this error. No easy workaround though.

[tniessen]

@joyeecheung Sorry, I (fortunately) didn't have to go back to Windows once you'd provided a (genius) workaround.

[RafaelGSS]

Do we have a .patch we can provide to the ones facing this issue until we fix it? I didn't get entirely the workaround:

Replacing [^\x00-\x7F] with an ASCII character in lib/internal/timers.js and deps/undici/undici.js appears to be a workaround for now.

[bnoordhuis]

Sounds like a git encoding issue. You can configure it to change the encoding on checkout.

Should be fixable by adding lib/internal/timers.js and deps/undici/undici.js to .gitattributes.

[joyeecheung]

It seems the OP was using VS2019, I wonder if others encountering the same issue are also using VS2019. If so, this might be fixed by #50091 (see simdutf/simdutf#328 for the reason, that explained why I couldn't reproduce it with VS2022)

[tniessen]

Sorry @joyeecheung, I (luckily) didn't get back to any Windows setup since you provided the workaround.