-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix building on Windows x86 #12184
Fix building on Windows x86 #12184
Conversation
Do you have a machine setup and ready to add, that you've tested this on? |
hmm... I don't have a 32 bit machine to test on. I can get a VM set up later on but won't have time for a few days. |
I've tested a 32 bit build locally. I can create a temporary job to test a 32 bit build in CI. There seem to be issues with Visual C++ Build Tools, I'll look into it. |
Do you mean a 32-bit machine, or running the 32-bit build on a 64-bit machine? If the latter, then we can use the existing machines. Node 8 is currently failing for me on a 64-bit Windows machine (with |
I mean whatever is required to verify that this PR does what it says it does. |
Would setting msvs_shard in common.gypi work? That would save us one floating patch. |
@gibfahn I mean building and running the 32-bit exe on a 64-bit machine. We currently have no way of getting a real 32-bit machine in CI. |
e830fd0
to
431c314
Compare
The problem with Visual C++ Build Tools was over-parallelization. MSBuild is invoked with @bnoordhuis I did not find a way to set |
CI: https://ci.nodejs.org/job/node-test-commit/8894/ Temporary jobs that will be merged when this lands:
|
@nodejs/v8 @nodejs/build @nodejs/platform-windows The over-parallelization error just happened testing another pull request (https://ci.nodejs.org/job/node-compile-windows/8076/label=win-vs2015/). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM if CI is green
Sorry João, missed your comment. The patch below may do the trick; the shard level is inherited by dependencies if I read MSVSUtil.py right. Is there an easy way for me to test? diff --git a/node.gypi b/node.gypi
index d78d24d..6321131 100644
--- a/node.gypi
+++ b/node.gypi
@@ -25,6 +25,7 @@
'deps/v8/src/v8.gyp:v8',
'deps/v8/src/v8.gyp:v8_libplatform'
],
+ 'msvs_shard': 10,
}],
[ 'node_use_v8_platform=="true"', {
'defines': [ |
vcbuild.bat
Outdated
@@ -190,9 +190,11 @@ echo Project files generated. | |||
if defined nobuild goto sign | |||
|
|||
@rem Build the sln with msbuild. | |||
set "msbcpu=/m:2" | |||
if "%NUMBER_OF_PROCESSORS%"=="1" set "msbcpu=" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you set /m:1
It make the log output consistent with /m:n
>1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit++
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Builds on my machine. |
@refack is that with this PR as is? Could you check @bnoordhuis's patch? |
👎 makes only 4 shards... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM then.
I think it is inherited, but then overridden by deps/v8/src/v8.gyp:1744 |
MSBuild invokes cl.exe with /MP (set in common.gypi), making it compile sources in parallel using a number of internal processes equal to the number of effective processors. MSBuild /m uses a similar mechanism, so the number of compiler processes can grow to the number of effective processors squared. This limits MSBuild to 2 processes, to still use some parallelization while requiring less memory. Cl.exe is still invoked with /MP, thus the maximum number of processes is limited to twice the number of effective processors. PR-URL: nodejs#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
431c314
to
d50bb93
Compare
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: #12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: #12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: #12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: #12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Increase the number of shards to divide v8_base into. This increases the number of calls to cl.exe but decreases the number of files compiled each time. Fixes: nodejs/v8#4 PR-URL: nodejs/node#12184 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Patch Set 1: > Patch Set 1: > > > Patch Set 1: > > > > > Patch Set 1: > > > > > > > P.S. Mark, what is the importance? > > > > > > The same as it is here: https://chromium.googlesource.com/external/gyp/+/e8850240a433259052705fb8c56e51795b7dc9c3/pylib/gyp/MSVSVersion.py#107. Not doing this for 2017 is effectively a regression from what we did for 2013 and 2015. > > > > > > Why wouldn’t you choose the native 64-bit executable, larger address space and all, if it’s available to you? The fact that you’re targeting 32-bit should have absolutely no bearing on the decision. > > > > So there's no "real" importance > > I wouldn’t say that at all. We have, in the past, found that the larger address space available to a 64-bit process meant the difference between being able to build a large chunk of code under heavy optimization and not. Yeah, we've seen that in nodejs/node#12184 Patch-set: 1
Building on Windows x86 has been broken on master since the update of V8 to 5.7 (#11752, nodejs/v8#4, nodejs/build#669). This was not detected at the time because the CI matrix does not include a 32 bit build (which I plan to add after this lands).
When building V8 with Gyp, the library
v8_base
is divided into several shards because its size exceeds the limit. For each shard, a singlecl.exe
invocation is used to compile all.cc
files into.obj
files. In this invocation, some functions fromruntime.cc
are not getting compiled intoruntime.obj
. Usingcl.exe
with all the same arguments but to compile onlyruntime.cc
compiles all functions as expected. I was not yet able to determine what set of files interferes, but none of the other files compiled alone withruntime.cc
causes the problem. This seems to happens only when a big set of files (>110) is compiled simultaneously.This might have passed unnoticed in V8 upstream because ninja compiles every source file one at a time.
This PR increases the number of shards to divide v8_base into. This increases the number of calls to
cl.exe
but decreases the number of files compiled each time.Fixes: nodejs/v8#4
cc @nodejs/v8 @jasnell
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passesAffected core subsystem(s)
Deps, V8, build, Windows.