Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build thread count #738

Open
wants to merge 3 commits into
base: tundra2
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 36 additions & 25 deletions tools/Windows/BuildDeps.cmd
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
@echo off
echo.

:: Enable the delayed environment variable expansion needed in VSConfig.cmd.
setlocal EnableDelayedExpansion

Expand Down Expand Up @@ -55,8 +54,19 @@ IF NOT EXIST "%DEPS%". mkdir "%DEPS%"
:: User-defined variables
set BUILD_OPENSSL=TRUE
set USE_JOM=TRUE
:: Set to the number of build processes to use
set BUILD_THREADS=%NUMBER_OF_PROCESSORS%
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should probably only do this set if BUILD_THREADS is not defined. Right now you cant configure this outside the script by setting BUILD_THREADS before running the deps build, it will always be overwritten by num of processors on the OS. I guess this works if you want to manually go ahead and edit the script on this line.

Also I would make it TUNDRA_BUILD_THREADS as BUILD_THREADS is quite generic and probably not good to force people to add to their env variables permanently. You know if someone wants to do that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I think in that case I probably should make the same changes to set of BUILD_OPENSSL  and USE_JOM, since those cannot be set from outside the build process.

Should I prefix these alse with  TUNDRA_ ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes thats a good idea if you feel up to it. Maybe even TUNDRA_DEP_X would be better as we already have TUNDRA_DEP_PATH

set JOM_THREADS=-j %BUILD_THREADS%
set USE_BOOST=FALSE

:: Special case: if we build for x64 with vs2008, clear /m:N for certain
:: projects, otherwise they abort with
:: 'error MSB4018: The "ResolveVCProjectOutput" task failed unexpectedly.'
SET MSB_THREADS=/m:%BUILD_THREADS%
IF %GENERATOR%=="Visual Studio 9 2008 Win64" (
SET MSB_THREADS=
)

:: Validate user-defined variables
IF NOT %BUILD_OPENSSL%==FALSE IF NOT %BUILD_OPENSSL%==TRUE (
cecho {0E}BUILD_OPENSSL needs to be either TRUE or FALSE!{# #}{\n}
Expand Down Expand Up @@ -100,6 +110,7 @@ echo - Build OpenSSL, requires Active Perl.
cecho {0D} Build Qt with JOM = %USE_JOM%{# #}{\n}
echo - Use jom.exe instead of nmake.exe to build qmake projects.
echo Default enabled as jom is significantly faster by usin all CPUs.
cecho {0D} Using %BUILD_THREADS% build processes.{# #}{\n}
echo.

:: Print scripts usage information
Expand Down Expand Up @@ -306,7 +317,7 @@ IF NOT EXIST %QT_INSTALL_WEBKIT_DLL_FILENAME%. (

IF %USE_JOM%==TRUE (
cecho {0D}Building %DEBUG_OR_RELEASE% Qt with jom. Please be patient, this will take a while.{# #}{\n}
"%DEPS%\qt\jom\jom.exe"
"%DEPS%\qt\jom\jom.exe" %JOM_THREADS%
) ELSE (
cecho {0D}Building %DEBUG_OR_RELEASE% Qt with nmake. Please be patient, this will take a while.{# #}{\n}
nmake /nologo
Expand Down Expand Up @@ -371,14 +382,14 @@ IF NOT EXIST "%DEPS%\qjson\lib\%BUILD_TYPE%\qjson.dll". (
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
)
cecho {0D}Building %BUILD_TYPE% QJson.{# #}{\n}
MSBuild qjson.sln /p:configuration=%BUILD_TYPE% /clp:ErrorsOnly /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild qjson.sln /p:configuration=%BUILD_TYPE% /clp:ErrorsOnly /nologo /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
)

:: Install the correct build type into qjson/build
cecho {0D}Deploying %BUILD_TYPE% QJson DLL to Tundra bin\ directory.{# #}{\n}
cd "%DEPS%\qjson\"
MSBuild INSTALL.%VCPROJ_FILE_EXT% /p:configuration=%BUILD_TYPE% /clp:ErrorsOnly /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild INSTALL.%VCPROJ_FILE_EXT% /p:configuration=%BUILD_TYPE% /clp:ErrorsOnly /nologo /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
copy /Y "%DEPS%\qjson\build\bin\qjson.dll" "%TUNDRA_BIN%"

Expand Down Expand Up @@ -410,7 +421,7 @@ IF NOT EXIST "%DEPS%\bullet\lib\%BUILD_TYPE%\BulletCollision.lib". (
)

cecho {0D}Building %BUILD_TYPE% Bullet. Please be patient, this will take a while.{# #}{\n}
MSBuild BULLET_PHYSICS.sln /p:configuration=%BUILD_TYPE% /clp:ErrorsOnly /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild BULLET_PHYSICS.sln /p:configuration=%BUILD_TYPE% /clp:ErrorsOnly /nologo /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
) ELSE (
cecho {0D}%BUILD_TYPE% Bullet already built. Skipping.{# #}{\n}
Expand Down Expand Up @@ -442,7 +453,7 @@ IF NOT EXIST "%DEPS%\boost". (
cd "%DEPS%\boost"
cecho {0D}Building Boost. Please be patient, this will take a while.{# #}{\n}
:: Building boost with single core takes ages, so utilize all cores for the build process
call .\b2 -j %NUMBER_OF_PROCESSORS% --without-mpi thread regex stage
call .\b2 -j %BUILD_THREADS% --without-mpi thread regex stage
) ELSE (
::TODO Even if %DEPS%\boost exists, we have no guarantee that boost is built successfully for real
cecho {0D}Boost already built. Skipping.{# #}{\n}
Expand Down Expand Up @@ -482,7 +493,7 @@ IF NOT EXIST "%DEPS%\assimp\bin\%BUILD_TYPE%\assimp%POSTFIX_D%.dll". (
)

cecho {0D}Building %BUILD_TYPE% OpenAssetImport. Please be patient, this will take a while.{# #}{\n}
MSBuild Assimp.sln /p:configuration=%BUILD_TYPE% /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild Assimp.sln /p:configuration=%BUILD_TYPE% /nologo /m:%BUILD_THREADS%
) ELSE (
cecho {0D}%BUILD_TYPE% OpenAssetImport already built. Skipping.{# #}{\n}
)
Expand Down Expand Up @@ -528,7 +539,7 @@ IF NOT EXIST "%DEPS%\kNet\lib\%BUILD_TYPE%\kNet.lib". set BUILD_KNET=TRUE

IF %BUILD_KNET%==TRUE (
cecho {0D}Building %BUILD_TYPE% kNet. Please be patient, this will take a while.{# #}{\n}
MSBuild kNet.sln /p:configuration=%BUILD_TYPE% /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild kNet.sln /p:configuration=%BUILD_TYPE% /nologo /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
) ELSE (
cecho {0D}%BUILD_TYPE% kNet already built and up to date. Skipping.{# #}{\n}
Expand Down Expand Up @@ -556,14 +567,14 @@ IF NOT EXIST "%DEPS%\qtscriptgenerator\generator\release\generator.exe". (
copy /Y "%TOOLS%\Mods\QtScriptGenerator_pp-iterator.h" "%DEPS%\qtscriptgenerator\generator\parser\rpp\pp-iterator.h"
qmake -tp vc
cecho {0D}Building QtScriptGenerator. Please be patient, this will take a while.{# #}{\n}
MSBuild generator.vcxproj /p:configuration=Release /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild generator.vcxproj /p:configuration=Release /nologo /m:%BUILD_THREADS%
) ELSE (
qmake
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
cecho {0D}Building QtScriptGenerator.{# #}{\n}
IF %USE_JOM%==TRUE (
cecho {0D}- Building QtScriptGenerator with jom. Please be patient, this will take a while.{# #}{\n}
"%DEPS%\qt\jom\jom.exe"
"%DEPS%\qt\jom\jom.exe" %JOM_THREADS%
) ELSE (
cecho {0D}- Building QtScriptGenerator with nmake. Please be patient, this will take a while.{# #}{\n}
nmake /nologo
Expand Down Expand Up @@ -603,7 +614,7 @@ IF NOT EXIST "%DEPS%\qtscriptgenerator\plugins\script\qtscript_webkit%POSTFIX_D%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
IF %USE_JOM%==TRUE (
cecho {0D}- Building %DEBUG_OR_RELEASE% QtBindings with jom. Please be patient, this will take a while.{# #}{\n}
"%DEPS%\qt\jom\jom.exe" %DEBUG_OR_RELEASE_LOWERCASE%
"%DEPS%\qt\jom\jom.exe" %JOM_THREADS% %DEBUG_OR_RELEASE_LOWERCASE%
) ELSE (
cecho {0D}- Building %DEBUG_OR_RELEASE% QtBindings with nmake. Please be patient, this will take a while.{# #}{\n}
nmake /nologo %DEBUG_OR_RELEASE_LOWERCASE%
Expand Down Expand Up @@ -698,7 +709,7 @@ IF NOT EXIST OGREDEPS.sln. (
)

cecho {0D}Building %BUILD_TYPE% Ogre dependencies. Please be patient, this will take a while.{# #}{\n}
MSBuild OGREDEPS.sln /p:configuration=%BUILD_TYPE% /nologo /m:%NUMBER_OF_PROCESSORS% /clp:ErrorsOnly
MSBuild OGREDEPS.sln /p:configuration=%BUILD_TYPE% /nologo /m:%BUILD_THREADS% /clp:ErrorsOnly
IF NOT %ERRORLEVEL%==0 GOTO :ERROR

MSBuild INSTALL.%VCPROJ_FILE_EXT% /p:configuration=%BUILD_TYPE% /nologo /clp:ErrorsOnly
Expand Down Expand Up @@ -761,7 +772,7 @@ cmake -G %GENERATOR% -DTBB_HOME=%TBB_HOME% -DOGRE_USE_BOOST:BOOL=%USE_BOOST% -DO
IF NOT %ERRORLEVEL%==0 GOTO :ERROR

cecho {0D}Building %BUILD_TYPE% ogre-safe-nocrashes. Please be patient, this will take a while.{# #}{\n}
MSBuild OGRE.sln /p:configuration=%BUILD_TYPE% /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild OGRE.sln /p:configuration=%BUILD_TYPE% /nologo /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR

cecho {0D}Deploying %BUILD_TYPE% ogre-safe-nocrashes SDK directory.{# #}{\n}
Expand Down Expand Up @@ -806,7 +817,7 @@ IF NOT EXIST SKYX.sln. (
)

cecho {0D}Building %BUILD_TYPE% SkyX. Please be patient, this will take a while.{# #}{\n}
MSBuild SKYX.sln /p:configuration=%BUILD_TYPE% /clp:ErrorsOnly /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild SKYX.sln /p:configuration=%BUILD_TYPE% /clp:ErrorsOnly /nologo /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR

cecho {0D}Deploying %BUILD_TYPE% SkyX DLL to Tundra bin\.{# #}{\n}
Expand All @@ -830,7 +841,7 @@ IF NOT EXIST Hydrax.sln. (
)

cecho {0D}Building %BUILD_TYPE% Hydrax. Please be patient, this will take a while.{# #}{\n}
MSBuild Hydrax.sln /p:configuration=%BUILD_TYPE% /nologo /clp:ErrorsOnly /m:%NUMBER_OF_PROCESSORS%
MSBuild Hydrax.sln /p:configuration=%BUILD_TYPE% /nologo /clp:ErrorsOnly /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR

cecho {0D}Deploying %BUILD_TYPE% Hydrax DLL to Tundra bin\.{# #}{\n}
Expand Down Expand Up @@ -858,7 +869,7 @@ IF NOT EXIST "%DEPS%\qt-solutions\qtpropertybrowser\lib\QtSolutions_PropertyBrow
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
IF %USE_JOM%==TRUE (
cecho {0D}- Building %DEBUG_OR_RELEASE% QtPropertyBrowser with jom{# #}{\n}
"%DEPS%\qt\jom\jom.exe" %DEBUG_OR_RELEASE_LOWERCASE%
"%DEPS%\qt\jom\jom.exe" %JOM_THREADS% %DEBUG_OR_RELEASE_LOWERCASE%
) ELSE (
cecho {0D}- Building %DEBUG_OR_RELEASE% QtPropertyBrowser with nmake{# #}{\n}
nmake /nologo %DEBUG_OR_RELEASE_LOWERCASE%
Expand Down Expand Up @@ -894,7 +905,7 @@ IF NOT EXIST "%DEPS%\ogg\win32\%VS2008_OR_VS2010%\%VS_PLATFORM%\%DEBUG_OR_RELEAS
cd "%DEPS%\ogg\win32\%VS2008_OR_VS2010%"

cecho {0D}Building %DEBUG_OR_RELEASE% Ogg. Please be patient, this will take a while.{# #}{\n}
MSBuild libogg_static.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /clp:ErrorsOnly /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild libogg_static.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /clp:ErrorsOnly /nologo /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
) ELSE (
cecho {0D}%DEBUG_OR_RELEASE% Ogg already built. Skipping.{# #}{\n}
Expand All @@ -909,7 +920,7 @@ IF NOT EXIST "%DEPS%\vorbis". (
IF NOT EXIST "%DEPS%\vorbis\win32\%VS2008_OR_VS2010%\%VS_PLATFORM%\%DEBUG_OR_RELEASE%\libvorbis_static.lib". (
cd "%DEPS%\vorbis\win32\%VS2008_OR_VS2010%"
cecho {0D}Building %DEBUG_OR_RELEASE% Vorbis. Please be patient, this will take a while.{# #}{\n}
MSBuild vorbis_static.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /clp:ErrorsOnly /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild vorbis_static.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /clp:ErrorsOnly /nologo %MSB_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
) ELSE (
cecho {0D}%DEBUG_OR_RELEASE% Vorbis already built. Skipping.{# #}{\n}
Expand All @@ -933,7 +944,7 @@ IF NOT EXIST "%DEPS%\theora\win32\VS2008\%VS_PLATFORM%\%THEORA_BUILD_TYPE%\libth
copy /Y "%TOOLS%\Mods\vs2008-%VS_PLATFORM%-libtheora_static.vcproj" libtheora_static.vcproj
IF NOT %VS_VER%==vs2008 VCUpgrade /nologo /overwrite libtheora_static.vcproj

MSBuild libtheora_static.%VCPROJ_FILE_EXT% /p:configuration=%THEORA_BUILD_TYPE% /p:platform="%VS_PLATFORM%" /clp:ErrorsOnly /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild libtheora_static.%VCPROJ_FILE_EXT% /p:configuration=%THEORA_BUILD_TYPE% /p:platform="%VS_PLATFORM%" /clp:ErrorsOnly /nologo /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
IF NOT EXIST "%DEPS%\theora\win32\VS2008\%VS_PLATFORM%\%THEORA_BUILD_TYPE%". mkdir "%DEPS%\theora\win32\VS2008\%VS_PLATFORM%\%THEORA_BUILD_TYPE%"
copy /Y "%VS_PLATFORM%\%THEORA_BUILD_TYPE%\libtheora_static.lib" "%DEPS%\theora\win32\VS2008\%VS_PLATFORM%\%THEORA_BUILD_TYPE%\libtheora_static.lib"
Expand All @@ -959,7 +970,7 @@ IF NOT EXIST "%DEPS%\speex\lib\%DEBUG_OR_RELEASE%\libspeexdsp.lib". (
copy /Y "%TOOLS%\Mods\%VS2008_OR_VS2010%-libspeex.%VCPROJ_FILE_EXT%" libspeex\libspeex.%VCPROJ_FILE_EXT%
copy /Y "%TOOLS%\Mods\%VS2008_OR_VS2010%-libspeexdsp.%VCPROJ_FILE_EXT%" libspeexdsp\libspeexdsp.%VCPROJ_FILE_EXT%

MSBuild libspeex.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /t:libspeex;libspeexdsp /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild libspeex.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /t:libspeex;libspeexdsp /nologo /m:%BUILD_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR

:: Speex only builds Release or Debug, to make things easier in cmake dublicate Release as RelWithDebInfo.
Expand Down Expand Up @@ -1007,7 +1018,7 @@ IF NOT EXIST "%DEPS%\protobuf\vsprojects\%DEBUG_OR_RELEASE%\libprotobuf.lib". (
echo.

cecho {0D}Building %DEBUG_OR_RELEASE% Google Protobuf. Please be patient, this will take a while.{# #}{\n}
MSBuild protobuf.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /t:libprotobuf;libprotoc;protoc /clp:ErrorsOnly /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild protobuf.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /t:libprotobuf;libprotoc;protoc /clp:ErrorsOnly /nologo %MSB_THREADS%
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
) ELSE (
cecho {0D}%DEBUG_OR_RELEASE% Google Protobuf already built. Skipping.{# #}{\n}
Expand Down Expand Up @@ -1039,7 +1050,7 @@ IF NOT EXIST "%DEPS%\celt\lib\%DEBUG_OR_RELEASE%\libcelt.lib" (
IF NOT %ERRORLEVEL%==0 GOTO :ERROR

cecho {0D}Building %DEBUG_OR_RELEASE% Celt 0.11.1.{# #}{\n}
MSBuild libcelt.%VCPROJ_FILE_EXT% /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /clp:ErrorsOnly /nologo /m:%NUMBER_OF_PROCESSORS%
MSBuild libcelt.%VCPROJ_FILE_EXT% /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /clp:ErrorsOnly /nologo /m:%BUILD_THREADS%

:: Celt only build release or debug, to make things easier in cmake copy Release as RelWithDebInfo
IF "!BUILD_TYPE!"=="%BUILD_TYPE_RELWITHDEBINFO%" (
Expand Down Expand Up @@ -1127,7 +1138,7 @@ IF NOT EXIST "%DEPS%\qxmpp\". (
IF NOT %ERRORLEVEL%==0 GOTO :ERROR
IF %USE_JOM%==TRUE (
cecho {0D}Building %DEBUG_OR_RELEASE% Qxmpp with jom{# #}{\n}
"%DEPS%\qt\jom\jom.exe" sub-src-all-ordered %DEBUG_OR_RELEASE_LOWERCASE%
"%DEPS%\qt\jom\jom.exe" %JOM_THREADS% sub-src-all-ordered %DEBUG_OR_RELEASE_LOWERCASE%
) ELSE (
cecho {0D}Building %DEBUG_OR_RELEASE% Qxmpp with nmake{# #}{\n}
nmake /nologo sub-src-all-ordered %DEBUG_OR_RELEASE_LOWERCASE%
Expand Down Expand Up @@ -1180,7 +1191,7 @@ IF NOT EXIST "%DEPS%\zlib\lib\%DEBUG_OR_RELEASE%\zlibstat.lib". (
cd ..\..
cd contrib\vstudio\%VC_VER%
cecho {0D}Building %DEBUG_OR_RELEASE% zlib %ZLIB_VERSION%{# #}{\n}
MSBuild zlibvc.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /nologo /clp:ErrorsOnly /m:%NUMBER_OF_PROCESSORS%
MSBuild zlibvc.sln /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /nologo /clp:ErrorsOnly /m:%BUILD_THREADS%

cd ..\..\..
copy /Y contrib\vstudio\%VC_VER%\%TARGET_ARCH%\ZlibStat%DEBUG_OR_RELEASE%\zlibstat.lib ..\lib\%DEBUG_OR_RELEASE%
Expand Down Expand Up @@ -1222,7 +1233,7 @@ IF NOT EXIST "%DEPS%\zziplib\lib\zziplib%POSTFIX_D%.lib". (
cecho {0D}Building %DEBUG_OR_RELEASE% zziplib from premade project %TOOLS%\Mods\vs2008-%VS_PLATFORM%-zziplib.vcproj{# #}{\n}
copy /Y "%TOOLS%\Mods\vs2008-%VS_PLATFORM%-zziplib.vcproj" zziplib.vcproj
IF NOT %VS_VER%==vs2008 VCUpgrade /nologo /overwrite zziplib.vcproj
MSBuild zziplib.%VCPROJ_FILE_EXT% /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /nologo /clp:ErrorsOnly /m:%NUMBER_OF_PROCESSORS%
MSBuild zziplib.%VCPROJ_FILE_EXT% /p:configuration=%DEBUG_OR_RELEASE% /p:platform="%VS_PLATFORM%" /nologo /clp:ErrorsOnly /m:%BUILD_THREADS%

:: Copy results to lib/include
copy /Y zziplib%POSTFIX_D%.lib ..\..\lib
Expand Down