Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unify the vcpkg package set across all gz libs. Remove all packages before every build. #907

Merged
merged 11 commits into from
Apr 13, 2023
Merged

Unify the vcpkg package set across all gz libs. Remove all packages before every build. #907

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0565b9d
Refactor install of vcpkg packages
j-rivero Apr 12, 2023
2d1067e
Add setup_vcpkg_all_dependencies
j-rivero Apr 12, 2023
6dfeae5
Remove previous installed packages in vcpkg
j-rivero Apr 10, 2023
e9b8ce3
Change pkgs installation to use common dependencies
j-rivero Apr 12, 2023
d244844
Use Temp for permissions problems
j-rivero Apr 10, 2023
d44e63e
Remove DEPEN_PKGS references to vcpkg
j-rivero Apr 12, 2023
e4ac2a8
Relocate vcpkg installation at the beginning of the build
j-rivero Apr 12, 2023
f433a46
Create cache directory if it does not exists
j-rivero Apr 12, 2023
05a2dcf
Remove death code
j-rivero Apr 12, 2023
7066a34
Clarify what should be an assertion in checking variable
j-rivero Apr 12, 2023
a1452e9
Remove death code in physics
j-rivero Apr 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions jenkins-scripts/ign_common-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,7 @@ set VCS_DIRECTORY=gz-common
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

set DEPEN_PKGS=assimp ffmpeg freeimage gts tinyxml2
for /f %%i in ('python "%SCRIPT_DIR%\tools\detect_cmake_major_version.py" "%WORKSPACE%\%VCS_DIRECTORY%\CMakeLists.txt"') do set GZ_MAJOR_VERSION=%%i
if %GZ_MAJOR_VERSION% GEQ 5 (
set DEPEN_PKGS=%DEPEN_PKGS% gdal
)

set COLCON_PACKAGE=gz-common
set COLCON_AUTO_MAJOR_VERSION=true
Expand Down
6 changes: 1 addition & 5 deletions jenkins-scripts/ign_fuel-tools-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,7 @@ set SCRIPT_DIR=%~dp0
set VCS_DIRECTORY=gz-fuel-tools
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

:: tinyxml2 from msgs
set DEPEN_PKGS=libyaml libzip tinyxml2 openssl protobuf curl

set COLCON_PACKAGE=gz-fuel_tools
set COLCON_AUTO_MAJOR_VERSION=true

call "%SCRIPT_DIR%\lib\colcon-default-devel-windows.bat"
call "%SCRIPT_DIR%\lib\colcon-default-devel-windows.bat"
7 changes: 0 additions & 7 deletions jenkins-scripts/ign_gazebo-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,6 @@ set SCRIPT_DIR=%~dp0
set VCS_DIRECTORY=gz-sim
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true
:: dlfcn
set DEPEN_PKGS=assimp boost-core ccd cppzmq cuda curl dlfcn-win32 eigen3 fcl ffmpeg freeimage gflags glib gts jsoncpp libyaml libzip ogre ogre2 ogre22 openssl protobuf qt5 qwt tinyxml2 zeromq
for /f %%i in ('python "%SCRIPT_DIR%\tools\detect_cmake_major_version.py" "%WORKSPACE%\%VCS_DIRECTORY%\CMakeLists.txt"') do set GZ_MAJOR_VERSION=%%i
if %GZ_MAJOR_VERSION% GEQ 7 (
set DEPEN_PKGS=%DEPEN_PKGS% gdal
)
:: This needs to be migrated to DSL to get multi-major versions correctly
set COLCON_PACKAGE=gz-sim
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
8 changes: 0 additions & 8 deletions jenkins-scripts/ign_gui-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,6 @@ set SCRIPT_DIR=%~dp0
set VCS_DIRECTORY=gz-gui
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

:: ogre2 from vcpkg-ports
set DEPEN_PKGS=qt5 qt5-winextras qwt protobuf tinyxml2 freeimage ogre ogre22
for /f %%i in ('python "%SCRIPT_DIR%\tools\detect_cmake_major_version.py" "%WORKSPACE%\%VCS_DIRECTORY%\CMakeLists.txt"') do set GZ_MAJOR_VERSION=%%i
if %GZ_MAJOR_VERSION% GEQ 7 (
set DEPEN_PKGS=%DEPEN_PKGS% gdal
)
:: This needs to be migrated to DSL to get multi-major versions correctly
set COLCON_PACKAGE=gz-gui
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
6 changes: 0 additions & 6 deletions jenkins-scripts/ign_launch-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,6 @@ set SCRIPT_DIR=%~dp0
set VCS_DIRECTORY=gz-launch
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

set DEPEN_PKGS=assimp boost-core ccd cppzmq cuda curl dlfcn-win32 eigen3 fcl ffmpeg freeimage gflags glib gts jsoncpp libyaml libzip ogre ogre2 ogre22 openssl protobuf qt5 qwt tinyxml2 zeromq
for /f %%i in ('python "%SCRIPT_DIR%\tools\detect_cmake_major_version.py" "%WORKSPACE%\%VCS_DIRECTORY%\CMakeLists.txt"') do set GZ_MAJOR_VERSION=%%i
if %GZ_MAJOR_VERSION% GEQ 6 (
set DEPEN_PKGS=%DEPEN_PKGS% gdal
)
set COLCON_PACKAGE=gz-launch
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
1 change: 0 additions & 1 deletion jenkins-scripts/ign_math-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ set SCRIPT_DIR=%~dp0
set VCS_DIRECTORY=gz-math
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true
set DEPEN_PKGS=pybind11
set COLCON_PACKAGE=gz-math
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
1 change: 0 additions & 1 deletion jenkins-scripts/ign_msgs-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ set VCS_DIRECTORY=gz-msgs
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

set DEPEN_PKGS=protobuf tinyxml2
set COLCON_PACKAGE=gz-msgs
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
7 changes: 0 additions & 7 deletions jenkins-scripts/ign_physics-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,6 @@ set SCRIPT_DIR=%~dp0
set VCS_DIRECTORY=gz-physics
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

set DEPEN_PKGS=eigen3 tinyxml2
for /f %%i in ('python "%SCRIPT_DIR%\tools\detect_cmake_major_version.py" "%WORKSPACE%\%VCS_DIRECTORY%\CMakeLists.txt"') do set GZ_MAJOR_VERSION=%%i
if %GZ_MAJOR_VERSION% GEQ 6 (
set DEPEN_PKGS=%DEPEN_PKGS% gdal
)
:: This needs to be migrated to DSL to get multi-major versions correctly
set COLCON_PACKAGE=gz-physics
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
1 change: 0 additions & 1 deletion jenkins-scripts/ign_plugin-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ set VCS_DIRECTORY=gz-plugin
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

set DEPEN_PKGS="dlfcn-win32"

set COLCON_PACKAGE=gz-plugin
set COLCON_AUTO_MAJOR_VERSION=true
Expand Down
11 changes: 0 additions & 11 deletions jenkins-scripts/ign_rendering-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,6 @@ set SCRIPT_DIR=%~dp0
set VCS_DIRECTORY=gz-rendering
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

set DEPEN_PKGS=boost-core cuda dlfcn-win32 eigen3 freeimage glib gts ogre ogre2
for /f %%i in ('python "%SCRIPT_DIR%\tools\detect_cmake_major_version.py" "%WORKSPACE%\%VCS_DIRECTORY%\CMakeLists.txt"') do set GZ_MAJOR_VERSION=%%i
if %GZ_MAJOR_VERSION% GEQ 5 (
set DEPEN_PKGS=%DEPEN_PKGS% ogre22
)
if %GZ_MAJOR_VERSION% GEQ 7 (
set DEPEN_PKGS=%DEPEN_PKGS% gdal
)

:: This needs to be migrated to DSL to get multi-major versions correctly
set COLCON_PACKAGE=gz-rendering
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
7 changes: 0 additions & 7 deletions jenkins-scripts/ign_sensors-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,6 @@ set SCRIPT_DIR=%~dp0
set VCS_DIRECTORY=gz-sensors
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

set DEPEN_PKGS=protobuf tinyxml2
for /f %%i in ('python "%SCRIPT_DIR%\tools\detect_cmake_major_version.py" "%WORKSPACE%\%VCS_DIRECTORY%\CMakeLists.txt"') do set GZ_MAJOR_VERSION=%%i
if %GZ_MAJOR_VERSION% GEQ 7 (
set DEPEN_PKGS=%DEPEN_PKGS% gdal
)
:: This needs to be migrated to DSL to get multi-major versions correctly
set COLCON_PACKAGE=gz-sensors
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
1 change: 0 additions & 1 deletion jenkins-scripts/ign_transport-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ set VCS_DIRECTORY=gz-transport
set PLATFORM_TO_BUILD=amd64
set IGN_CLEAN_WORKSPACE=true

set DEPEN_PKGS=cppzmq dlfcn-win32 protobuf sqlite3 tinyxml2 zeromq
set COLCON_PACKAGE=gz-transport
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
7 changes: 2 additions & 5 deletions jenkins-scripts/ign_utils-default-devel-windows-amd64.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,11 @@
@echo on
set SCRIPT_DIR=%~dp0

:: ign-utils shouldn't have external dependencies

set VCS_DIRECTORY=gz-utils
set PLATFORM_TO_BUILD=x86_amd64
set IGN_CLEAN_WORKSPACE=true

:: ign-utils shouldn't have external dependencies
:: set DEPEN_PKGS=

:: This needs to be migrated to DSL to get multi-major versions correctly
set COLCON_PACKAGE=gz-utils
set COLCON_AUTO_MAJOR_VERSION=true

Expand Down
16 changes: 7 additions & 9 deletions jenkins-scripts/lib/colcon-default-devel-windows.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,13 @@
:: - COLCON_AUTO_MAJOR_VERSION (default false): auto detect major version from CMakeLists
:: - COLCON_PACKAGE_EXTRA_CMAKE_ARGS : (optional) CMake arg to inject into colcon
:: - BUILD_TYPE : (default Release) [ Release | Debug ] Build type to use
:: - DEPEN_PKGS : (optional) list of dependencies (separted by spaces)
:: - KEEP_WORKSPACE : (optional) true | false. Clean workspace at the end
:: - ENABLE_TESTS : (optional) true | false. Do not compile and run tests
::
:: Actions
:: - Configure the compiler
:: - Clean and create the WORKSPACE/ws
:: - Download and unzip the DEPEN_PKGS
:: - Install the binary external dependencies
:: - configure, compile and install
:: - run tests

Expand Down Expand Up @@ -84,6 +83,12 @@ echo # BEGIN SECTION: configure the MSVC compiler
call %win_lib% :configure_msvc2019_compiler
echo # END SECTION

:: Prepare a clean vcpkg environment with external dependencies
call %win_lib% :remove_vcpkg_installation || goto :error
echo # BEGIN SECTION: install all vcpkg dependencies
call %win_lib% :setup_vcpkg_all_dependencies || goto :error
echo # END SECTION
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice (maybe a follow-up PR) to print the list of installed packages and their versions for debugging purposes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+100. I'll do it after merging this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#908


echo # BEGIN SECTION: setup workspace
if not defined KEEP_WORKSPACE (
IF exist %LOCAL_WS_BUILD% (
Expand All @@ -107,13 +112,6 @@ if exist %LOCAL_WS_SOFTWARE_DIR% ( rmdir /q /s %LOCAL_WS_SOFTWARE_DIR% )
xcopy %WORKSPACE%\%VCS_DIRECTORY% %LOCAL_WS_SOFTWARE_DIR% /s /e /i > xcopy_vcs_directory.log || goto :error
echo # END SECTION

for %%p in (%DEPEN_PKGS%) do (
call %win_lib% :enable_vcpkg_integration || goto :error
echo # BEGIN SECTION: install external dependency %%p
call %win_lib% :install_vcpkg_package %%p || goto :error
echo # END SECTION
)

echo # BEGIN SECTION: packages in workspace
call %win_lib% :list_workspace_pkgs || goto :error
echo # END SECTION
Expand Down
4 changes: 4 additions & 0 deletions jenkins-scripts/lib/windows_env_vars.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,14 @@ if DEFINED MAKE_JOBS (
set VCPKG_MAX_CONCURRENCY=%MAKE_JOBS%
)
set VCPKG_OSRF_DIR=%VCPKG_DIR%\osrf_vcpkg_ports
set VCPKG_INSTALLED_FILES_DIR=%VCPKG_DIR%\installed
set VCPKG_CMD=%VCPKG_DIR%\vcpkg.exe
set VCPKG_CMAKE_TOOLCHAIN_FILE=%VCPKG_DIR%/scripts/buildsystems/vcpkg.cmake
if NOT DEFINED VCPKG_SNAPSHOT (
:: see https://github.com/microsoft/vcpkg/releases
set VCPKG_SNAPSHOT=2022.02.23
)
:: Set of common gz dependencies expected up to Garden
set VCPKG_DEPENDENCIES_LEGACY=assimp boost-core bullet3 ccd console-bridge cppzmq cuda curl dlfcn-win32 eigen3 fcl ffmpeg freeimage gdal gflags glib gts jsoncpp libyaml libzip ogre ogre2 ogre22 openssl protobuf pybind11 qt5 qt5-winextras qwt sqlite3 tinyxml2 zeromq
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the per-package dependencies have all been dropped in favor of this single list of dependencies. I think that's fine because the packaging for Windows is done using conda-forge and so there's limited utility in using the vcpkg installations to try and smoke out missing dependencies.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the per-package dependencies have all been dropped in favor of this single list of dependencies. I think that's fine because the packaging for Windows is done using conda-forge and so there's limited utility in using the vcpkg installations to try and smoke out missing dependencies.

This is correct.


goto :EOF
52 changes: 45 additions & 7 deletions jenkins-scripts/lib/windows_library.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,8 +268,20 @@ if NOT %VCPKG_HEAD% == %VCPKG_TAG% (
goto :EOF

:: ##################################
:install_vcpkg_package
:: arg1: package to install
:remove_vcpkg_installation
:: remove the installed directory to simulate all packages removal
:: vcpkg cli does not support the operation
set LIB_DIR="%~dp0"
call %LIB_DIR%\windows_env_vars.bat || goto :error
if [%VCPKG_INSTALLED_FILES_DIR%]==[] (
echo VCPKG_INSTALLED_FILES_DIR variable seems empty, this is a bug
goto :error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there situations where we'd expect this directory to be empty, such as with a freshly provisioned host, and if so is the goto :error recoverable or would this halt the build?

)
del /s /f /q %VCPKG_INSTALLED_FILES_DIR%
goto :EOF

:: ##################################
:_prepare_vcpkg_to_install
set LIB_DIR=%~dp0
call %LIB_DIR%\windows_env_vars.bat || goto :error
call %win_lib% :check_vcpkg_snapshot || goto :error
Expand All @@ -278,14 +290,36 @@ pushd .
cd %VCPKG_OSRF_DIR%
git pull origin master || goto :error
popd
goto :EOF

:: ##################################
:_install_and_upgrade_vcpkg_package
:: arg1: package to install
if [%1] == [] (
echo "_install_and_upgrade_vcpkg_package called with no argument"
goto :error
)
:: workaround on permissions problems for default VCPKG_DEFAULT_BINARY_CACHE
set VCPKG_DEFAULT_BINARY_CACHE=C:\Windows\Temp\vcpkg
if not exist %VCPKG_DEFAULT_BINARY_CACHE% mkdir %VCPKG_DEFAULT_BINARY_CACHE%
%VCPKG_CMD% install --recurse "%1" --overlay-ports="%VCPKG_OSRF_DIR%"
:: vcpkg does not upgrade installed packages using the install command
:: since most of the packages are coming from a frozen snapshot, it is
:: not a problem. However upgrading is needed for the osrf port overlay
%VCPKG_CMD% upgrade "%1" --no-dry-run --overlay-ports="%VCPKG_OSRF_DIR%"
goto :EOF

:: ##################################
:install_vcpkg_package
:: arg1: package to install
if [%1] == [] (
echo "install_vcpkg_package called with no argument"
goto :error
)
call %win_lib% :_prepare_vcpkg_to_install|| goto :error
call %win_lib% :_install_and_upgrade_vcpkg_package "%1" || goto :error
goto :EOF

:: ##################################
:remove_vcpkg_package
:: arg1: package to install
Expand All @@ -305,11 +339,15 @@ goto :EOF
%VCPKG_CMD% integrate remove || goto :error
goto :EOF


:: copy port to the official tree
xcopy %VCPKG_OSRF_DIR%\%PKG% %PORT_DIR% /s /i /e || goto :error

call %win_lib% :install_vcpkg_package %1 || goto :error
:: ##################################
:setup_vcpkg_all_dependencies
set LIB_DIR=%~dp0
call %LIB_DIR%\windows_env_vars.bat || goto :error
call %win_lib% :enable_vcpkg_integration || goto :error
call %win_lib% :_prepare_vcpkg_to_install|| goto :error
for %%p in (%VCPKG_DEPENDENCIES_LEGACY%) do (
call %win_lib% :_install_and_upgrade_vcpkg_package %%p || goto :error
)
goto :EOF

:: ##################################
Expand Down