Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add the possitility to use other build generators than ninja inside the ci #3276

Merged
merged 13 commits into from
Sep 12, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,11 @@ on:
type: string
LOCAL_BASELINE_DIR:
required: false
type: string
type: string
BUILD_GENERATOR:
required: false
type: string
default: '--ninja'
secrets:
GOOGLE_CLOUD_GCP:
required: false
Expand Down Expand Up @@ -140,6 +144,10 @@ jobs:
script_args+=(--nproc ${NPROC})
fi

if [[ -n "${{ inputs.BUILD_GENERATOR }}" ]]; then
script_args+=(${{ inputs.BUILD_GENERATOR }})
fi

docker_args+=(${{ inputs.DOCKER_RUN_ARGS }})

COMMIT=${{ github.event.pull_request.head.sha }}
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/ci_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,7 @@ jobs:
- name: Ubuntu CUDA debug (20.04, clang 10.0.0 + gcc 9.4.0, open-mpi 4.0.3, cuda-11.8.89)
BUILD_AND_TEST_CLI_ARGS: "--build-exe-only --no-install-schema"
CMAKE_BUILD_TYPE: Debug
BUILD_GENERATOR: "--ninja"
DOCKER_REPOSITORY: geosx/ubuntu20.04-clang10.0.0-cuda11.8.89
ENABLE_HYPRE_DEVICE: CUDA
ENABLE_HYPRE: ON
Expand All @@ -288,6 +289,7 @@ jobs:
- name: Ubuntu CUDA (20.04, clang 10.0.0 + gcc 9.4.0, open-mpi 4.0.3, cuda-11.8.89)
BUILD_AND_TEST_CLI_ARGS: "--no-install-schema"
CMAKE_BUILD_TYPE: Release
BUILD_GENERATOR: "--ninja"
DOCKER_REPOSITORY: geosx/ubuntu20.04-clang10.0.0-cuda11.8.89
ENABLE_HYPRE_DEVICE: CUDA
ENABLE_HYPRE: ON
Expand All @@ -297,11 +299,11 @@ jobs:
DOCKER_RUN_ARGS: "--cpus=8 --memory=256g --runtime=nvidia --gpus all -v /etc/pki/ca-trust/source/anchors/:/usr/local/share/ca-certificates/llnl:ro"
DOCKER_CERTS_DIR: "/usr/local/share/ca-certificates"
DOCKER_CERTS_UPDATE_COMMAND: "update-ca-certificates"

# compiler error in ElasticFirstOrderWaveEquationSEMKernel::StressComputation::launch in call to FE_TYPE::computeFirstOrderStiffnessTermX
# - name: Rockylinux (8, clang 17.0.6, cuda 12.5)
# BUILD_AND_TEST_CLI_ARGS: "--no-run-unit-tests --no-install-schema"
# CMAKE_BUILD_TYPE: Release
# BUILD_GENERATOR: "--ninja"
# DOCKER_REPOSITORY: geosx/rockylinux8-clang17-cuda12.5
# RUNS_ON: streak2
# NPROC: 2
Expand All @@ -313,6 +315,7 @@ jobs:
# - name: Rockylinux (8, gcc 8.5, cuda 12.5)
# BUILD_AND_TEST_CLI_ARGS: "--no-run-unit-tests --no-install-schema"
# CMAKE_BUILD_TYPE: Release
# BUILD_GENERATOR: "--ninja"
# DOCKER_REPOSITORY: geosx/rockylinux8-gcc8-cuda12.5
# RUNS_ON: streak2
# NPROC: 2
Expand All @@ -326,6 +329,7 @@ jobs:
with:
BUILD_AND_TEST_CLI_ARGS: ${{ matrix.BUILD_AND_TEST_CLI_ARGS }}
CMAKE_BUILD_TYPE: ${{ matrix.CMAKE_BUILD_TYPE }}
BUILD_GENERATOR: ${{ matrix.BUILD_GENERATOR }}
DOCKER_CERTS_DIR: ${{ matrix.DOCKER_CERTS_DIR }}
DOCKER_CERTS_UPDATE_COMMAND: ${{ matrix.DOCKER_CERTS_UPDATE_COMMAND }}
DOCKER_IMAGE_TAG: ${{ needs.is_not_draft_pull_request.outputs.DOCKER_IMAGE_TAG }}
Expand Down
37 changes: 26 additions & 11 deletions scripts/ci_build_and_test_in_container.sh
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,18 @@ Usage: $0
run a code build and test.
--data-basename output.tar.gz
If some data needs to be extracted from the build, the argument will define the tarball. Has to be a `tar.gz`.
--eclipse
Use "Eclipse CDT4 - Unix Makefiles" as build system generator.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would we include this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

--exchange-dir /path/to/exchange
Folder to share data with outside of the container.
--host-config host-config/my_config.cmake
The host-config. Path is relative to the root of the repository.
--install-dir-basename GEOS-e42ffc1
GEOS installation basename.
--makefile
Use "Unix Makefiles" as build system generator.
--ninja
Use "Ninja" as build system generator.
--no-install-schema
Do not install the xsd schema.
--no-run-unit-tests
Expand All @@ -58,6 +64,8 @@ Usage: $0
Basename of the json credentials file to connect to the sccache cloud cache.
--test-code-style
--test-documentation
--xcode
Use "Xcode" as build system generator.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also...why include this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was for genericity purpose. The script can be executed outside the ci by a local docker run call so I was guessing that we may want to use one of the generators supported by the project.
But I have not tested it so ok to remove it and to focus on the ci only.

-h | --help
EOF
exit 1
Expand All @@ -68,10 +76,11 @@ exit 1
or_die cd $(dirname $0)/..

# Parsing using getopt
args=$(or_die getopt -a -o h --long build-exe-only,cmake-build-type:,code-coverage,data-basename:,exchange-dir:,host-config:,install-dir-basename:,no-install-schema,no-run-unit-tests,nproc:,repository:,run-integrated-tests,sccache-credentials:,test-code-style,test-documentation,help -- "$@")
args=$(or_die getopt -a -o h --long build-exe-only,cmake-build-type:,code-coverage,data-basename:,eclipse,exchange-dir:,host-config:,install-dir-basename:,makefile,ninja,no-install-schema,no-run-unit-tests,nproc:,repository:,run-integrated-tests,sccache-credentials:,test-code-style,test-documentation,xcode,help -- "$@")

# Variables with default values
BUILD_EXE_ONLY=false
BUILD_GENERATOR=""
GEOS_INSTALL_SCHEMA=true
HOST_CONFIG="host-configs/environment.cmake"
RUN_UNIT_TESTS=true
Expand All @@ -91,6 +100,9 @@ do
RUN_UNIT_TESTS=false
shift;;
--cmake-build-type) CMAKE_BUILD_TYPE=$2; shift 2;;
--eclipse|--ninja|--xcode)
BUILD_GENERATOR=$1;
shift;;
--data-basename)
DATA_BASENAME=$2
DATA_BASENAME_WE=${DATA_BASENAME%%.*}
Expand All @@ -104,6 +116,7 @@ do
--exchange-dir) DATA_EXCHANGE_DIR=$2; shift 2;;
--host-config) HOST_CONFIG=$2; shift 2;;
--install-dir-basename) GEOS_DIR=${GEOSX_TPL_DIR}/../$2; shift 2;;
--makefile) BUILD_GENERATOR=""; shift;;
--no-install-schema) GEOS_INSTALL_SCHEMA=false; shift;;
--no-run-unit-tests) RUN_UNIT_TESTS=false; shift;;
--nproc) NPROC=$2; shift 2;;
Expand Down Expand Up @@ -211,7 +224,7 @@ fi
# The option `--oversubscribe` tells OpenMPI to allow more MPI ranks than the node has cores.
# This is needed because our unit test `blt_mpi_smoke` is run in parallel with _hard coded_ 4 ranks.
# While some of our ci nodes may have less cores available.
#
#
# In case we have more powerful nodes, consider removing `--oversubscribe` and use `--use-hwthread-cpus` instead.
# This will tells OpenMPI to discover the number of hardware threads on the node,
# and use that as the number of slots available. (There is a distinction between threads and cores).
Expand All @@ -221,7 +234,7 @@ or_die python3 scripts/config-build.py \
-bt ${CMAKE_BUILD_TYPE} \
-bp ${GEOS_BUILD_DIR} \
-ip ${GEOS_DIR} \
--ninja \
${BUILD_GENERATOR} \
-DBLT_MPI_COMMAND_APPEND='"--allow-run-as-root;--oversubscribe"' \
-DGEOS_INSTALL_SCHEMA=${GEOS_INSTALL_SCHEMA} \
-DENABLE_COVERAGE=$([[ "${CODE_COVERAGE}" = true ]] && echo 1 || echo 0) \
Expand All @@ -245,10 +258,10 @@ fi

# Performing the requested build.
if [[ "${BUILD_EXE_ONLY}" = true ]]; then
or_die ninja -j $NPROC geosx
or_die cmake --build . -j $NPROC --target geosx
else
or_die ninja -j $NPROC
or_die ninja install
or_die cmake --build . -j $NPROC
or_die cmake --install .

if [[ ! -z "${DATA_BASENAME_WE}" ]]; then
# Here we pack the installation.
Expand All @@ -269,7 +282,7 @@ if [[ ! -z "${SCCACHE_CREDS}" ]]; then
fi

if [[ "${CODE_COVERAGE}" = true ]]; then
or_die ninja coreComponents_coverage
or_die cmake --build . --target coreComponents_coverage
cp -r ${GEOS_BUILD_DIR}/coreComponents_coverage.info.cleaned ${GEOS_SRC_DIR}/geos_coverage.info.cleaned
fi

Expand All @@ -284,9 +297,11 @@ fi

if [[ "${RUN_INTEGRATED_TESTS}" = true ]]; then
# We split the process in two steps. First installing the environment, then running the tests.
or_die ninja ats_environment

# The tests are not run using ninja (`ninja --verbose ats_run`) because it swallows the output while all the simulations are running.
or_die cmake --build . --target ats_environment

# The tests are not run using cmake (`cmake --build . --verbose --target ats_run`)
# because with ninja it swallows the output while all the
# simulations are running.
# We directly use the script instead...
echo "Available baselines:"
ls -lR /tmp/geos/baselines
Expand Down Expand Up @@ -321,7 +336,7 @@ if [[ "${RUN_INTEGRATED_TESTS}" = true ]]; then
fi

# Cleaning the build directory.
or_die ninja clean
or_die cmake --build . --target clean


# Clean the repository
Expand Down