Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Rocky build automation via PNNL GitLab #173

Closed
wants to merge 127 commits into from
Closed

Conversation

jacksavage
Copy link
Collaborator

  • updates install_gridpack_deps.sh and install_gridpack.sh.
  • adds definition for a gitlab build pipeline via .gitlab-ci.yml. this contains jobs that install gridpack and run gridpack tests. there is a third job that builds a container image used as the environment for those jobs.
  • adds dockerfile, defining the container image.

abhyshr and others added 30 commits August 28, 2023 11:22
- Split the script into two (one for installing dependencies and other for GridPACK)
- Ignore boost includes and boost libraries in GridPACK.cmake.in. Otherwise python
  installation fails.
- Update GridPACK installation script with addition CMake flags for Boost.
- Remove directory changes and assume current directory
- Use local variables in functions
- Capture logs
Also disable the ubuntu jobs for now
This always adds the job to the pipeline and blocks - ignoring the intent of it being an optional dependency downstream. I suppose if we want to rebuild the container, you could always go back to the last pipeline where it ran and rerun. This should be the last place that any files relevant to the container build changed.
This should make the pipeline summary icons in the GitLab UI more useful.
@abhyshr
Copy link
Collaborator

abhyshr commented Jan 23, 2024

@jacksavage : What is the status of this work?

@jacksavage
Copy link
Collaborator Author

@jacksavage : What is the status of this work?

hi @abhyshr, I was able to get the ubuntu build working and the tests are able to run (as of 31d1fb2). a few tests are failing, maybe the ones that are expected to.

right now, I'm trying to get the rocky linux (RHEL-like) build to work. I ran into an issue with the PNNL proxy and needed to request a url for the package repo list be unblocked. now I'm troubleshooting an issue with the Global Arrays install - "Could not determine the C compiler wrapped by MPI".

@jacksavage
Copy link
Collaborator Author

jacksavage commented Jan 24, 2024

@wperkins, thanks for the help in regard to loading the module for mpi4py. on the rocky linux build, I'm now running into in error with the gridpack installation when running

cmake \
  -D GA_DIR:STRING=/gridpack-dependencies/ga/install_for_gridpack \
  -D BOOST_ROOT:STRING=/gridpack-dependencies/boost/install_for_gridpack \
  -D Boost_DIR:STRING=/gridpack-dependencies/boost/install_for_gridpack/lib/cmake/Boost \
  -D Boost_LIBRARIES:STRING=/gridpack-dependencies/boost/install_for_gridpack/lib \
  -D Boost_INCLUDE_DIRS:STRING=/gridpack-dependencies/boost/install_for_gridpack/include \
  -D PETSC_DIR:PATH=/gridpack-dependencies/petsc/install_for_gridpack \
  -D MPI_CXX_COMPILER:STRING=mpicxx \
  -D MPI_C_COMPILER:STRING=mpicc \
  -D MPIEXEC:STRING=mpiexec \
  -D GRIDPACK_TEST_TIMEOUT:STRING=30 \
  -D CMAKE_INSTALL_PREFIX:PATH=/build/src/install \
  -D CMAKE_BUILD_TYPE:STRING=Debug \
  -D BUILD_SHARED_LIBS=YES \
  -D Boost_NO_SYSTEM_PATHS:BOOL=TRUE \
  -D Boost_NO_BOOST_CMAKE:BOOL=TRUE \
  -D MPIEXEC_MAX_NUMPROCS:STRING=2 \
  ..

the error is

CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Boost (missing: mpi) (found suitable version "1.78.0",
  minimum required is "1.53")
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindBoost.cmake:2344 (find_package_handle_standard_args)
  CMakeLists.txt:270 (find_package)

any ideas about this?

@wperkins
Copy link
Member

wperkins commented Jan 24, 2024

[...]

CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Boost (missing: mpi) (found suitable version "1.78.0",
  minimum required is "1.53")
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindBoost.cmake:2344 (find_package_handle_standard_args)
  CMakeLists.txt:270 (find_package)

any ideas about this?

Are you using the system boost package? If so, did you install the boost-openmpi package (https://rockylinux.pkgs.org/8/rockylinux-powertools-x86_64/boost-openmpi-1.66.0-13.el8.x86_64.rpm.html)?

If you are building Boost from source, do not install any system Boost packages, or add -D Boost_NO_SYSTEM_PATHS=TRUE to GridPACK configuration.

@wperkins
Copy link
Member

hi @abhyshr, I was able to get the ubuntu build working and the tests are able to run (as of 31d1fb2). a few tests are failing, maybe the ones that are expected to.

@jacksavage, it looks like the tests that fail (wind DSA) are just timing out. You can increase the time out time with the GridPACK configuration option -D GRIDPACK_TEST_TIMEOUT (in seconds). Or, if they take more time that you can stand, you might just exclude them (ctest -E 'wind.*dsa').

Just out of curiosity, are we going to keep hiding the CI test results on PNNL's Gitlab, or is there a plan to expose the results/logs on Github?

@jacksavage jacksavage changed the title Add build automation via PNNL GitLab Add Rocky build automation via PNNL GitLab Jan 25, 2024
@jacksavage
Copy link
Collaborator Author

@wperkins, looks like I was able to get all tests passing with a 120 second timeout. thanks.

in regard to exposing test results to GitHub, I haven't come across a simple way to do that. the only thing that seems to be supported by the GitLab integration is reporting the build status and providing a link to results. did we choose to build on PNNL GitLab for the eventual access to HPC? if not, I suppose moving the build to GitHub would be worth exploring - if we want results here and public.

@jacksavage
Copy link
Collaborator Author

Closing this PR because #195 contains everything needed to continue work adding Rocky Linux support.

@jacksavage jacksavage closed this Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants