-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MPI: CMake 3.18+ support #1114
Comments
@ptheywood's research from Slack chat
|
Unable to reproduce this on x86_64 ubuntu machines, which dont' seem to require the flags being passed. We probably just want to warn on <= 3.20.1 that it might error as a dev warning, as its not a universal mpi + old cmake error. |
I've just confirmed on Bede that CMake 3.20.0 fails to link with the error above, while 3.20.1 does work (when MPI is enabled and the MPI installation requires some extra env variables passing to the host linker, i.e. the OpenMPI install on Bede). So adding a |
Add optional support for distributing simulations within an ensemble across multiple machines via MPI * CMake: Add FLAMEGPU_ENABLE_MPI option. * Breaking: CUDAEnsemble::getLogs() now returns a map They key corresponds to the index of the corresponding RunPlan within the input RunPlanVector. * BugFix: Ensemble log already exist exception message contained bad filepath. * BugFix: Replace occurences of throw with THROW * MPI tests in new target tests_mpi * Warn about MPI link failures with CMake < 3.20.1 * Warn at CMake Configure when mpich forces -flto. * CI: Add MPI CI * Assigns GPUs to MPI ranks per node, allowing more flexible MPI configurations MPI ensembles can use multiple mpi ranks per node, evenly(ish) distributing GPUs across the ranks per shared memory system. If more MPI ranks are used on a node than GPUs, additional ranks will do nothing and a warning is reported. I.e. any number of mpi ranks can be launched, but only the sensible amount will be used. If the user specifies device indices, they will be load balanced, otherwise all visible devices within the node will be balanced. Only one rank per node sends the device string back for telemetry, others send back an empty string (while the assembleGPUsString method is expecting a message from each rank in the world. If no valid CUDAdevices are provided, an exception is raised Device allocation is implemented in a static method so it can be tested programmatically, without launching the test N times with different MPI configurations. Closes #1114 --------- Co-authored-by: Peter Heywood <p.heywood@sheffield.ac.uk>
Device link was failing building
tests_mpi
target ofdistributed_ensemble
branch on Bede with CMake 3.18, but fixed when using CMake 3.22.May need to consider updating minimum required CMake version.
CMake 3.18
CMake 3.22
The text was updated successfully, but these errors were encountered: