Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPI sometimes doesn't exit when exceptions are thrown in Python #269

Open
bburke38 opened this issue Jan 16, 2024 · 0 comments
Open

MPI sometimes doesn't exit when exceptions are thrown in Python #269

bburke38 opened this issue Jan 16, 2024 · 0 comments
Labels
bug Something isn't working wontfix This will not be worked on

Comments

@bburke38
Copy link
Contributor

Just experienced this issue when running an aeroelastic optimization case, which after manually killing the job showed that it hung on the following code in fun3d_interface.py:

# throw a runtime error if adjoint didn't converge sufficiently
if abs(resid) > self.forward_tolerance:
    raise RuntimeError(
        f"Funtofem/Fun3dInterface: fun3d forward flow residual = {resid} > {self.forward_tolerance:.2e}, is too large..."
    )

This is a known issue in mpi4py (here). A simple fix seems to be to modify the execution call with the addition of -m mpi4py:

mpiexec -n ${nprocs} python -m mpi4py script.py 2>&1> output.txt

@bburke38 bburke38 added bug Something isn't working wontfix This will not be worked on labels Jan 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

1 participant