Openmpi CC_FOR_BUILD and friends also target host now

[zklaus]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

For hdf5, we need to build a build time tool with the mpifort and mpicc. Up to now, this works nicely by setting CC_FOR_BUILD/FC_FOR_BUILD to $BUILD_PREFIX/bin/mpicc (mpifort, respectively).
However, with openmpi 5.0.5, these compilers also target the host, i.e. just like the regular $PREFIX/bin/mpicc, they produce osx-arm64 binaries in cross compilation mode, which cannot link with the right library (see here) or fail to execute due to bad cpu type.

I am not 100% sure which change is responsible for this or which version is the last one that worked, but I can say that things worked with openmpi 5.0.3-h324cb95_104 and that I suspect the culprit to be this line.

Installed packages

-

Environment info

-

[zklaus]

I forgot to say: this all comes up in conda-forge/hdf5-feedstock#231, an effort to update hdf5.

[dalcinl]

IIRC, the --disable-wrapper-runpath option you pointed to as possible culprit should not affect macOS. Something else must be going on.

[minrk]

I believe it's the newish $OPAL_PREFIX default. hdf5 does cross-compilation a lot different from any other recipe I've seen, but unsetting $OPAL_PREFIX is probably right since the recipe handles that itself in a different way.

I don't understand how cross-compilation is working at all in hdf5, because it sets CC=$PREFIX/bin/mpicc, which shouldn't be able run unless emulation is available because that's a host binary. I don't see what is invoking the opal_wrapper, since it must come before $PREFIX/bin/mpicc ever gets called.

[minrk]

I'm not sure if there's a better way to support CC_FOR_BUILD here, but unsetting all the OPAL, OMPI_ env works (I updated conda-forge/hdf5-feedstock#231 to do this).

We could provide our own wrapper that does this, if it makes sense. I'm not sure it's worth it.

[zklaus]

Thanks for the help with hdf5! This really scratches the itch. So I was about to suggest to close this issue, but then I was thinking that our docs suggest that a compiler version that targets the build platform should be available via XX_FOR_BUILD.

Do you agree that this is currently not so for openmpi?

Do you think this should be changed or noted?

[minrk]

Do you agree that this is currently not so for openmpi?

Sort of - mpi provides compiler wrappers, but not compilers per se. Setting CC=mpicc is a shortcut to setting CFLAGS=$(pkg-config --cflags ompi-c), and LDFLAGS=$(pkg-config --libs ompi-c). You could keep the default CC_FOR_BUILD and target build mpi that way.

Do you think this should be changed or noted?

That's what I was thinking with the wrapper - we could define something like MPICC_FOR_BUILD, etc. since we are now setting the variables for mpicc to work correctly by default for cross compilation (hdf5 is the first recipe I've seen that's not strictly improved by the updated env variables in the openmpi activation).

At the very least, it should be mentioned that mpicc targets the host environment by default, and:

(
  export OPAL_PREFIX=$BUILD_PREFIX
  unset ${!OMPI_@}
  mpicc ...
)

will target the build environment. But maybe it's worth it to implement a wrapper to make it standard/easier. hdf5 may end up being the only consumer, though.

This would get simpler if we switched to script wrappers like mpich (#161), since none of these environment steps would be needed.