4.0.2-h846660c_100 vs. 4.0.2-external_0

[jan-janssen]

Comment:

I try to build a package which uses the mpic++ command to compile:
conda-forge/n2p2-feedstock#15

For Python 3.10 everything works fine but for Python < 3.10 the builds fail. The reason is that rather than installing mpich=4.0.2=h846660c_100 the package mpich=4.0.2=external_0 is installed which does not provide the corresponding executable.

[dalcinl]

Well, it actually makes sense, despite of of the fact that the external mpicc will not forward to the conda-forge C/C++/Fortran compilers.

IMHO, the solution is to create a special mpich-mpicc-=external that sets the MPICH_{CC|CXX|FC} environment variables pointing to the conda-forge C/C++/Fortran compilers upon activation after installation within an environment. I don't know all the details, but I believe it is doable.

Sorry, I totally misunderstood the issue.

[jan-janssen]

I ran a couple of more tests and I had the feeling it is more or less random when the external mai package is chosen. Can I fix this in the conda_build_config.yaml file ? I am sorry I am not exactly sure where to set mpich-mpicc-_=external_

[isuruf]

This seems like a mamba bug. cc @wolfv

[wolfv]

I just had a look at the metadata for the two packages.
The non-external packages have additional constraints on libgcc & libgfortran:

    "mpich-4.0.2-external_0.tar.bz2": {
      "build": "external_0",
      "build_number": 0,
      "depends": [
        "mpi 1.0 mpich"
      ],
      "license": "LicenseRef-MPICH",
      "license_family": "Other",
      "md5": "ecd2c3a7ecea016c68b298f87dfb949c",
      "name": "mpich",
      "sha256": "cdc0f95b93fba729a46b57332544ce99f74498cad55406c61e0deb98b842f5d4",
      "size": 7914,
      "subdir": "linux-64",
      "timestamp": 1649456754281,
      "track_features": "mpich_external",
      "version": "4.0.2"
    },
    "mpich-4.0.2-h846660c_100.tar.bz2": {
      "build": "h846660c_100",
      "build_number": 100,
      "depends": [
        "libgcc-ng >=10.3.0",
        "libgfortran-ng",
        "libgfortran5 >=10.3.0",
        "libstdcxx-ng >=10.3.0",
        "mpi 1.0 mpich"
      ],
      "license": "LicenseRef-MPICH",
      "license_family": "Other",
      "md5": "36a36fe04b932d4b327e7e81c5c43696",
      "name": "mpich",
      "sha256": "8df20ce96df5a019a2e60cd9cb04105c18a12ee5557d7e008e32a339a2672980",
      "size": 6264180,
      "subdir": "linux-64",
      "timestamp": 1649457281832,
      "version": "4.0.2"
    },

Could it be possible that those conflict somehow with the rest of the recipe?

[wolfv]

Maybe the key is the additional libgfortran5 dependency -- I see the CI resolves to libgfortran4.

[wolfv]

The blas variant with *netlib is depending on libgfortran4 which pulls in the "wrong" mpich dependency.

I don't know why this recipe depends on the netlib implementation. Should we rebuild that package?

We could also change the sorting that is applied with track features. IIRC currently, the sorting would look like

mpi 4.0.2
mpi 4.0.2 external (track feature)
mpi 3.0
mpi 3.0   external (track feature)
...

We could instead use

mpi 4.0.2 
mpi 3.0
mpi 4.0.2 external (track feature)
mpi 3.0   external (track feature)

So that the algorithm would try older non-external versions first...

libblas 3.8.0 10_h6e990d7_netlib
--------------------------------
file name   : libblas-3.8.0-10_h6e990d7_netlib.tar.bz2
name        : libblas
version     : 3.8.0
build       : 10_h6e990d7_netlib
build number: 10
size        : 175 KB
license     : BSD 3-Clause
subdir      : linux-64
url         : https://conda.anaconda.org/conda-forge/linux-64/libblas-3.8.0-10_h6e990d7_netlib.tar.bz2
md5         : 9b6100f5e76dcc1dfb59ee07a61569db
track features: blas_netlib
timestamp   : 2020-10-11 17:00:49 UTC
dependencies:
  - libgfortran-ng
  - libgcc-ng >=7.5.0
  - libgfortran4 >=7.5.0

[isuruf]

AFAIK, the second sorting option is the one that conda implements.

[wolfv]

in that case, mamba would resolve to mpich 3.3.2 which seems to be pretty old and just as "surprising".

Just wondering if it is possible to rebuild the netlib blas for libgfortran5?

I'll take a note of the sorting order of variants with track features.

[jan-janssen]

Could it be possible that those conflict somehow with the rest of the recipe?

Currently the requirements are:

    - python
    - cython
    - {{ mpi }}
    - libblas
    - libcblas
    - gsl
    - eigen

So I do not directly see the reason why it aims for libgfortran4.

The other thing I noticed is that when I restart the build sometimes some of the failing builds work but it is rather random.

Is there anything I can do to enforce the choice of the internal mpich?

[wolfv]

@jan-janssen the *netlib pin comes from the .ci_support/linux... yaml files and is applied to the libblas and libcblas packages (which in turn require libgfortran4).

[jan-janssen]

Can I enforce the use of mpich internal in conda_build_config.yaml?

[isuruf]

(which in turn require libgfortran4).

Not true. libblas and libcblas has two packages. one built with libgfortran4 and one with libgfortran5.

[jan-janssen]

@isuruf I saw you rerendered the recipe and now it works. Does this mean the issue is resolved? I do not really understand what exactly changed.

[jan-janssen]

Ok, it seems to be the difference between netlike 3.8 and 3.9

[isuruf]

Yes, netlib 3.8 had two builds (one for libgfortran4 and one for libgfortran5). mamba was getting confused by that.
changing to 3.9 which has only one build makes mamba less confused.