Too many libgfortrans

[ChrisBarker-NOAA]

My situation:

I had a environment up and running, all was well

I needed fortran

I installed the gcc package to get gfortran.

all was well.

BUT ......

It turns out that that gave me a gcc as well, that kinda contaminated the rest of my system, so I had to remove it.

But now numpy is broken:

---> 14 from . import multiarray
     15 for envkey in env_added:
     16     del os.environ[envkey]

ImportError: dlopen(/Users/chris.barker/miniconda2/lib/python2.7/site-packages/numpy/core/multiarray.so, 2): Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/chris.barker/miniconda2/lib/libopenblasp-r0.2.18.dylib
  Reason: image not found

So I am making an educated guess that numpy depends on openblas, which requires libgfortran, and apparently installs it itself.

gcc also install the same lib. -- so when I remove gcc, I break gfortran.

Ideally, conda would do some kind of reference counting or something, so that it wouldn't remove a file that another package is also using -- but that's a major feature that may never come.

So: shouldn't libgfortran (and similar) be a package by itself, and then gcc and openblas, and who knows what can depend on it?

Also nice for gfortran to be separate from gcc as well...

[ChrisBarker-NOAA]

OK -- turns out there IS a libgfortran package!

But does gcc still install/remove it's own?

If this has all been fixed, feel free to close.

[jakirkham]

TL;DR this could be better, but will require substantial improvements in conda-build. Not sure what can be done at conda-forge until that happens.

Reso: To "fix" the issue just reinstall libgfortran and you should be good to go. Advice would be to keep gcc in a separate environment if you need it often and just leave it installed in that environment.

...and into the weeds.

It turns out that that gave me a gcc as well, that kinda contaminated the rest of my system, so I had to remove it.

Was it not possible to just deactivate that environment?

So I am making an educated guess that numpy depends on openblas, which requires libgfortran, and apparently installs it itself.

Correct.

So: shouldn't libgfortran (and similar) be a package by itself

It is. (though it seems you already figured this out)

...openblas, and who knows what can depend on it?

Generally they do depend on libgfortran.

Also nice for gfortran to be separate from gcc as well...

👍 I think I tried this before and it became a bit painful, but it has been awhile. May try it again as this would make our lives much easier when building multilingual packages. I think it should be doable.

...gcc...and who knows what can depend on it?

This is a bit complicated. gcc builds all of these libraries normally as they are part of gcc.

So what we really want would be a build process that builds gcc that divides it into smaller packages. Then it can define how they are related as part of that build so we can construct the dependency relationships.

Maybe something like issue ( conda/conda-build#1338 ) is relevant to this problem. Haven't followed it that closely though so can't give too much more guidance.

It might be possible to do some really hacky splitting of our own. Not sure how well that would go or whether people would like it.

Also there is this question of how to version all of the pieces.

Ideally, conda would do some kind of reference counting or something, so that it wouldn't remove a file that another package is also using -- but that's a major feature that may never come.

That's a good idea and probably should be suggested.

However, there is more than meets the eye. What if two different files share the same name and location? How then do we install them? Also who "wins" the filename? This has come up before, but unfortunately the refs aren't coming to mind ATM.

[ChrisBarker-NOAA]

I"ve got the fix working -- and I think in this case, the real problem is that the root environment is fully available when you are in a conda environment -- conda issue -- I brought it up on the list.

As for shared files, conda could do a checksum to make sure files with the same name are really the same -- and it probably should otherwise you can fairly easily break a package by overwriting some key part.

It could even be pedantic and not allow a package to install a file that's already there.

If that was the case, how would we deal with libgfortran?

1. if you want a lib to be available to other packages, then it should be in a package itself. WE can't really do that unless we break up the gcc package, but that would be a worthy goal, though a pain to do.

2. if you have something that needs a lib, but that lib is not available as a package, and it's not easy to make a package for it i.e. openblas needs libgfortran -- then the lib should be statically linked, or delivered alongside, or in a custom location, so other packages won't accidentally find it.

Anyway, that's my take.

AS for splitting up gcc:

Couldn't a really kludgy way be to have a recipe that builds and installs the whole mess, then goes in and deletes everything you don't want? Ugly and expensive, but do-able.

-CHB

[jakirkham]

I think we are down to one libgfortran and even upgraded the Fortran version somewhat recently. Not to mention have a toolchain that lives within the conda-build system.

Are there other concerns here?