Overlinkage warnings raised for python and numpy

[jakirkham]

Appears conda-build is trying to detect linkages to Python dependencies needed during the build, but it seems to be incorrectly drawing the conclusion that they are not needed as there is not a library linkage found. This happens most notably with python and numpy as shown below.

Actual Behavior

   INFO (python-spams): plugin library package conda-forge::numpy-1.9.3-py27_blas_openblashd3ea46f_206 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (python-spams): dso library package conda-forge::python-2.7.15-h33da82c_6 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)

ref: https://circleci.com/gh/conda-forge/python-spams-feedstock/245

Expected Behavior

No complaints about overlinkage are issued.

Steps to Reproduce

Build this recipe with one of the variants in .ci_support.

Output of conda info

     active environment : base
    active env location : /Users/distiller/miniconda3
            shell level : 1
       user config file : /Users/distiller/.condarc
 populated config files : /Users/distiller/.condarc
          conda version : 4.5.12
    conda-build version : 3.17.5
         python version : 3.6.7.final.0
       base environment : /Users/distiller/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/osx-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/osx-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /Users/distiller/miniconda3/pkgs
                          /Users/distiller/.conda/pkgs
       envs directories : /Users/distiller/miniconda3/envs
                          /Users/distiller/.conda/envs
               platform : osx-64
             user-agent : conda/4.5.12 requests/2.20.1 CPython/3.6.7 Darwin/16.7.0 OSX/10.12.6
                UID:GID : 501:20
             netrc file : None
           offline mode : False

[mingwandroid]

Thanks @jakirkham.

INFO (python-spams): plugin library package conda-forge::numpy-1.9.3-py27_blas_openblashd3ea46f_206 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)

This isn't a warning, it's just informative but the text is incorrect. I propose changing it to:

INFO (python-spams): plugin library package conda-forge::numpy-1.9.3-py27_blas_openblashd3ea46f_206 in requirements/run

I personally prefer to see it, but am happy to go with the majority here (or some suggestion for better logic/user control around the info/warning/error messages in this code in general).

WARNING (python-spams): dso library package conda-forge::python-2.7.15-h33da82c_6 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)

This one is trickier. The logic is currently:

if the dependent package contains unused DSOs:
    if dependent package begins_with('lib') or has run_exports:
        ERROR
    else:
        WARNING

Now this is generally speaking what we want (I contend there aren't many dependent packages with DSOs where you don't autoload any of them or load them dynamically as plugins). For Python it's only complaining because we've been kind enough to provide python DSOs for people who wish to embed our Python (even though we never use them ourselves - when this happens in an extension module it is always a bug).

I do have logic to determine what constitutes a plugin package, and that basically hardcodes to them being dependent upon r-base, mro-base or python so I guess I could hardcode this logic to exclude those from the else: WARNING above too. Suggestions welcome!

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[h-vetinari]

Contrary to the title of this issue, the warnings here are about overdepending, which is a separate concern from overlinking (though in fairness, the names aren't exactly self-explanatory, aside from being underdocumented).

[h-vetinari]

Talking about numpy specifically, the following comment from @xhochy is relevant:

Due to the nature of NumPy's linking mechanism (which is really great!), you should never see an actual linkage against it. If the warning isn't always showing up, I would be very curious why it happens. All references are filled at runtime through the C function import_numpy().

Rephrasing according to my (limited) understanding, that means the linkage does not happen at build time (which is when conda-through-lief would be detecting things), and therefore that warning is inevitable -- unless numpy gets special-cased, or the link check gets enhanced to detect such runtime linkage.