conda-build doesn't respect --override-channels ?

[beenje]

Actual Behavior

When running conda build with --override-channels --channel conda-forge I would expect conda-build to only look and download packages from conda-forge. We can see below that some packages are downloaded from the defaults channel.

# conda config --show-sources
==> /opt/conda/.condarc <==
channels:
  - conda-forge
  - defaults
show_channel_urls: True

# conda build --override-channels --channel conda-forge recipe
...
Solving environment: ...working... done

## Package Plan ##

  environment location: /opt/conda/conda-bld/flask_1567173097489/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho


The following NEW packages will be INSTALLED:

    _libgcc_mutex:   0.1-main             defaults
    bzip2:           1.0.8-h516909a_0     conda-forge
    ca-certificates: 2019.6.16-hecc5488_0 conda-forge
    certifi:         2019.6.16-py37_1     conda-forge
    libffi:          3.2.1-he1b5a44_1006  conda-forge
    libgcc-ng:       9.1.0-hdf63c60_0     defaults
    libstdcxx-ng:    9.1.0-hdf63c60_0     defaults
    ncurses:         6.1-hf484d3e_1002    conda-forge
    openssl:         1.1.1c-h516909a_0    conda-forge
    pip:             19.2.3-py37_0        conda-forge
    python:          3.7.3-h33d41f4_1     conda-forge
    readline:        8.0-hf8c457e_0       conda-forge
    setuptools:      41.2.0-py37_0        conda-forge
    sqlite:          3.29.0-hcee41ef_1    conda-forge
    tk:              8.6.9-hed695b0_1002  conda-forge
    wheel:           0.33.6-py37_0        conda-forge
    xz:              5.2.4-h14c3975_1001  conda-forge
    zlib:            1.2.11-h516909a_1005 conda-forge

Expected Behavior

When using --override-channels conda-build should only use the channels given on the command line.

Steps to Reproduce

Try to build a recipe using --override-channels:

$ conda config --system --set show_channel_urls true
$ git clone https://github.com/conda-forge/flask-feedstock.git
$ cd flask-feedstock
$ conda build --override-channels --channel conda-forge recipe

Output of conda info

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 4.7.11
    conda-build version : 3.18.9
         python version : 3.7.3.final.0
       virtual packages :
       base environment : /opt/conda  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.7.11 requests/2.22.0 CPython/3.7.3 Linux/4.9.184-linuxkit debian/10 glibc/2.28
                UID:GID : 0:0
             netrc file : None
           offline mode : False

[marcelotrevisani]

For those specific packages such as libgcc-ng, _libgcc_mutex and libstdcxx-ng conda will get from the main package because conda-forge does not packaged them. Those packages are related to the compiler and conda-forge is using the compilers from pkgs/main

If you try to search for libgcc-ng (with conda-forge channel already added) you will see something like:

➜ conda search libgcc-ng
Loading channels: done
# Name                       Version           Build  Channel             
libgcc-ng                      5.4.0      hdf63c60_3  pkgs/main           
libgcc-ng                      7.2.0      h7cc24e2_2  pkgs/main           
libgcc-ng                      7.2.0      hcbc56d2_1  pkgs/main           
libgcc-ng                      7.2.0      hdf63c60_3  pkgs/main           
libgcc-ng                      7.3.0      hdf63c60_0  pkgs/main           
libgcc-ng                      8.2.0      hdf63c60_0  pkgs/main           
libgcc-ng                      8.2.0      hdf63c60_1  pkgs/main           
libgcc-ng                      9.1.0      hdf63c60_0  pkgs/main  

Did you see the same behavior for another package?

[beenje]

I tried to give a simple example. My use case is with some private channels.
I noticed that the channels in .condarc were used because the "solving environment" should be very fast when using only the given channel. All required packages are available in that channel but some newer packages were still downloaded from defaults. If I remove the extra channels from .condarc, the conda-build command is much faster and all packages are downloaded from the same channel.

In the described example, I'd expect the command to fail (due to some packages not found).
If you run the command with bioconda channel, the result will be the same:
conda build --override-channels --channel bioconda recipe

Nothing will be downloaded from bioconda. Packages will be download from the conda-forge and defaults channels (defined in the .condarc file). The command should fail if the required packages are not available in bioconda.

When using --override-channels, the channels defined in .condarc should be ignored.

[marcelotrevisani]

Indeed, it seems like a bug. I will try to take a look a bit further.
Thanks for reporting it

[mforbes]

Related: how can one ignore .condarc in general? I have tried finding a coherently expressed solution to this problem but don't see it anywhere. I would like to be able to specify the proper channel configurations etc. required to build a package as a config file within that package, not relying on whatever the current user happens to have decided they want to do at the moment with their top-level .condarc file. Is this what --override-channels is for? Can this be included somehow in the local build config files like meta.yaml? The documentation is very confusing on these points.

[brianwise5107]

I've been using conda-build recently, and I also am seeing the same behavior where --override-channels does not override the local .condarc. I have the same situation where I want to ignore all channels in .condarc, and would like it to throw an error if it cannot find the package in the user-defined channels.

Looking through the relevant code a bit, and I think the get_build_index function already has a built in optional argument that can be used to accomplish this: omit_defaults. This argument appears to be used within the function to handle conda HTTP errors, but I believe it could also be used to input from the conda-build config.

For example, where get_build_index is defined here, an optional argument could be added such as omit_defaults=m.config.override_channels. This would then pick up the override-channels config from the user, and build the conda package while ignoring the .condarc as desired.

The get_build_index is called in a few places throughout the repo that probably should be updated, but I believe the variable infrastructure to make this change already exists in the code.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[beenje]

I can still reproduce the issue:

$ conda info

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 22.11.1
    conda-build version : 3.23.3
         python version : 3.10.6.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.17=0
                          __linux=5.15.49=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/22.11.1 requests/2.28.1 CPython/3.10.6 Linux/5.15.49-linuxkit centos/7.9.2009 glibc/2.17
                UID:GID : 0:0
             netrc file : None
           offline mode : False

$ git clone https://github.com/conda-forge/flask-feedstock.git
$ cd flask-feedstock
# I'd expect this command to fail because required packages aren't in bioconda but packages are downloaded from conda-forge:
$ conda build --override-channels --channel bioconda recipe
...
The following NEW packages will be INSTALLED:

    _libgcc_mutex:    0.1-conda_forge           conda-forge
    _openmp_mutex:    4.5-2_gnu                 conda-forge
    bzip2:            1.0.8-h7f98852_4          conda-forge
    ca-certificates:  2022.12.7-ha878542_0      conda-forge
    ld_impl_linux-64: 2.40-h41732ed_0           conda-forge
    libffi:           3.4.2-h7f98852_5          conda-forge
    libgcc-ng:        12.2.0-h65d4601_19        conda-forge
    libgomp:          12.2.0-h65d4601_19        conda-forge
    libnsl:           2.0.0-h7f98852_0          conda-forge
    libsqlite:        3.40.0-h753d276_0         conda-forge
    libuuid:          2.32.1-h7f98852_1000      conda-forge
    libzlib:          1.2.13-h166bdaf_4         conda-forge
    ncurses:          6.3-h27087fc_1            conda-forge
    openssl:          3.0.8-h0b41bf4_0          conda-forge
    pip:              23.0.1-pyhd8ed1ab_0       conda-forge
    python:           3.11.0-he550d4f_1_cpython conda-forge
    readline:         8.1.2-h0f457ee_0          conda-forge
    setuptools:       67.3.3-pyhd8ed1ab_0       conda-forge
    tk:               8.6.12-h27826a3_0         conda-forge
    tzdata:           2022g-h191b570_0          conda-forge
    wheel:            0.38.4-pyhd8ed1ab_0       conda-forge
    xz:               5.2.6-h166bdaf_0          conda-forge
...

[dkomisar]

We need the ability for conda build to not download from defaults in order for us to complete some of the build automation we want for the rocket platform.

This could be accomplished with a new option if that's easier. We specifically want an option that overrides all channels and doesn't try to be "smart" and contact defaults if it can't find the package. We explicitly want it to fail in this case.

[JeanChristopheMorinPerso]

Repro steps:

Let's take a recipe that looks like

package:
  name: test
  version: 1.0.0

requirements:
  host:
    - libcurl

Now, create an empty channel locally

$ mkdir /tmp/asd/mychannel
$ cd /tmp/asd/mychannel
$ conda index .

Now build the test feedstock with conda build <path to feedstock> -c /tmp/asd/mychannel --override-channels.

I would expect conda-build to fail. Instead, it fetches packages from defaults.