Conda Index ignores unrecognized platforms, errors if they are specified on CLI

[ialb97]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

Using conda_index to generate a local channel. If the subdirectory isn't a typical platform conda_index ignores that subdirectory when indexing the channel. If that subdirectory is explicitly specified at the command line, it starts to process it but fails with an assert error in the conda Subdir module. The subdirectory being tested is linux-64-cxx03

Here is the output when calling conda_index without specifying the subdir:

(test_env) ialb:~/test_conda_index$ tree local_channel/
local_channel/
└── linux-64-cxx03
    └── openssl-1.1.1s-h7f8727e_0.tar.bz2

1 directory, 1 file
(test_env) ialb:~/UPE/bazel_work/test_conda_index$ python -m conda_index local_channel/
2023-11-10T14:30:43 Migrate database
2023-11-10T14:30:43 CONVERT .cache
2023-11-10T14:30:43 Migrate database
2023-11-10T14:30:43 noarch cached 0 B from 0 packages at 0 B/second
2023-11-10T14:30:44 Subdir: noarch Gathering repodata
2023-11-10T14:30:44 noarch Writing pre-patch repodata
2023-11-10T14:30:44 noarch Applying patch instructions
2023-11-10T14:30:44 noarch Writing patched repodata
2023-11-10T14:30:44 noarch Building current_repodata subset
2023-11-10T14:30:44 noarch Writing current_repodata subset
2023-11-10T14:30:44 noarch Writing index HTML
2023-11-10T14:30:44 Completed noarch
(test_env) ialb:~test_conda_index$ tree local_channel/
local_channel/
├── linux-64-cxx03
│   └── openssl-1.1.1s-h7f8727e_0.tar.bz2
└── noarch
    ├── current_repodata.json
    ├── index.html
    ├── repodata.json
    └── repodata_from_packages.json

2 directories, 5 files

Here is the error when explicitly specifying the subdir at the command line:

(test_env) ialb:~/test_conda_index$ tree local_channel/
local_channel/
└── linux-64-cxx03
    └── openssl-1.1.1s-h7f8727e_0.tar.bz2

1 directory, 1 file
(test_env) ialb:~/test_conda_index$ python -m conda_index --verbose --subdir linux-64-cxx03 local_channel/
2023-11-10T14:33:17 Indexing ['linux-64-cxx03'] does not include 'noarch'
2023-11-10T14:33:17 CondaIndexCache channel_root=/home/ialb/test_conda_index/local_channel, subdir=linux-64-cxx03 db_filename=/home/ialb/test_conda_index/local_channel/linux-64-cxx03/.cache/cache.db cache_is_brand_new=True
2023-11-10T14:33:17 Migrate database
2023-11-10T14:33:17 CONVERT .cache
2023-11-10T14:33:17 Migrate database
2023-11-10T14:33:17 linux-64-cxx03 listdir
2023-11-10T14:33:17 linux-64-cxx03 save fs state
2023-11-10T14:33:17 linux-64-cxx03 find packages to extract
2023-11-10T14:33:17 linux-64-cxx03 extract 1 packages
2023-11-10T14:33:17 linux-64-cxx03 cached 3.8 MB from 1 packages at 21.8 MB/second
2023-11-10T14:33:17 Subdir: linux-64-cxx03 Gathering repodata
2023-11-10T14:33:17 linux-64-cxx03 Writing pre-patch repodata
2023-11-10T14:33:17 linux-64-cxx03 Applying patch instructions
2023-11-10T14:33:17 linux-64-cxx03 Writing patched repodata
2023-11-10T14:33:17 linux-64-cxx03 Building current_repodata subset
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/conda_index/index/__init__.py", line 634, in index_prepared_subdir
    current_repodata = _build_current_repodata(
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/conda_index/index/__init__.py", line 448, in _build_current_repodata
    r = _get_resolve_object(subdir, repodata=repodata)
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/conda_index/index/__init__.py", line 344, in _get_resolve_object
    sd = SubdirData(channel)
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/conda/core/subdir_data.py", line 80, in __call__
    assert channel.subdir
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/conda_index/__main__.py", line 3, in <module>
    conda_index.cli.cli()
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/conda_index/cli/__init__.py", line 127, in cli
    channel_index.index(
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/conda_index/index/__init__.py", line 597, in index
    result = future.result()
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/ialb/umamba/envs/test_env/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
AssertionError
(test_env) ialb:~/test_conda_index$ tree local_channel/
local_channel/
├── linux-64-cxx03
│   ├── openssl-1.1.1s-h7f8727e_0.tar.bz2
│   ├── repodata.json
│   └── repodata_from_packages.json
└── noarch

2 directories, 3 files

Conda Info

(test_env) ialb:~/test_conda_index$ conda info

     active environment : base
    active env location : /home/ialb/umamba/envs/test_env
            shell level : 2
       user config file : /home/ialb/.condarc
 populated config files : /home/ialb/.condarc
          conda version : 22.9.0
    conda-build version : not installed
         python version : 3.10.13.final.0
       virtual packages : __cuda=12.2=0
                          __linux=5.15.90.1=0
                          __glibc=2.35=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/ialb/umamba/envs/test_env  (writable)
      conda av data dir : /home/ialb/umamba/envs/test_env/etc/conda
  conda av metadata url : None
           channel URLs : 
          package cache : /home/ialb/umamba/envs/test_env/pkgs
                          /home/ialb/.conda/pkgs
       envs directories : /home/ialb/umamba/envs/test_env/envs
                          /home/ialb/.conda/envs
               platform : linux-64
             user-agent : conda/22.9.0 requests/2.31.0 CPython/3.10.13 Linux/5.15.90.1-microsoft-standard-WSL2 ubuntu/22.04.3 glibc/2.35
                UID:GID : 1000:1000
             netrc file : /home/ialb/.netrc
           offline mode : False

Conda Config

No response

Conda list

(test_env) ialb:~/test_conda_index$ conda list --show-channel-urls
# packages in environment at /home/ialb/umamba/envs/test_env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py310h2fee648_0    conda-forge
charset-normalizer        3.3.1              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
conda                     22.9.0          py310hff52083_2    conda-forge
conda-index               0.3.0           py310ha770c72_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
cryptography              41.0.5          py310h75e40e8_0    conda-forge
filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libsqlite                 3.43.2               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
markupsafe                2.1.3           py310h2372a71_1    conda-forge
more-itertools            10.1.0             pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
openssl                   3.1.4                hd590300_0    conda-forge
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pycosat                   0.6.6           py310h2372a71_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         hd12c33a_0_cpython    conda-forge
python_abi                3.10                    4_cp310    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.18.3          py310h2372a71_0    conda-forge
ruamel.yaml.clib          0.2.7           py310h2372a71_2    conda-forge
ruamel_yaml               0.15.80         py310h2372a71_1009    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.13               h2797004_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.0.7              pyhd8ed1ab_0    conda-forge
wheel                     0.41.3             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zstandard                 0.21.0          py310h1275a96_1    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Additional Context

All packages are installed from conda-forge.

[dholth]

That's part of conda File "/home/ialb/umamba/envs/test_env/lib/python3.10/site-packages/conda/core/subdir_data.py", line 80, in __call__ and it might be necessary to add to conda's list of known subdirs to solve.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!