Removing sortedcontainers from meta.yaml (#29) breaks my downstream builds

[ralexx]

Issue:
sortedcontainers was removed from meta.yaml run requirements (#29) but the accompanying upstream PR (pdfminer/pdfminer.six#525) has not yet been merged.

When I attempt to build a package that has pdfminer.six as an unpinned dependency,conda-build fails on version pdfminer.six-20201018-py_2:

Processing dependencies for mypackage==1.1.1
Searching for sortedcontainers
[...]
RuntimeError: Setuptools downloading is disabled in conda build. Be sure to add all dependencies 
in the meta.yaml  url=https://pypi.org/simple/sortedcontainers/

I have pinned my recipe to avoid this build but it would help if #29 was reverted until pdfminer/pdfminer.six#525 is merged.


Environment (conda list):

$ conda list
# packages in environment at /Users/me/anaconda/envs/build38:
#
# Name                    Version                   Build  Channel
beautifulsoup4            4.9.3              pyhb0f4dca_0    conda-forge
brotlipy                  0.7.0           py38h94c058a_1001    conda-forge
bzip2                     1.0.8                haf1e3a3_3    conda-forge
ca-certificates           2020.6.20            hecda079_0    conda-forge
certifi                   2020.6.20        py38h5347e94_2    conda-forge
cffi                      1.14.3           py38h9edaa1b_1    conda-forge
chardet                   3.0.4           py38h5347e94_1008    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
conda                     4.9.0            py38h5347e94_1    conda-forge
conda-build               3.20.4           py38h5347e94_1    conda-forge
conda-package-handling    1.7.2            py38h94c058a_0    conda-forge
conda-verify              3.1.1           py38h32f6830_1002    conda-forge
cryptography              3.1.1            py38h52adbb4_0    conda-forge
filelock                  3.0.12             pyh9f0ad1d_0    conda-forge
future                    0.18.2           py38h32f6830_2    conda-forge
glob2                     0.7                        py_0    conda-forge
icu                       67.1                 hb1e8313_0    conda-forge
idna                      2.10               pyh9f0ad1d_0    conda-forge
jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
libarchive                3.4.3                hf12134e_0    conda-forge
libblas                   3.8.0               17_openblas    conda-forge
libcblas                  3.8.0               17_openblas    conda-forge
libcxx                    11.0.0               h439d374_0    conda-forge
libffi                    3.2.1             hb1e8313_1007    conda-forge
libgfortran               5.0.0               h7cc5361_13    conda-forge
libgfortran5              9.3.0               h7cc5361_13    conda-forge
libiconv                  1.16                 haf1e3a3_0    conda-forge
liblapack                 3.8.0               17_openblas    conda-forge
liblief                   0.10.1               hb1e8313_2    conda-forge
libopenblas               0.3.10          openmp_h54245bb_5    conda-forge
libxml2                   2.9.10               h2c6e4a5_2    conda-forge
llvm-openmp               11.0.0               h73239a0_1    conda-forge
lz4-c                     1.9.2                hb1e8313_3    conda-forge
lzo                       2.10              haf1e3a3_1000    conda-forge
markupsafe                1.1.1            py38h94c058a_2    conda-forge
ncurses                   6.2                  hb1e8313_2    conda-forge
numpy                     1.19.2           py38ha98150c_1    conda-forge
openssl                   1.1.1h               haf1e3a3_0    conda-forge
pip                       20.2.4                     py_0    conda-forge
pkginfo                   1.6.0              pyh9f0ad1d_0    conda-forge
psutil                    5.7.2            py38h94c058a_1    conda-forge
py-lief                   0.10.1           py38heedec30_2    conda-forge
pycosat                   0.6.3           py38h94c058a_1005    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pyopenssl                 19.1.0                     py_1    conda-forge
pysocks                   1.7.1            py38h5347e94_2    conda-forge
python                    3.8.6           hcfdab8c_0_cpython    conda-forge
python-libarchive-c       2.9              py38h5347e94_2    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytz                      2020.1             pyh9f0ad1d_0    conda-forge
pyyaml                    5.3.1            py38h94c058a_1    conda-forge
readline                  8.0                  h0678c8f_2    conda-forge
requests                  2.24.0             pyh9f0ad1d_0    conda-forge
ripgrep                   12.1.1               haf1e3a3_1    conda-forge
ruamel_yaml               0.15.80         py38h94c058a_1003    conda-forge
setuptools                49.6.0           py38h5347e94_2    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
soupsieve                 2.0.1                      py_1    conda-forge
sqlite                    3.33.0               h960bd1c_1    conda-forge
tk                        8.6.10               hb0a8c7a_1    conda-forge
tqdm                      4.50.2             pyh9f0ad1d_0    conda-forge
urllib3                   1.25.11                    py_0    conda-forge
versioneer                0.18                       py_1    conda-forge
wheel                     0.35.1             pyh9f0ad1d_0    conda-forge
xz                        5.2.5                haf1e3a3_1    conda-forge
yaml                      0.2.5                haf1e3a3_0    conda-forge
zlib                      1.2.11            h7795811_1010    conda-forge
zstd                      1.4.5                h289c70a_2    conda-forge

Details about conda and system ( conda info ):

$ conda info

     active environment : build38
    active env location : /Users/me/anaconda/envs/build38
            shell level : 1
       user config file : /Users/me/.condarc
 populated config files : /Users/me/.condarc
          conda version : 4.9.0
    conda-build version : 3.20.3
         python version : 3.7.9.final.0
       virtual packages : __osx=10.14.6=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /Users/arg/anaconda  (writable)
           channel URLs : https://conda.anaconda.org/ralexx/osx-64
                          https://conda.anaconda.org/ralexx/noarch
                          https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/intel/osx-64
                          https://conda.anaconda.org/intel/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/arg/anaconda/pkgs
                          /Users/me/.conda/pkgs
       envs directories : /Users/me/anaconda/envs
                          /Users/me/.conda/envs
               platform : osx-64
             user-agent : conda/4.9.0 requests/2.24.0 CPython/3.7.9 Darwin/18.7.0 OSX/10.14.6
                UID:GID : 501:20
             netrc file : None
           offline mode : False

[estshorter]

I don't mind reverting the commit, but I wonder why the build was broken.
According to the bot, sortedcontainers is not used any more.
Does conda-build check setup.py?

[estshorter]

When I attempt to build a package that has pdfminer.six as an unpinned dependency

Could you give me a code example?

[ralexx]

Does conda-build check setup.py?

Apparently it does when build.sh or meta.yaml/build/script calls python setup.py install, as my package's build recipe does.

I see that your pdfminer.six-feedstock/recipe/meta.yaml uses build/script: {{ PYTHON }} -m pip install . --no-deps -vv rather than python setup.py install, and it appears to make a difference in the build process. If you customarily use this invocation in your recipes you may not see the error that I get.

Here is the source for a test package that reproduces my error when built with setuptools:
[setup_test.zip], but not when built with pip install.

The build log using setuptools shows the latest pdfminer.six-20201018-py_2 build being used throughout the build process. conda-build's call to python setup.py install on line 8185 triggers the exception.

The log from the pip install build shows pdfminer.six-20201018-py_2 in the build environment (line 11903) -- which does not break -- but somehow pdfminer.six-20200726-py_1 (line 12034) and its sortedcontainers-2.2.2-pyh9f0ad1d_0 dependency (line 12051) are included in the test environment.

[estshorter]

Thank you for your explanation, but what command did you run?
I'm not familiar with conda-build...

[ralexx]

Ha, neither am I 😅.

I run the builds from the package root using conda build --debug conda.recipe. Last argument is the directory containing meta.yaml.

[estshorter]

Thanks!
Could you provide the build recipe too? I'd like to test in my environment.

[ralexx]

It's in the source I attached setup_test.zip/conda.recipe/meta.yaml. [link]

[estshorter]

Sorry, I missed it. I'll check it later.

[estshorter]

I can reproduce the issue when specifying build.script as {{ PYTHON }} setup.py install.
But I don't know whether the command is suitable because the following commands are used in the documentation of conda-build and conda-forge:

• python -m pip install --no-deps --ignore-installed .
• python setup.py install --single-version-externally-managed --record=record.txt
• {{ PYTHON }} -m pip install . --no-deps -vv

I think you should use one of these command. (confirmed the successful build in my environment)

[ralexx]

Thank you, @estshorter. I will follow what conda-forge does.