Pip dependency resolution error pyscaffold<3.3a0,>=3.2a0

[rdmolony]

Describe the bug
While following the excellent guide Installation guide for Windows (from the source) - with minor changes for Windows Subsystem for Linux 2 (WSL2) Ubuntu a dependency conflict prevented install.

To Reproduce
Steps to reproduce the behavior:

(base) wsl-rowanm@Rowan:/mnt/c/users/rowanm/Code/CityEnergyAnalyst$ git clone https://github.com/architecture-building-systems/CityEnergyAnalyst
(base) wsl-rowanm@Rowan:/mnt/c/users/rowanm/Code/CityEnergyAnalyst$ conda env create -f=environment.ubuntu.yml
(base) wsl-rowanm@Rowan:/mnt/c/users/rowanm/Code/CityEnergyAnalyst$ conda activate cea
(cea) wsl-rowanm@Rowan:/mnt/c/users/rowanm/Code/CityEnergyAnalyst$ pip install -e ".[dev]"
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Obtaining file:///mnt/c/users/rowanm/Code/CityEnergyAnalyst
  WARNING: cityenergyanalyst 3.11.0a5 does not provide the extra 'dev'
Collecting SALib
  Using cached https://files.pythonhosted.org/packages/f7/33/cee4d64f7c40f33c08cf5ef5c9b1fb5e51f194b5deceefb5567112800b70/SALib-1.3.11.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /home/wsl-rowanm/miniconda3/envs/cea/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-__aWbf/SALib/setup.py'"'"'; __file__='"'"'/tmp/pip-install-__aWbf/SALib/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-__aWbf/SALib/pip-egg-info
         cwd: /tmp/pip-install-__aWbf/SALib/
    Complete output (24 lines):
    DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
    ERROR: Could not find a version that satisfies the requirement pyscaffold<3.3a0,>=3.2a0 (from versions: 0.1.2, 0.2, 0.3, 0.4, 0.4.1, 0.5, 0.6, 0.6.1, 0.7, 0.8, 0.9, 1.0rc1, 1.0rc2, 1.0rc3, 1.0rc4, 1.0, 1.1rc1, 1.1, 1.2rc1, 1.2, 1.3rc1, 1.3rc2, 1.3, 1.3.1, 1.3.2, 1.4, 2.0b1, 2.0b2, 2.0b4, 2.0b5, 2.0, 2.0.1rc1, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.1b1, 2.1, 2.2b1, 2.2b2, 2.2b3, 2.2b4, 2.2b5, 2.2rc1, 2.2rc2, 2.2, 2.2.1, 2.3rc1, 2.3rc3, 2.3rc4, 2.3rc5, 2.3rc6, 2.3rc7, 2.3rc8, 2.3rc9, 2.3rc10, 2.3rc11, 2.3rc12, 2.3rc13, 2.3, 2.4rc1, 2.4rc2, 2.4rc3, 2.4, 2.4.1, 2.4.2, 2.4.4rc1, 2.4.4, 2.5a2, 2.5b1, 2.5b2, 2.5b3, 2.5b4, 2.5b5, 2.5b6, 2.5b7, 2.5b9, 2.5b10, 2.5rc1, 2.5rc2, 2.5rc3, 2.5rc4, 2.5, 2.5.1, 2.5.2, 2.5.3, 2.5.4rc1, 2.5.4, 2.5.5b1, 2.5.5b2, 2.5.5, 2.5.6b1, 2.5.6, 2.5.7, 2.5.8rc1, 2.5.8rc2, 2.5.8rc3, 2.5.8, 2.5.9rc1, 2.5.9, 2.5.10rc1, 2.5.10, 2.5.11, 3.0a2, 3.0a4, 3.0a5, 3.0a6, 3.0a7, 3.0a8, 3.0a9, 3.0a10, 3.0b1, 3.0b2, 3.0rc1, 3.0rc2, 3.0rc3, 3.0rc4, 3.0rc5, 3.0rc7, 3.0, 3.0.1rc1, 3.0.1rc2, 3.0.1, 3.0.2rc3, 3.0.2rc4, 3.0.2, 3.0.3, 3.1b1, 3.1b2, 3.1rc1, 3.1rc2)
    ERROR: No matching distribution found for pyscaffold<3.3a0,>=3.2a0
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-__aWbf/SALib/setup.py", line 28, in <module>
        setup(use_pyscaffold=True, scripts=scripts)
      File "/home/wsl-rowanm/miniconda3/envs/cea/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/home/wsl-rowanm/miniconda3/envs/cea/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/wsl-rowanm/miniconda3/envs/cea/lib/python2.7/site-packages/setuptools/dist.py", line 721, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/wsl-rowanm/miniconda3/envs/cea/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/home/wsl-rowanm/miniconda3/envs/cea/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/home/wsl-rowanm/miniconda3/envs/cea/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/home/wsl-rowanm/miniconda3/envs/cea/lib/python2.7/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/home/wsl-rowanm/miniconda3/envs/cea/lib/python2.7/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
        raise DistutilsError(str(e))
    distutils.errors.DistutilsError: Command '['/home/wsl-rowanm/miniconda3/envs/cea/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpd6t_nU', '--quiet', 'pyscaffold<3.3a0,>=3.2a0']' returned non-zero exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Expected behaviour
pip install -e ".[dev]" installs all CityEnergyAnalyst dependencies successfully.

Hardware (please complete the following information):

• OS and version: [e.g. iOS 10.7 or Windows 10]
  Windows Subsystem for Linux 2 (WSL2)

❯ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

• Version of CEA you are using [e.g. v. 2.31.1]

❯ git describe --tag
v3.8.0a0-78-g7e55b2861

---

I'm finding poetry very useful for another project drem for dependency management. I saw that dephell seems to solve the issue of moving between setup.py, requirements.txt, Pipfile, poetry - I'm not sure if this is something that interests you but thought to raise it here in case! :) Btw really liking cityenergyanalyst so far, the defaults are extremely useful to start off and the GUI and dashboard very easy to use and sleek

[daren-thomas]

@rdmolony we've just ported master to python3. so that might be adding to the conflicts. also we'll have to update the manual installation instructions to adjust.

as luck would have it, i'm tinkering with a dockerized version of cea right now and running into conflicts too - i'll publish a new environment.ubuntu.yml for you as soon as I have it ready. i actually expect a conda-packed environment to be an end result of this. including binaries for daysim. that would make installation as easy as unzipping.

the reason we're not using poetry is mainly because of some non-python dependencies (gdal and friends, numpy etc.) that are really hard to do on windows. as far as i understand, conda solves this easier. even so, getting all the dependencies versions right is always a bit of a challenge and a bit of a black art.

[daren-thomas]

btw: i did a grep -hr --include \*.py "^import" | sort | uniq | grep -v "import cea" > ..\docker\server\imports.py on the cea folder yesterday and the results would be a good starting point, if you'd like to give it a go - still need to filter out standard libraries, but that's the list of stuff we need to be able to import.

[rdmolony]

Sounds good to me :)

I'm not in any hurry but tried your handy bash command grep -hr --include \*.py "^import" | sort | uniq | grep -v "import cea" > requirements.txt, eliminated the import strings in vim (then just deleted all the matplotlib.cm as cm as got lazy with regex ...) to trial out conda dependency resolution.

I tried conda install --file requirements.txt but annoyingly conda didn't resort to pip when it couldn't find a particular package. I'm not sure how to get around this with conda other than pip install -r requirements.txt (is there a better way?!) so I had a further look into to this to see if poetry could be used with a C-library dependency such as GDAL (I have a similar dependency on geopandas in drem). From this poetry issue I found that it lets you add Windows wheels poetry add ./GDAL-3.1.3-cp38-cp38-win_amd64.whl (available here) or binaries as dependencies. Would this resolve your issue of non-python dependencies?

This issue has a one-liner cat requirements.txt | perl -pe 's/([<=>]+)/:$1/' | xargs -t -n 1 -I {} poetry add '{}' for reading above requirements.txt file.

[daren-thomas]

@rdmolony you might be interested in this branch: https://github.com/architecture-building-systems/CityEnergyAnalyst/tree/2834-docker-container-for-remap

In it, I have update the file environment.ubuntu.yml and am able to install the CEA on WSL2 (running Ubuntu). I've attached that file: environment.ubuntu.yml.zip

Steps to create your environment:

• (assuming you have Anaconda/Miniconda installed in WSL2):
• (assuming your current working directory is the root of the cea repository)
• conda env create -n cea -f environment.ubuntu.yml
• pip install -e . (NOTE: the [dev] is not necessary anymore. pip will even complain a bit if you use it, see your output above)

[rdmolony]

Thanks @daren-thomas!

Btw I managed to get around using poetry & conda by using poetry2conda which converts my pyproject.toml on the fly to an env.yml for conda