Consolidate files listing dependencies

[adamjstewart]

Currently, the following files all list torchgeo's dependencies:

• setup.cfg
• requirements.txt
• docs/requirements.txt
• spack.yaml
• environment.yml
• .github/workflows/{docs/release/style/tests}.yaml
• .pre-commit-config.yaml

Whenever someone needs to add a new dependency to torchgeo, they need to add it to most of these files, making it easy to forget and difficult to keep track of. We should try to consolidate these. Here are some suggestions:

setup.cfg

We may be able to use requirements.txt instead of setup.cfg by using something like pbr or setuptools-scm. But see below for another option.

requirements.txt

I don't think this file is necessary. pip install .[all] in the root directory should read the dependencies from setup.cfg and provide much of the same support as pip install -r requirements.txt. I think pip install git+https://github.com/microsoft/torchgeo may also work.

docs/requirements.txt

This can be removed pending pytorch/pytorch_sphinx_theme#143

spack.yaml

This can be removed by adding a py-torchgeo package to Spack. It can install both stable releases (once we release) as well as development versions.

environment.yml

I'm not sure how to remove this one. Can conda-forge be used to install development versions?

.github/workflows/{docs/release/style/tests}.yaml

If we pip install all of our dependencies from requirements.txt, it will be slower than only installing the deps for a specific tests, but we won't have to duplicate info as much.

.pre-commit-config.yaml

No ideas, the developers don't seem interested in sharing configuration files with any other tool: pre-commit/pre-commit#1165

[ashnair1]

Poetry might be a good solution for this.

Advantages:

• Integrates well with pyproject.toml. In fact poetry considers pyproject.toml the single source of dependencies.
• Once integrated we won't need requirements.txt, environment.yaml and setup.cfg.
• Also makes it easy to publish to pypi Not sure if it's that significant if you already have a workflow defined
• Supports optional dependencies and recently added support for Dependency groups python-poetry/poetry#4260. Could be useful for installing specific group of deps for specfic tests.

Disadvantages (maybe?):

• Dependencies will need to be moved to a single location - pyproject.toml
• Switching to poetry might be non-trivial for a large project like this.

Personally, I've switched from using conda to poetry in my projects and have found it much better but YMMV. If there is interest I can draft a PR.

Edit: Btw, I'm not endorsing poetry by any means. Just wanted to point out another option. Poetry like any other library has its own share of gotchas especially if you try to leverage some features of pip like --find-links, --build-options / --install-options etc.

Some references:

• Support for pip install --build-options / --global-options / --install-options python-poetry/poetry#845
• Equivalent behaviour to pip install --find-links python-poetry/poetry#1511
• Trouble with git link dependency rooted in subfolder python-poetry/poetry#4380

I think python-poetry/poetry-core#192 might have fixed my (3rd) issue.

[adamjstewart]

Yes, I've been keeping an eye on alternative build tools like poetry and flit. So far I haven't seen a significant advantage compared to setuptools and setup.cfg.

Supports optional dependencies and recently added support for python-poetry/poetry/pull/4260. Could be useful for installing specific group of deps for specific tests.

Setuptools has always had support for this, so I don't see this as an advantage.

Personally, I've switched from using conda to poetry in my projects and have found it much better but YMMV.

How does poetry replace conda?

[ashnair1]

Perhaps better is too strong a word. Convenient would be more appropriate. Poetry doesn't replace conda. conda manages environments and resolves dependencies whereas poetry only does the latter. The key here is how a project uses conda. Is it used for environment management? Installing specific non-python libraries? Both?

I mainly used conda for environment management and I preferred poetry because:

• Manual update and maintenance of environment.yml
  With conda I needed to maintain an environment.yml file and had to manually update it for every package I added.
  With poetry I can keep adding dependencies via poetry add which updates pyproject.toml and the poetry.lock file (for transitive dependencies) and go about my work. Also one less file for me to maintain.

• conda is generally slower.
  This isn't a knock against conda since it's scope extends beyond python and python packages. But due to this, it takes a while for dependencies to be resolved. poetry is a dependency manager for python and I've found it to be faster to set up an environment when I work with python projects.

Misc issues:

• Need to keep track of conda-forge and pypi versions.
• Cross platform environments

Poetry and pyenv together gives me enough environment management without the overhead of maintaining an additional file of dependencies and tracking a separate package registry (conda-forge). Here I prefer to use two specialised tools over a single massive one.

That being said, for projects that utilise non-python libraries (like GDAL for example), I would still use conda.

[adamjstewart]

Gotcha. I think a lot of developers (especially @calebrob6) prefer conda. Personally I use Spack. And a lot of people just want to use pip. I think supporting more package managers than that would add more headaches.

[adamjstewart]

As far as I can tell, there's no way to have a rolling development version in conda-forge or have optional dependencies like there is with pip or spack, so we may have to keep environment.yml forever ☹️