poetry install is slow / make resolver smarter (more lazy)

[blueyed]

Running poetry install on poetry itself (with poetry 0.8.5 from pip install --user poetry) is slow:

% poetry install
Updating dependencies
Resolving dependencies...........................................................


Package operations: 33 installs, 0 updates, 0 removals

Writing lock file

  - Installing attrs (17.4.0)
  - Installing certifi (2018.4.16)
  - Installing chardet (3.0.4)
  - Installing click (6.7)
  - Installing coverage (4.5.1)
  - Installing first (2.0.1)
  - Installing idna (2.6)
  - Installing jinja2 (2.10)
  - Installing livereload (2.5.1)
  - Installing markdown (2.6.11)
  - Installing markupsafe (1.0)
  - Installing more-itertools (4.1.0)
  - Installing pastel (0.1.0)
  - Installing pluggy (0.6.0)
  - Installing py (1.5.3)
  - Installing pyyaml (3.12)
  - Installing tornado (4.5.3)
  - Installing urllib3 (1.22)
  - Installing cachecontrol (0.12.4)
  - Installing cachy (0.1.1)
  - Installing lockfile (0.12.2)
  - Installing mkdocs (0.17.3)
  - Installing msgpack-python (0.5.6)
  - Installing pip-tools (2.0.1)
  - Installing pkginfo (1.4.2)
  - Installing pygments (2.2.0)
  - Installing pymdown-extensions (4.10)
  - Installing pyparsing (2.2.0)
  - Installing pyrsistent (0.14.2)
  - Installing pytest (3.5.0)
  - Installing pytest-cov (2.5.1)
  - Installing requests (2.18.4)
  - Installing requests-toolbelt (0.8.0)
poetry install  73.17s user 8.03s system 18% cpu 7:12.94 total

It took 7:12 minutes, mostly resolving dependencies - is that expected given a clear/cold cache?

Looking at poetry -vvv debug:resolve it turns out that it downloads/considers
all available versions of requests for example, which are 120+:

:  11: Getting dependencies for requests-2.18.4
…
:  11: Getting dependencies for requests-0.2.0
:   0: Creating possibility state for requests (*) (39 remaining)

[sdispater]

On a cold cache, yes, this is expected. See https://poetry.eustace.io/docs/basic-usage/#specifying-dependencies.

The resolver flattens the dependency graph to be sure to be exhaustive. So, if your package or one the dependency sets wildcard dependencies, the resolver will pick up every package.

For now, there is no way around it since the resolver was designed with exhaustiveness in mind rather than speed.

However, I am working on improving the speed of the resolution but this is rather tricky.

[sdispater]

@blueyed Commit f719f4b on the develop branch improves the speed of the dependency resolution. For instance, we go from ~30 minutes to resolve boto3 dependencies to ~11 seconds.

It will be available in the next 0.9.0 release.

[agoose77]

Awesome work @sdispater, that will be really useful

[sdispater]

Release 0.9.0 is out!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.