Using --all-extras does not install from project.optional-dependencies

[mbrancato]

• Poetry version: Poetry (version 1.3.2)
• Python version: Python: 3.10.7
• OS version and name: macOS 12.6
• pyproject.toml: https://gist.github.com/mbrancato/d9ba6b9b64fa021a333e2892b33a210a

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

The documentation indicates that poetry install --all-extras will install all "optional dependencies". Those in both the main group marked as optional, and the dev group. I've noticed that other tools with optional dependencies are not installed when using --all-extras.

Example in attached gist, summary:

• FastAPI is a direct dependency
• FastAPI depends on starlette
• starlette uses pyproject.toml and has httpx as an optional dependency
• FastAPI / starlette has a TestClient class
• The TestClient class uses httpx
• httpx is defined in pyproject.toml
• Using the TestClient fails as httpx is not installed.
• Workaround is that httpx must be added as a direct dependency

[dimbleby]

You have misunderstood: --all-extras installs... all extras. The pyproject.toml that you have provided defines no extras.

I don't immediately see where the docs imply anything other than this: but if they are unclear then I expect a merge request putting this right would be welcome.

[evanrittenhouse]

I think the confusion may stem from

Poetry supports extras to allow expression of:

• optional dependencies, which enhance a package, but are not required; and

But the issue is reporting that starlette has httpx in its project.optional-dependencies, and running poetry install --all-extras doesn't actually install httpx. While the attached pyproject.toml doesn't have any optional dependencies, starlette does. The user expects all optional-dependencies to be installed in all subpackages when running --all-extras - @mbrancato, please correct me if I'm wrong.

FWIW, after reading the docs I'd also expect that behavior. I'm not sure if it's intended or not, but a documentation update/bugfix may be in order.

[mbrancato]

yes @evanrittenhouse - exactly that. I did have to dig into the dependency of starlette to identify that httpx was an "optional dependency". And that is where the documentation twice refers to extras as including "optional dependencies".

A little ways down in the "extras" section, it does say:

Any extras you don’t specify will be removed. Note this behavior is different from optional dependency groups not selected for install, e.g. those not specified via install --with.

The project.optional-dependencies in the pyproject.toml defines that the keys provide project extras.

So I would think one of the following commands would work, but they do not.

poetry install --with full
poetry install --extras full

So I think at a minimum, the term "optional dependencies" needs to be removed from the "extras" section of the documentation.

I am able to solve my issue by doing the following:

[tool.poetry.group.dev.dependencies]
fastapi = {version = "^0.92.0", extras = ["all"]}

Basically, the extras must be defined in the direct dependency, and that dependency must include things needed for its sub-dependency.

[evanrittenhouse]

@dimbleby if we want --with full or --extras full to include subdependency extras, I can try to implement that

[dimbleby]

No code change should be needed, this all is working as intended. --extras foo is to install the foo extra defined by the current project, --all-extras is to install all extras defined by the current project. If you want to install extras defined by dependencies then express that in the dependency, per #7556 (comment).

Again, docs improvements are encouraged

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.