Support dependencies groups

[pawamoy]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

As @mozartilize stated in this comment, groups from Ruby's gemfile could be a great addition and a solution to our complex workflows and different environments (local, production, ci, docs, etc.).

Instead of supporting more tool.poetry.*-dependencies sections, we could add support for a groups property on dependencies. It would be more flexible than rigid sections.

I also understand there is the extras functionality, but using extras changes the behavior of poetry install, as one needs to now specify the extras to install all the dependencies. Also, extras are opt-in only, and often times we need opt-out.

[tool.poetry.dependencies]
python = "^3.6"
requests = "*"

[tool.poetry.dev-dependencies]
bandit = { version = "^1.5", groups = ["ci"] }
black = { version = "*", allows-prereleases = true, groups = ["ci"] }
coverage = { version = "=5.0a8", allows-prereleases = true, groups = ["ci"] }
flake8 =  { version = "^3.6", groups = ["ci"] }
ipython =  { version = "^7.2", groups = ["local"] }
isort = { version = "^4.3", extras = ["pyproject"], groups = ["ci"] }
jinja2-cli =  { version = "^0.7.0", groups = ["local"] }
pylint = { git = "https://github.com/PyCQA/pylint", groups = ["ci"] }
pytest =  { version = "^4.3", groups = ["ci"] }
pytest-cov =  { version = "^2.8", groups = ["ci"] }
pytest-sugar =  { version = "^0.9.2", groups = ["ci"] }
pytest-xdist =  { version = "^1.26", groups = ["ci"] }
recommonmark =  { version = "^0.6.0", groups = ["docs"] }
safety =  { version = "^1.8", groups = ["ci"] }
sphinx =  { version = "^1.8", groups = ["docs"] }
sphinx-rtd-theme =  { version = "^0.4.2", groups = ["docs"] }
toml =  { version = "^0.10.0", groups = ["docs"] }

Also:

• default group would be attributed to dependencies without groups (requests in the above example)
• each dependencies would be attributed its own group as well, something like /bandit for bandit, etc.

Then we would have the following options for the poetry install command:

• --groups: explicitely specify which groups to install (these and only these groups, no implicit default group)
• --with-groups or --with: default group plus the specified groups
• --without-groups or --without: all groups minus the specified groups

Usage would be:

# in CI
poetry install --groups ci

# or in specific CI jobs
poetry install --groups /bandit  # bandit reads the code, we don't need anything else
poetry install --with /safety    # safety needs production dependencies to be installed

# locally, all three lines are equivalent
poetry install  # installs everything, as usual
poetry install --with local,ci,docs
poetry install --groups default,local,ci,docs

# on readthedocs
poetry install --with docs  # sphinx-autodoc needs production dependencies to be installed

#1007 could then probably be closed in favor of this issue.

[pawamoy]

I didn't show any use-case for --without, any ideas?

Neither for multiple groups on dependencies. This one must be clarified/specified more thoroughly I guess:

default-dep = "*"
group-dep = { version = "*", groups = ["hello", "world"] }

Then what happens when I do:

poetry install --with hello
# since we do not include the "world" group, do we include group-dep or not? 

poetry install --without hello
# since we exclude the "hello" group, but keep the other ones including "world", do we exclude group-dep or not?

From the group page mentioned above (for --without):

Install all gems, except those in the listed groups. Gems in at least one non-excluded group will still be installed.

So if we follow the same logic:

poetry install --without hello
# will still install group-dep since it's in the non-excluded "world" group

poetry install --with hello
# should be the inverse? so it would not include group-dep since "world" is non-included?

[thejohnfreeman]

@pawamoy I'll try to answer your questions as I understand them.

1. What are the use-cases for --without?

The first is excluding documentation dependencies from my test environment in continuous integration. I have a library, project-template-python, that fails on AppVeyor during installation of dependencies because some of them report An error occurred when reading setup.py or setup.cfg: 'Attribute' object has no attribute 'id'. Even though none of these warnings halts the installation or breaks the environment for testing, and even though I specified $ErrorActionPreference = "Continue", AppVeyor halts the test as a failure. I know that at least one of these dependencies is pyobjc-core, depended on through sphinx-autobuild, a documentation dependency, and I suspect that the others are related to non-test dependencies too. Instead of figuring out how to get AppVeyor to ignore these warnings, I'd rather tell Poetry to skip their installation.

This adds a performance benefit, too, as @rmorshea put it:

A common use case for having dev-extras is bifurcated requirements for local development vs remote CI/CD pipelines. Given how slow dependency resolution is, being able to reduce the number of packages you install in your CI/CD pipeline can shave a lot of time off each build.

The second use case, which led me down a rabbit hole until I landed here, is... related to a conversation you and I had almost a year ago! (Wow, I honestly didn't notice either me or you in that thread until I started writing this comment.) Here's my example:

• I have a library, xpring-py, that depends on a library, fastecdsa, that requires libgmp to be installed on the system.
• libgmp is not a dependency available in PyPI, which means I cannot install it into the ReadTheDocs (RTD) build environment.
• These days RTD installs dependencies with pip, but if it could use Poetry, and if Poetry could exclude dependencies via --without, then I could use Poetry to install all the installable dependencies in the RTD environment and let autodoc mock the rest.

2. I think it's a good place to start if dependencies can only exist in one group. It will be easy to extend from one group to many groups if a use case arises, but it will be a breaking change to go the other direction.

3. However, if we must have multiple groups, then given:

group-dep = { version = "*", groups = ["hello", "world"] }
optional-dep = { version = "*", groups = ["hello", "world"], optional = true }

here is the behavior I would expect:

poetry install # group-dep in, optional-dep out
poetry install --with hello # both in
poetry install --with world # both in
poetry install --with hello,world # both in
poetry install --without hello # group-dep in, optional-dep out
poetry install --without world # group-dep in, optional-dep out
poetry install --without hello --with world # both in
poetry install --without world --with hello # both in
poetry install --without hello,world # both out

Let me just remark that the Ruby semantics are very poorly documented, but let me try to take their rule and tweak it to match these semantics as simply and concisely as possible:

Install all gems, except those that are optional or in the --without groups, but including those in the --with groups. Gems in at least one non-excluded group will still be installed.

I hope this is intuitive.

[kkozmic]

This issue is pretty important to me.
Is there any way I can help move it forward?
Is it something that the maintainers see as a priority?

[matthijskooijman]

How would this feature relate to the "extras" feature? Would this be able to replace extras, or is there something that extras provides that these groups could not? It seems like groups would be somewhat of a superset? The way you specify what extras/groups are installed is different, but once you select the same set of extras/groups to install, would there be other differences?

It seems that extras are not currently really working or documented well (#1145, #1076), so maybe this could be replacement for extras with clearer and more complete semantics?

Note that where I say "extras" above, I refer to the "clusters of dependencies"-interpretation of that, not the "extras to apply to dependencies"-interpretation. See #1076 (comment) for some discussion of the difference. I believe the "extras to apply to dependencies"-interpretation is clearer and would not need to be replaced.

[Cielquan]

How would this feature relate to the "extras" feature? Would this be able to replace extras, or is there something that extras provides that these groups could not? It seems like groups would be somewhat of a superset? The way you specify what extras/groups are installed is different, but once you select the same set of extras/groups to install, would there be other differences?

It seems that extras are not currently really working or documented well (#1145, #1076), so maybe this could be replacement for extras with clearer and more complete semantics?

I would not replace the extras system but add the group system on top.

Reason: (Blow I added a shorted version of pyproject.toml like I use and like I want to use.)

So the thing is that I use the extras system to split my development dependencies into categories. Then I can install mypackage[docs] into the venv which tests and builds my docs. I can install mypackage[testing] into the venvs testing my code. etc.
Of cause I could add the testing, docs and pre-commit dependencies directly to the dev dependencies, but then each and every venv created for testing will have multiple dependencies it does not need. This increases time for installation and space. (I use tox for automation)

So I think a good idea would be to keep the [tool.poetry.extras] section for extras meant to enhance usability of your package to the user like toml for coverage to enable pyproject.toml as config source. The group feature could then be a development thing to split up development dependencies for certain tasks.

And I would only allow group assignments for optional dependencies. I see no use for the feature for mandatory dependencies.

Currently:

...
    [tool.poetry.dependencies]
        python = "^3.6.1"
        importlib-metadata = {version = "^1.6", python = "<3.8"}
        mandatory-dep = {version = "^1.5.3", extras = ["some-extra"]}
        enhancing-dep = {version = "^5.1.1", optional = true}
        # Testing
        pytest = {version = "^5.4.2", optional = true}
        # Docs
        sphinx = {version = "^3", optional = true}
        # Code check
        pre-commit = {version = "^2.4", optional = true}
        mypy = {version = "0.770", optional = true}
        pylint = {version = "^2.4", optional = true}

    [tool.poetry.dev-dependencies]
        devtools = {version = "^0.5", extras = ["pygments"]}

    [tool.poetry.extras]
        better-pkg-usability = ["enhancing-dep"]
        testing = ["pytest"]
        docs = ["sphinx"]
        pre-commit = ["pre-commit", "mypy", "pylint"]
...

How I would like it:

...
    [tool.poetry.dependencies]
        python = "^3.6.1"
        importlib-metadata = {version = "^1.6", python = "<3.8"}
        mandatory-dep = {version = "^1.5.3", extras = ["some-extra"]}
        enhancing-dep = {version = "^5.1.1", optional = true}
        # Testing
        pytest = {version = "^5.4.2", optional = true, group = ["testing"]}
        # Docs
        sphinx = {version = "^3", optional = true, group = ["docs"]}
        # Code check
        pre-commit = {version = "^2.4", optional = true, group = ["pre-commit"]}
        mypy = {version = "0.770", optional = true, group = ["pre-commit"]}
        pylint = {version = "^2.4", optional = true, group = ["pre-commit"]}

    [tool.poetry.dev-dependencies]
        devtools = {version = "^0.5", extras = ["pygments"]}

    [tool.poetry.extras]
        better-pkg-usability = ["enhancing-dep"]
...

With this a user could install the extras to enhance usability like always: pip install mypackage[better-pkg-usability].
And for the docs testing env I could run poetry install --with docs and so on.

Note that where I say "extras" above, I refer to the "clusters of dependencies"-interpretation of that, not the "extras to apply to dependencies"-interpretation. See #1076 (comment) for some discussion of the difference. I believe the "extras to apply to dependencies"-interpretation is clearer and would not need to be replaced.

Both are extras but only viewed from another perspective. The "clusters of dependencies"-extra for your project is an "extras to apply to dependencies"-extra for someone else using your package.

Also in setuptools the first is specified in extras_require section and the latter just inside [] after the dependency in install_requires section.

PS: The extras for dependencies like this devtools = {version = "^0.5", extras = ["pygments"]} are fine as they are IMHO.

[fredrikaverpil]

Oh yes, this is exactly what I've been missing in poetry. Groups as well as individual deps that I need to install on their own or together with the production code.

I use poetry to setup multiple tox environments with isolated builds in tox.ini, and I currently need to wait for ~100 deps to be installed for each (!!!) tox env. The total installation time alone is many times longer than what it takes to run all tests. This feature would cut a large chunk out of the total wait time, and would probably be more valuable to me than speeding up the general installation time, which is something I understand is being worked on at the moment.

[SimonBiggs]

I really liked this idea so I implemented it as a temporary work around until it lands in poetry actual.

The idea, is you flag the groups with a comment:
https://github.com/pymedphys/pymedphys/blob/ec837162da76016223e59ce0ab121147a00d0329/pyproject.toml#L33-L35

And then the following code takes those comments and writes them into tool.poetry.extras:
https://github.com/pymedphys/pymedphys/blob/ec837162da76016223e59ce0ab121147a00d0329/scripts/propegate-extras.py#L25-L51

[matthijskooijman]

Re-reading this discussion, I am still wondering what the essential difference between these new "groups" and the existing "extras" are? It seems that both are just specifying sets (I'm intentionally not saying "groups" here, to emphasize I'm talking about the generic concept of sets rather than the proposed "groups" feature) of dependencies, the difference seems to be only in the way these sets are addressed an their default.

@Cielquan wrote:

So I think a good idea would be to keep the [tool.poetry.extras] section for extras meant to enhance usability of your package to the user like toml for coverage to enable pyproject.toml as config source. The group feature could then be a development thing to split up development dependencies for certain tasks.

What I read there is that "extras" are things that you can add to your dependencies (default omitted) and "groups" are things that you can omit from your (dev) dependencies (default included).

The implementation @SimonBiggs follows a similar flow, where "groups" are things in the default dependency list that are annotated, resulting in an entry in the "extras" section as well.

So I wonder if maybe the "extras" feature should just be somewhat
generalized. It already allows specifying sets, so what if we would
enhance extras with a way to make extras installed by default? And then
add a -with and --without option to include or exclude these sets?
Since extras dependencies are currently already specified with their
version and optional=true in the main dependencies list and the name is
duplicated in the extras section, making installed-by-default extras
could be a matter of just omitting the optional tag (which I think is
pretty much what optional does right now, if it's false or omitted, the
package is installed by default, otherwise only when requested).

E.g. taking the example from @Cielquan, that could become as below. Note
that I moved a bunch of dependencies to dev-dependencies, as I think
that's where they would belong?

[tool.poetry.dependencies]
    python = "^3.6.1"
    importlib-metadata = {version = "^1.6", python = "<3.8"}
    mandatory-dep = {version = "^1.5.3", extras = ["some-extra"]}
    enhancing-dep = {version = "^5.1.1", optional = true}

[tool.poetry.dev-dependencies]
    devtools = {version = "^0.5", extras = ["pygments"]}
    # Testing
    pytest = {version = "^5.4.2"}
    # Docs
    sphinx = {version = "^3"}
    # Code check
    pre-commit = {version = "^2.4"}
    mypy = {version = "0.770"}
    pylint = {version = "^2.4"}

[tool.poetry.extras]
    better-pkg-usability = ["enhancing-dep"]
    pre-commit = ["pre-commit", "mypy", "pylint"]
    docs = ["sphix"]
    testing = ["pytest"]

However, this does end up with the information spread out, so now I've
writen it down, I'm not so sure I like this approach much. I can also
imagine using the "groups" attribute, as suggested before, and then
still use "optional" to indicate installed-or-omitted by default. This
could become something like:

[tool.poetry.dependencies]
    python = "^3.6.1"
    importlib-metadata = {version = "^1.6", python = "<3.8"}
    mandatory-dep = {version = "^1.5.3", extras = ["some-extra"]}
    enhancing-dep = {version = "^5.1.1", optional = true, groups = ["better-pkg-usability"]}

[tool.poetry.dev-dependencies]
    devtools = {version = "^0.5", extras = ["pygments"]}
    pytest = {version = "^5.4.2", groups = ["testing"]}
    sphinx = {version = "^3", groups = ["docs"]}
    pre-commit = {version = "^2.4", groups = ["pre-commit"]}
    mypy = {version = "0.770", groups = ["pre-commit"]}
    pylint = {version = "^2.4", groups = ["pre-commit"]}

Compared to the original example by @Cielquan, this:

• Renames "group" to "groups", since it contains a list.
• Removes the "optional=true" from all dependencies that should be installed by default (e.g. all but enhancing-dep).
• Moves the dev dependencies to the dev-dependencies section, so they get installed by default without --no-dev.
• Removes the "extras" section, in favor of adding a "groups" field on the ehancing dep.

The idea here is that everything in the dependencies section is installed always, except for the enhancing-dep which needs --with better-pkg-usability (or mypackage[better-pkg-usability]). Everything in dev-dependencies is installed when --no-dev is omitted, but can be granularly chosen using e.g. --without pre-commit.

Essentially, this would mean that "extras" and "groups" are just the same thing. More specifically:

• Any dependency foo that has optional=true and group=bar is equivalent to specyfing bar = ["foo"] in the extra section (and vice versa).
• Running poetry install --with foo installs all dependencies that include foo in their groups (iow, ignores "optional=true" for those dependencies).
• Running poetry install --without foo omits all dependencies that include foo in their groups (iow, adds "optional=true" for those dependencies).
• Installing mypackage[foo] should do the same as running poetry install --with foo in the mypackage directory.
• It would be nice of installing mypackage[no-foo] would be equivalent to running poetry install --without-foo, but I suspect that this requires changes to setuptools or what not.

One thing that still needs some thought is how --with and --without interact when both are specified and match a single dependency (i.e. which gets precedence?)

This seems to offer a generic group feature, that cleanly maps onto the existing "extras" mechanism (as exposed to other packaging tooling such as pip), without duplicating information.

One could even go one step further and say that dev is just a group. So:

• Any dependencies in the dev-dependencies section are equivalent to regular dependencies with the "dev" group added.
• poetry install --no-dev is equivalent to poetry install --without dev
• poetry install is equivalent to poetry install --with dev

There might be some value in still storing dev dependencies in a separate section of the pyproject.toml file as now (easier to distinguish them), though just having a single list with explicit groups = ["dev"] would make it more obvious that dev deps are just a group.

As a last extra step, you could consider allowing negated groups, e.g.:

    some-package = {version = "^1.5.3", groups = ["!foo"]}

This would be a package that is normally installed, unless --with foo is
given. I can typically see this used for production-only dependencies,
something like:

    some-production-package = {version = "^1.5.3", groups = ["!dev"]}

This would be installed only when --no-dev or --without dev is specified
(which is slightly different from the foo example above, but that's because
--with dev is usually implied by poetry install.

Maybe this negation is one step too far and you should just add a "production"
group for this, though.

[SimonBiggs]

Re-reading this discussion, I am still wondering what the essential difference between these new "groups" and the existing "extras" are?

So, within a given production dependency definition currently an "extra" parameter is allowed. When that is defined within the dependency parameters it refers to a pypi extra for that dependency, not for your package, eg. alldeps within scikit-learn refers to an extra of scikit-learn, not your package.

Let's say, your package is called "mypackage", and we want our package to have an optional dependency of scikit-learn within a group called ml (short for machine learning). And we want to make sure that when scikit-learn installs it installs with all of its own dependencies. I would expect this would look like:

[tool.poetry.dependencies]
scikit-learn = {version = "*", extras = ["alldeps"], groups = ["ml"]}

That way, running pip install mypackage[ml] would also install scikit-learn, and when scikit-learn is installed it would be installed as if pip install scikit-learn[alldeps] had been run.

Importantly, when groups is defined, I believe this implies an implicit optional=true. So optional should actually be replaced by the groups parameter, and no longer used.

[matthijskooijman]

Re-reading this discussion, I am still wondering what the essential difference between these new "groups" and the existing "extras" are?

So, within a given production dependency definition currently an "extra" parameter is allowed. When that is defined within the dependency parameters it refers to a pypi extra for that dependency, not for your package, eg. alldeps within scikit-learn refers to an extra of scikit-learn, not your package.

Yeah, I realize that that is what the extras in the tools.poetry.dependencies section mean, but I was referring to the other use of extras (i.e. the tool.poetry.extras section). But I think there is no essential difference between the proposed "groups" and the existing "extras" in this sense, except that the "groups" feature would be a feature superset of the existing "extras" (in that everything you can do with tool.poetry.extras can also be expressed using groups, and with my proposal above, one can in fact be mapped onto the other).

Importantly, when groups is defined, I believe this implies an implicit optional=true. So optional should actually be replaced by the groups parameter, and no longer used.

I do not think this is true. If you do this, then this "groups" concept becomes again the same as the existing "extras" concept, except with a different name. If you do this, you lose the ability to have dependencies that have a group set and are installed by default, but can be omitted using poetry install --without some-group (unless you require an explicit optional=false for this case, but that would be confusing IMHO).

It might be true that optional=true is useless without also specifying one or more groups (since IIUC, optional=true is not installed by default, and without a group, there is no way to cause such a dependency to become used).

[pawamoy]

@matthijskooijman I really like what you described above. I think it's very sensible.

• "group" dependencies can be written into the dev-dependencies section instead of the dependencies one (or both): it's less confusing (to me)
• the tool.poetry.extras section is automatically populated from groups: easier, less error-prone
• grouped dependencies don't have to be optional anymore, meaning new contributors don't have to read the pyproject.toml file to know what extras they should install using poetry install -E .... Everything just gets installed, like when doing poetry install.

I like it!

[matthijskooijman]

"group" dependencies can be written into the dev-dependencies section instead of the dependencies one (or both): it's less confusing (to me)

Do you mean "dependencies in the dev group" here? Or maybe I do not understand what you say exactly?

the tool.poetry.extras section is automatically populated from groups: easier, less error-prone

Do you mean that something would actually write them to the tool.poetry.extras section in the pyproject.toml file? Or just that this happens virtually, in memory, by poetry when interpreting the file?

[SimonBiggs]

I was referring to the other use of extras (i.e. the tool.poetry.extras section). But I think there is no essential difference between the proposed "groups" and the existing "extras" in this sense

Yup, I was not intending there to be any difference in the way I was wanting to use them. The aim would be to minimise inherent duplication of the package names and make a cleaner config API.

I personally was not proposing any feature changes, instead just what I see as an improved API.

I found manually managing the following lists was tedious and error prone, this is in contrast to adding group flags which I found to be simple and hard to make a mistake:

https://github.com/pymedphys/pymedphys/blob/ec837162da76016223e59ce0ab121147a00d0329/pyproject.toml#L98-L106

[pawamoy]

Do you mean "dependencies in the dev group" here? Or maybe I do not understand what you say exactly?

What I mean is that currently, to define extras, you have to put the dependencies in the tool.poetry.dependencies section, and set them as optional. You can also duplicate them in dev-dependencies if you want them to be installed by poetry install. With this feature, one could put the dependencies in tool.poetry.dev-dependencies instead.

Do you mean that something would actually write them to the tool.poetry.extras section in the pyproject.toml file? Or just that this happens virtually, in memory, by poetry when interpreting the file?

The latter, it would happen in memory, while building the wheel, etc., the pyproject.toml file would not be modified.

[Cielquan]

After some time and based on the discussion above from a week ago I changed my idea:

Currently we have the dev section with dev dependencies and the normal dependency section which includes the package's dependencies and optional dependencies which can be included via the extras section. IMO the extras section should be used to add additional dependencies which enable some additional functionality for the package and NOT the development. But currently the extras are misused (by me e.g.) to also group dev dependencies. I use this to not always install e.g. the linter tools when I run unit tests (with tox) like I mentioned earlier.

So the primary thing I would like to achieve with the "group" feature is to add an extra like feature to the dev dependencies also.

With this you have the normal dependency section including the must have dependencies for the package to run and the optional ones which can be included via the extras like it is now. No change needed here.
The dev dependency section would be allowed to also have optional dependencies in it. Those would be, just like the extras from the normal section, grouped in another section like tool.poetry.groups to separate those from the package enhancing dependencies in the tool.poetry.extras section.
So this would be just a copy of the extras functionality to the dev dependencies to split them from the runtime dependencies, which is my personal primary goal here. This change shouldn't be that much work because it just copies existing behavior.

This basic change/step can be enhanced in multiple ways:

1. Change group behavior to opt-out instead of opt-in
   The extras for the runtime deps are opt-in and need to be explicitly named to be install, like it should be. But for the dev-dependencies I think it should be the other way around and the groups should be a opt-out feature. The reason for this is that if a new developer joins the team who likes to call pytest, linter etc. by hand instead of using for example a makefile or tox he/she would use only one venv I think. And to install all needed dependencies into this venv for developing the package he/she would have to look into the pyproject.toml file and add all the optional dev deps (groups) to the install command to have all the tools installed. So from a general perspective I think it would be good to install all dev dependencies and only opt-out some if explicitly said so in the command.
   This is a change I would recommend and like to see.

2. Relocate the groups definition (dev dependencies not runtime dependencies)
   So next thing could be to remove/relocate the tool.poetry.group section into the dev dependencies, because in conjunction with No. 1 all dev deps are installed by default and there is no optional=true anymore. So one could directly set the group(s) which should include the dependency at the dependency itself instead of writing them in a designated section. This is a design decision because you could say for both ways that they are more clear and better to read.
   I personally am fine with both ways.

3. Relocate the extras definition (runtime dependencies not dev dependencies)
   Next one could also remove/relocate the tool.poetry.extras section into the dependencies, like in No.2, because all runtime deps which are extras are also optional=true. So one could directly set the extras which should include the dependency instead of writing optional=true. This is also a design decision because you could say for both ways that they are more clear and better to read.
   I personally am fine with both ways.

4. Opt-in dev dependencies
   Also possible would be to leave the option to set optional=true in the dev dependency section to make a dev dependency opt-in counter to the default behavior of opt-out (see No.1). But I personally see no real use case for this and I think it would over-complicate things.
   Change my mind.

DISCLAIMER: The ideas above are not all mine but also taken from earlier comments. I just combined them to something I would like to see.

[ScriptAutomate]

I'm glad to see this is on the poetry roadmap for v1.2! 🚀

Could groups also influence the poetry build target, not just poetry install?

I'd be interested in that. too, and same with poetry export (would help provide easier, compatible support for tools like safety, etc.).

Each of these targets influenced would considerably help dev workflows I've been involved in.

For example, here is a PR submitted for salt in order to support requirements files generated for multiple versions of Python, multiple different OS targets that involve differences in the dependency trees (such as Windows reqs vs. Linux), docs-only-related pipelines, etc. It becomes a lot to work with. The best approach has been to work with tools like pip-compile / pip-tools in order to properly manage dependency trees, but poetry support for groups of dependencies locked, isolated, and targeted at-will would be huge.

[suryaavala]

the ability to add environments/targets as proposed would be massively helpful in ML projects imo!

we usually strip out quite a few of the requirements when serving models that would have been otherwise required during training. i reckon the traditional way for separating requirements prod and dev is not always feasible for ML projects.

[adriangb]

Thank you for getting this done @sdispater !

Any plans to release another pre-release with the feature? My team has a pretty convoluted setup (not using Poetry) that could be simplified greatly with this feature, I'd love to migrate to Poetry.

[matthijskooijman]

Great to see this feature is now available, thanks. Also great that this neatly generalizes the dev-dependencies in a compatible way :-)

I've read through the documentation, and am left with a few questions:

1. After reading this section, I wonder: What's the difference between poetry install --default and a plain poetry install? Does the former install only dependencies, and the latter also the package itself? Or is there another reason for --default to exist?
2. How do dependency groups interact with extras? From the docs, it seems that these two features now coexist but are completely separate? Wouldn't it be possible to also integrate these? Essentially, AFAICS, extras are just optional dependency groups, so it would be nice if they could be treated (and specified) as such? That would mean that --with and --extras become essentially the same option (--extras could be deprecated maybe), and extras could be specified using the new dependency group syntax (and the [tool.poetry.extras] section and optional attribute on individual dependencies could also become deprecated, I think). I might be missing some subtle differences that prevent this, though?

[FichteFoll]

Regarding point 1, the current wording tells me that plain poetry install installs the default non-grouped dependencies plus the dependencies of all groups. Now, I would also presume that it does not install optional groups, but the wording of

By default, dependencies across all groups will be installed when executing poetry install.

(emphasis not mine) seems conflicting to me with the following statement:

You can also opt in optional groups by using the --with option:

[jaymegordo]

+1 for @matthijskooijman 's point 2. Would be very nice to be able to add another poetry-managed package using its groups in place of extras.

Currently it seems you have to re-define (bad) the groups as extras in order to use them as extras when installing in another package.

[eltbus]

Would it be possible to add this functionallity to poetry export so that multiple requirements can be created?

Something like:

poetry export -f requirements.txt --output requirements.txt
poetry export -f requirements.txt --without-groups=* --with-groups=test --output requirements-test.txt
poetry export -f requirements.txt --without-groups=* --with-groups=docs --output requirements-docs.txt

[edgarrmondragon]

@eltbus The export plugin already supports that: https://github.com/python-poetry/poetry-export-plugin#available-options.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.