Support installing Poetry dependency groups

[johnthagen]

Poetry 1.2 will be adding added support for dependency groups. It would be ergonomic if you could install a dependency group (as a whole) into a nox session.

• https://python-poetry.org/docs/managing-dependencies/#dependency-groups

Consider a Poetry group:

[tool.poetry.group.lint.dependencies]
flake8 = "*"
flake8-bugbear = "*"
flake8-broken-line = "*"
flake8-comprehensions = "*"
pep8-names = "*"

The cooresponding nox session could be simplified to:

def test(session: nox_poetry.Session):
    session.install(".", "pytest", poetry_groups=["lint", "type_check"])
    ...

This avoids the DRY problem of listing all of the lint dependencies twice, once in a Poetry group and again in a nox session.

Related to

• Warnings with Poetry 1.2.0 #873
• nox-poetry does not respect the new dependency groups from Poetry #977

[johnthagen]

May be a duplicate of

• Poetry dependency groups #481

[johnthagen]

@thedrow You mentioned at python-poetry/poetry#1856 (comment) wanting to use Poetry dependency groups with nox-poetry. I'm curious, is what I describe close to what you were thinking, or were you imagining something else?

[thedrow]

Yup. Exactly.

[edgarrmondragon]

This would be useful for my projects. I can start a PR if there's interest.

[thedrow]

@edgarrmondragon Go ahead :)

[johnthagen]

poetry export supports --only

poetry export --help

Description:
  Exports the lock file to alternative formats.

Usage:
  export [options]

...

--only=ONLY         The only dependency groups to include. (multiple values allowed)

So I think this would be a relatively easy way to get a requirements.txt file out for a specific group.

[johnthagen]

Thinking about this some more, perhaps a better API for this that would allow users to mix the current functionality with groups in a single command could use a name-only argument:

def test(session: nox_poetry.Session):
    session.install(".", "pytest", poetry_groups=["lint", "type_check"])

That way a single command can used locked dependencies from any source available.

[cjolowicz]

If you have dependency groups, why not use plain Nox with a small helper like this?

https://github.com/cjolowicz/cookiecutter-hypermodern-python-instance/blob/90e45cee4725c51c77864ab21efead23d5226306/noxfile.py#L26-L51

I'd be curious what value you see in nox-poetry given that you can now easily install locked dependency groups using just Nox and Poetry >=1.2?

[edgarrmondragon]

@cjolowicz I'm probably missing something from your example but how do you ensure that poetry install targets the Nox session's virtual environment?

[cjolowicz]

Poetry honors the VIRTUAL_ENV environment variable, which is set by Nox when it runs processes in the session.

(At least, it did when I wrote this code 😅 )

[johnthagen]

The first time I tried this (on Windows with latest Poetry/Nox) using the install() function in your comment, the invoked Poetry started messing with the outer Poetry environment:

# noxfile.py
...
@session
def fmt(s: Session) -> None:
    install(s, groups=["fmt"], root=False)
    s.run("isort", ".")
    s.run("black", ".")

It started to remove a bunch of packages in the outer environment due to --sync:

> nox -s fmt
nox > Running session fmt
nox > Re-using existing virtual environment at .nox\fmt.
nox > poetry install --no-root --sync --only=fmt
Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 58 removals

  • Removing argcomplete (2.0.0)
  • Removing attrs (22.2.0)
  • Removing beautifulsoup4 (4.11.2)
  • Removing certifi (2022.12.7)

I think this demonstrates that relying on VIRTUAL_ENV being set/passed correctly to a bare poetry install command can be at least brittle/unreliable on some platforms.

[Niicck]

If you have dependency groups, why not use plain Nox with a small helper like this?

https://github.com/cjolowicz/cookiecutter-hypermodern-python-instance/blob/90e45cee4725c51c77864ab21efead23d5226306/noxfile.py#L26-L51

I'd be curious what value you see in nox-poetry given that you can now easily install locked dependency groups using just Nox and Poetry >=1.2?

I like nox-poetry's workflow of exporting requirements and then installing them with pip, I don't want to install with poetry directly into my nox sessions. For now I'm just copy/pasting a basic helper function to handle installs:

def install_poetry_groups(session, *groups: str) -> None:
    """Install dependencies from poetry groups."""
    with tempfile.NamedTemporaryFile() as requirements:
        session.run(
            "poetry",
            "export",
            *[f"--only={group}" for group in groups],
            "--format=requirements.txt",
            "--without-hashes",
            f"--output={requirements.name}",
            external=True,
        )
        session.install("-r", requirements.name)

But it would be nice to have that logic handled in a common package, and it seems like nox-poetry is a good candidate to house it. My copy/paste function doesn't have nox-poetry's test coverage or nice "Warning:" message filter, so for me there's value to using it. But to your point, my current workflow is to just use plain nox with a helper function.

[johnthagen]

seems like nox-poetry is a good candidate to house it

That was my thought as well. We have dozens of Poetry/Nox projects and having a shared package where we know others are also collaborating/testing/etc. has been a great way to avoid code duplication and engage with others using a similar workflow. For that nox-poetry has been great ❤️ .

[cjolowicz]

The first time I tried this (on Windows with latest Poetry/Nox) using the install() function in your comment, the invoked Poetry started messing with the outer Poetry environment:

This looks like a Poetry bug. Can you provide instructions to reproduce this?

To my knowledge, poetry install respects $VIRTUAL_ENV. For example, try the following inside docker run -ti python bash:

pip install pipx
pipx install poetry
export PATH=$HOME/.local/bin:$PATH
poetry new project
cd project
poetry add -G test pytest
poetry add -G docs sphinx
python -m venv /tmp/venv
. /tmp/venv/bin/activate
poetry install --sync --only docs
deactivate
poetry install

You should see Poetry install the docs group into the virtual environment. Outside the virtual environment, the last poetry install command reports: No dependencies to install or update.

[johnthagen]

Can you provide instructions to reproduce this?

• Platform: Windows 10
• Python: 3.10.7
• Poetry: 1.3.2
• Shell: Powershell

> poetry show --tree
black 23.1.0 The uncompromising code formatter.
├── click >=8.0.0
│   └── colorama *
├── mypy-extensions >=0.4.3
├── packaging >=22.0
├── pathspec >=0.9.0
├── platformdirs >=2
└── tomli >=1.1.0
isort 5.12.0 A Python utility / library to sort Python imports.
nox 2022.11.21 Flexible test automation.
├── argcomplete >=1.9.4,<3.0
├── colorlog >=2.6.1,<7.0.0
│   └── colorama *
├── packaging >=20.9
└── virtualenv >=14
    ├── distlib >=0.3.6,<1
    ├── filelock >=3.4.1,<4
    └── platformdirs >=2.4,<4
pydantic 1.10.5 Data validation and settings management using python type hints
└── typing-extensions >=4.2.0

[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = []
readme = "README.md"
packages = [{include = "poetry_test"}]

[tool.poetry.dependencies]
python = "^3.10"
pydantic = "*"

[tool.poetry.group.nox.dependencies]
nox = "*"

[tool.poetry.group.fmt.dependencies]
black = "*"
isort = "*"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

from typing import Iterable

import nox


def install(session: nox.Session, *, groups: Iterable[str], root: bool = True) -> None:
    """Install the dependency groups using Poetry.
    This function installs the given dependency groups into the session's
    virtual environment. When ``root`` is true (the default), the function
    also installs the root package and its default dependencies.
    To avoid an editable install, the root package is not installed using
    ``poetry install``. Instead, the function invokes ``pip install .``
    to perform a PEP 517 build.
    Args:
        session: The Session object.
        groups: The dependency groups to install.
        root: Install the root package.
    """
    session.run_always(
        "poetry",
        "install",
        "--no-root",
        "--sync",
        "--{}={}".format("only" if not root else "with", ",".join(groups)),
        external=True,
    )
    if root:
        session.install(".")


@nox.session
def fmt(s: nox.Session) -> None:
    install(s, groups=["fmt"], root=False)
    s.run("isort", ".")
    s.run("black", ".")

> poetry install --sync
> poetry shell
(poetry-test-py3.10) > nox -s fmt
nox > Running session fmt
nox > Creating virtual environment (virtualenv) using python.exe in .nox\fmt
nox > poetry install --no-root --sync --only=fmt
Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 8 removals

  • Removing argcomplete (2.0.0)
  • Removing colorlog (6.7.0)
  • Removing distlib (0.3.6)
  • Removing filelock (3.9.0)
  • Removing nox (2022.11.21)
...
PermissionError: [WinError 5] Access is denied:

The run_always("poetry install") is uninstalling packages from the parent Poetry venv (in fact even tries to uninstall nox itself 😅).

So note that nox is already running from a poetry shell and now the poetry install run_always() is trying to work within the parent Poetry environment, not the venv that nox just made.

nox-poetry avoids any of these potential issues because it invokes the correct venv's pip directly (as I understand it).

[cjolowicz]

@johnthagen Does this reproduce when you don't have Nox as a development dependency?

I recommend installing both Nox and Poetry using pipx, side by side. You're running Nox from a Poetry environment, and then Poetry from a Nox environment. Theoretically, it should work, but it's unneeded complexity IMO. I'd also avoid constraining the dependency tree of your application with tools that can be part of the global dev environment.

FWIW, I tried your example in a Windows runner on GHA, but it didn't reproduce in that environment. Here's an example run: https://github.com/cjolowicz/poetry-test/actions/runs/4801627813/jobs/8544094456

[johnthagen]

Theoretically, it should work, but it's unneeded complexity IMO.

In our case, we want to lock the exact version of Nox (and nox-poetry) because it is part of our CI process and thus we want it to be reproducible.

In my experience it also simplifies the setup for each dev if when they run poetry install it brings everything they need, including nox, nox-poetry, etc. That way, the only thing they need to have installed is Poetry itself, to bootstrap everything else.

I'd also avoid constraining the dependency tree of your application with tools that can be part of the global dev environment.

I don't disagree in principle (perhaps one day Hatch or something like it can replace the need for Poetry and put everything in its own environment), but for now to make the workflow simple using Poetry (which chooses to lock everything together) and reproducible we are already constraining our environment with a host of other tools we need to lock and have reproducible versions (mypy, pytest, black, isort, ruff, etc.).

[johnthagen]

A concrete example in recent past of why it's important to pin Nox is when Nox changed its behaviour in the presence of the CI environment variable:

• Missing interpreters now error the session on CI by default wntrblm/nox#567

Without Nox being pinned, this would have broken CI builds that depended on the previous behaviour. An example fix:

• Adapt for default behaviour change in nox pypa/get-pip#172

So our teams insist on locking as much of the development tools as possible and keeping both developer tools and CI in sync because things like this are bound to come up as FOSS projects evolve.

[ndjenkins85]

Just wanted to share a workaround i'm using to parse the toml for the groups

from typing import List
from pathlib import Path
import toml

def install_dependency_groups(session, groups: List[str] = []):
    """Manually parse the pyproject file to find group(s) of dependencies, then install."""
    pyproject_path = Path("pyproject.toml")
    data = toml.load(pyproject_path)
    all_dependencies = []
    for group in groups:
        dependencies = data.get("tool", {}).get("poetry", {}).get("group", {}).get(group, {}).get("dependencies", {})
        all_dependencies += list(dependencies.keys())
    all_dependencies = list(set(all_dependencies))
    session.install(*all_dependencies)

...

install_dependency_groups(session, groups=["lint", "tests"])

[edgarrmondragon]

Just wanted to share a workaround i'm using to parse the toml for the groups

from typing import List
from pathlib import Path
import toml

def install_dependency_groups(session, groups: List[str] = []):
    """Manually parse the pyproject file to find group(s) of dependencies, then install."""
    pyproject_path = Path("pyproject.toml")
    data = toml.load(pyproject_path)
    all_dependencies = []
    for group in groups:
        dependencies = data.get("tool", {}).get("poetry", {}).get("group", {}).get(group, {}).get("dependencies", {})
        all_dependencies += list(dependencies.keys())
    all_dependencies = list(set(all_dependencies))
    session.install(*all_dependencies)

...

install_dependency_groups(session, groups=["lint", "tests"])

Nox 2024.04.15 also has nox.project.load_toml:

data = nox.project.load_toml(pyproject_path)

https://nox.thea.codes/en/stable/tutorial.html#loading-dependencies-from-pyproject-toml-or-scripts

So, one fewer import 😅

[MatthieuOdeArturia]

I improve the previous workaround to install extras from pyproject.toml:

[tool.poetry.group.nox_xdoctest.dependencies]
xdoctest = { version = "*", extras = ["colors"] }

In noxfile.py:

def install_dependency_groups(session, groups: List[str]):
    """Manually parse the pyproject file to find group(s) of dependencies, then install."""
    if not groups:
        return

    pyproject_path = Path("pyproject.toml")
    data = nox.project.load_toml(pyproject_path)
    all_dependencies = []

    for group in groups:
        dependencies = (
            data.get("tool", {})
            .get("poetry", {})
            .get("group", {})
            .get(group, {})
            .get("dependencies", {})
        )

        for name, details in dependencies.items():
            # Check if the dependency has extras
            if isinstance(details, dict) and "extras" in details:
                extras = details["extras"]
                extras_str = ",".join(extras)
                all_dependencies.append(f"{name}[{extras_str}]")
            else:
                all_dependencies.append(name)

    # Remove any duplicate dependencies
    all_dependencies = list(set(all_dependencies))
    session.install(*all_dependencies)