Frequent "Requested groups not in lockfile" errors.

[warsaw]

• I have searched the issue tracker and believe that this is not a duplicate.

Despite #1796 and #2050 I think this is a different bug. It definitely occurs for more than just the default group and optional dependencies. E.g. on a $work PR, I see the missing group is testing. I've also seen this in the past with some of my OSS projects. pdm lock doesn't seem to solve it.

Steps to reproduce

Unfortunately, I don't have a reliable reproducer. Sometimes this happens when local testing passes, but CI testing fails. Sometimes local testing also fails. Sometimes it happens after pdm update.

The only reliable way to resolve the problem is to rm -f pdm.lock (and note that pdm.lock is committed to git), then re-run my tests which will have the effect of recreating a usable pdm.lock file. I haven't come up with another way to fix the problem.

Actual behavior

I'll see an error like [PdmUsageError]: Requested groups not in lockfile: testing

Expected behavior

I have a testing group in my [tool.pdm.dev-dependencies] section, so I would expect PDM to be able to execute the pdm install -G testing command in my tox.ini testenv.

Environment Information

# Paste the output of `pdm info && pdm info --env` below:

Sorry, I can't paste that directly because of $work, but I am using PDM 2.7.4 installed from brew and Python 3.8.17, but I have definitely seen it with newer Pythons on my OSS work.

[warsaw]

Okay, I have a reproducer! But now I wonder if this is intended behavior or not. I'd argue that it makes for confusing UX.

% git clone https://gitlab.com/warsaw/flufl.enum.git
Cloning into 'flufl.enum'...
% cd flufl.enum/
% grep groups pdm.lock
summary = "Backport of PEP 654 (exception groups)"
groups = ["default", "docs", "qa", "testing"]
% pdm lock -G qa
🔒 Lock successful
Changes are written to pdm.lock.
% grep groups pdm.lock
groups = ["default", "qa"]

Notice how pdm lock -G qa drops the docs and testing groups from the lock file. pdm lock -G :all restores all the groups. I'm sure there are other scenarios where those groups get dropped, since I rarely use pdm lock -G.

[frostming]

pdm lock, by its intention, is to regenerate the lockfile with what groups are given by the CLI. If you want to add instead of rewrite, use pdm add -G qa. I don't think it is confusing.

[warsaw]

Okay, that kind of makes sense, although it's a little less intuitive that pdm add -G <group> can be useful without specifying any packages. That said, I wonder if there are situations where pdm lock -G <group> is being called under the hood? It's rare that I explicitly pdm lock -G <group> and yet still get this error.

[warsaw]

Maybe, the following is related? I added some dependencies to a docs group in my [tool.pdm.dev-dependencies] section. I did this by manually editing pyproject.toml. Then I ran tox which eventually runs pdm install -G docs. now my testing group got deleted from pdm.lock.

[frostming]

That seems a bug, thanks

[Vinno97]

This feels unintuitive. I get the usefullness in certain cases, but I feel the majority of users will use pdm lock to regenerate their lock file when, for example, it becomes corrupt. Wouldn't it be useful to have it include all groups by default?

[rafalkrupinski]

What's the reason to not pin dependencies from all groups by default?

[frostming]

What's the reason to not pin dependencies from all groups by default?

Cause it is likely to cause dependency conflicts and it's easier to add group than to remove.

[ai-nikolai]

Thanks for this thread. @frostming

A few follow-ups on the above:

1. What is the desired workflow for building sub-packages / specific dependencies for local dev?

• pdm lock -G "all,the,groups,we,want,to,add,it,does,not,remove,previously,added,groups"
• pdm install -G "specific,groups"

2. What is the workflow for publishing packages (are all the groups automatically accounted for)?

• ?

---

Also, following up on @rafalkrupinski questions, @frostming:

• Assume we have two groups that conflict dev1 and dev2
• If we want to do local install / published install of the like pdm install -G "dev2" (why can't the lock file just avoid that conflict?)