Add ability to have custom package sets. pipenv --install --pack=dev --pack=test

[sanscore]

Feature: Custom Package Sets

It would be nice to be able to create custom package sets. For instance, pipenv already has support for two package sets "default" and "dev". However, I can see use cases where it would be nice to have multiple package sets.

$ pipenv install --pack=test    # Custom "test" package set; [test-packages]
$ pipenv install --pack=dev     # Existing "dev" package set; [dev-packages]
$ pipenv install --pack=default # Existing "default" package set; [packages]
$ pipenv install --pack=        # "; [packages]

Description of Problem

For me, I have a use case where I want to test the changes made in a another package, DependencyProject, to see the effects it in MainProject before I release it (DependencyProject).

In MainProject, I want everyone to have DependencyProject.

$ pipenv install dependency-project

For --dev work, I want contributors to install testing tools.

$ pipenv install --dev pylint pytest tox

But, normal contributors (nor the CI release process) shouldn't have pre-alpha releases of DependencyProject installed. This should be reserved experienced contributors (or, "coordinated" contributors), so they should use a different package set; not "default", not "dev".

$ pipenv install --pack=dep-test -e git+https://github.com/fictional-org/dependency-project#egg=dependency-project

Current Workaround

To be honest, I haven't attempted a workaround yet. I've used Pipenv sparingly over the past year, but over the past month, I have made it part of my standard process. And, I'm seeing this scenario (as described above) coming. So, at the moment, it is a hypothetical problem, but I do have a real workaround.

Use pipenv run pip install ... when I need to do alpha testing.

$ pipenv run pip install -e git+https://github.com/fictional-org/dependency-project#egg=dependency-project

Then run pipenv install --dev to go back to the standard development environment.

$ pipenv install --dev

I've tested this scenario by installing requests and requests==2.9.1. (NOTE: The that the current version of Requests ==2._19_.1 is very similar to the older version ==2._9_.1.)

$ pipenv install requests

$ pipenv run pip freeze
requests=2.19.1

$ pipenv run pip install requests==2.9.1

$ pipenv run pip freeze
requests=2.9.1

$ pipenv install

$ pipenv run pip freeze
requests=2.19.1

Existing Problems

In my hypothetical problem, I am trying to add two different version of the same package into the Pipfile. Because Pipenv does have two package set options (default and dev) I can attempt the scenario using them. As it turns out, it isn't currently possible to have differing versions between the two package sets. Again, I'll demonstrate with Requests.

Explicit Default, Relaxed Dev

$ pipenv install requests==2.9.1
# [packages]
# requests = "==2.9.1"

$ pipenv install --dev requsets
# [packages]
# requests = "==2.9.1"
# [dev-packages]
# requests = "*"

$ pipenv run pip freeze
requests==2.9.1

Explicit Dev, Relaxed Default

$ pipenv install --dev requests==2.9.1
# [dev-packages]
# requests = "==2.9.1"

$ pipenv install requests
# [packages]
# requests = "*"
# [dev-packages]
# requests = "==2.9.1"

$ pipenv run pip freeze
requests==2.19.1

$ pipenv install --dev
$ pipenv run pip freeze
requests==2.9.1

NOTE: This problem is, likely, a combination problem, so I should test scenarios for "Relaxed Dev, Explicit Default" and "Relaxed Default, Explicit Dev" as well (And, there's likely more cases that I haven't conceptualized yet.) But, I think the two scenarios show an inconsistency.

So, just to get to my desired feature, this problem will have to be solved first. (I have not researched the Issues about this potential bug, so I haven't created a new one. I can, and will, upon request. ~"When Duplicate Dependencies exists, the dependency defined in the Default package set always wins.")

Conclusion

Despite my specific use case (and the current limitations that prevent it), there's potential for this feature.

I suppose, conceptually, my request is most similar to extra_requires as found in setuptools. Most notably projects like IPython (setup.py) depend on this functionality to provide different levels of dependencies to their users; ex: pip install ipython['all'].

A project like Tox allows for multiple "package sets", so conceptually the expectation exists. Although, I'll concede that Tox implements this scenario by having multiple virtualenvs. That's a potential solution to the problem, but by separating sets into virtualenvs, that creates a combination problem between each set. (Does pipenv install --pack=test imply installing --pack=dev too?)

... I'm doing a terrible job of concluding this request because now I have more questions pertaining to resolution scenarios between multiple sets. So, in order to stop my digressions, I'll stop here to say, ...

I think this would be a great feature. I can't possibly think up of all the cases where it would be amazing. And, I can't conceive of all the cases where it becomes hard to implement, so I'm just putting it out there. Let's discus.

Due Dilligence

I haven't been able to find any mention of this capability in the Basic Usage, Advanced Usage, nor in the Issues, (is:issue label:enhancement dev packages), so I created this request.

[ed-alertedh]

This is sort of related to #1353. Although in my use-case I want the different package sets to have versions that are as similar as possible. Whereas you want to have deliberately different versions of packages. Both cases result in different dependency trees which means you need a new lock file per set of packages (and also per platform, as it turns out).

I extracted this out of our repo as an example for the other issue, you might find the same templating approach helpful too (it would need some modifications to be a more general tool but the basic idea is there).

[uranusjr]

Hi, could you convert this to PEEP format? You already have most of the content ready, but by organising it in a document, it helps to effectively summarise and record discussions. Thanks for putting this together!

[techalchemy]

This is an excellently structured proposal, but Kenneth needs to look it over and he only reads PR's submitted against the peeps folder as @uranusjr indicated, so if you can simply take the markdown and PR it against there it would be much appreciated!

[sanscore]

@uranusjr, @techalchemy: PEEP-001 is a bit ambiguous about starting the PEEP process.

Should I just create PEEP-003 in a local branch and submit a PR? It would seem that this process would create some contention if there were multiple, unmerged PEEPs at a given time. However, at the moment, this doesn't seem to be a problem.

From what I can tell, it looks like PEEP-002 followed this process in #2819.

Or, is there an informal, "preliminary" phase to hammer out the text before making the PR.

While I wait for clarification, I'll just assume the former by writing "PEEP-003" and pushing it to a branch to my personal fork.

cc: @kennethreitz

[uranusjr]

The number does not need to be the lowest possible, you can choose whatever you think is appropriate. The maintainers may request you to change the number before it is merged.

[marco-silva0000]

is there any progress on this?
is there any repo where a draft implementation is being worked on?

[sanscore]

@marco-silva0000: Sorry, no. I haven't taken the opportunity to write-up the PEP for my own request. If you have the need and motivation to start the write up, then I'll be more than happy to help review drafts and, if necessary, suggest edits.

[matteius]

Interesting -- so was this ever supported? There has been a bug starting in pipenv 2021.11.23 where it breaks locking if other package sets are defined. Ref: #4866

[matteius]

It seems like the original requestors of this are no longer with us, but I think there is some merits to someone considering this type of enhancement. I think the complicated thing is there are already possible discrepancies between package versions in groups such as the default and dev dependencies. However it could make parts of the code more generic too where right now we specifically are checking default or dev related groups. Other benefits would be fixing bugs like #4866. I am not going to have time in the near term to get to this, so I have marked it as a Contributor Candidate for anyone to consider the approach described in the ticket and the trade-offs to implementing this enhancement or not.

[matteius]

This is very close to being a reality -- please track with #4745