Documentation: --ignore-pipfile should be more clear; mention pipenv sync

[de-code]

The --ignore-pipfile flag reads:

Ignore Pipfile when installing, using the Pipfile.lock.

That would suggest it always installs dependencies as per the Pipfile.lock file which doesn't exactly seem to be the case.

pipenv sync was added and only in the advanced documentation it mentions:

pipenv install --ignore-pipfile is nearly equivalent to pipenv sync, but pipenv sync will never attempt to re-lock your dependencies as it is considered an atomic operation. pipenv install by default does attempt to re-lock unless using the --deploy flag.

That distinction seems to be key and shouldn't be hidden in the advanced section.

An intuitive command for installing dependencies is pipenv install which was used before. It would make sense that if the documentation for pipenv install would itself point to pipenv sync as both are installing dependencies. And it's not obvious one should maybe look for pipenv sync.

[alexpirine]

Hi, could you please clarify pipenv install, pipenv install --deploy, pipenv install --ignore-pipfile, a combination of both, and pipenv sync? Seems like a lot of variations to me…

[alexpirine]

Without forgetting that pipenv install --system seems to be appropriate for building docker images…

[anentropic]

The docs are totally unclear

It seems like the default when doing pipenv install should be to install the versions pinned in the lockfile, but if it did that then why does pipenv install --ignore-pipfile exist?

Re-reading the docs it seems that pipenv sync does what I want:

"you can install packages exactly as specified in Pipfile.lock using the sync command"

(whereas presumably pipenv install would install latest versions and update the lock)

But then what does pipenv install --ignore-pipfile do differently?

The docs say:

pipenv install --ignore-pipfile is nearly equivalent to pipenv sync, but pipenv sync will never attempt to re-lock your dependencies as it is considered an atomic operation. pipenv install by default does attempt to re-lock unless using the --deploy flag.

It is not clear what this actually means in practice. I think it means that if you have installed/updated packages outside of pipenv, leaving your Pipfile.lock out of date, pipenv install --ignore-pipfile will update the lock file before installing from the lock. If it doesn't mean that then I have no idea.

[joaomcarlos]

This is extremely confusing.

What I think it should be (and already is ... for the most part):

• install should just install dependencies pinned on the pipfile.lock. If no pipfile.lock exists, install latest and create a pipfile.lock. In any other instance, do not touch the pipfile.lock.
• lock should be used when we are sure we want the versions currently installed. All other commands should warn us to run it, if we mess up, we have it versioned (in GIT) already anyway.
• sync in its current state, makes no sense, in my mind its the same as lock but currently thats not the case.
• install --ignore-pipfile makes no sense.
• update should get the latest version and warn us about updating pipfile.lock.
• update <specific> should update get the latest of a single package and warn us about updating pipfile.lock.
• install --dev (install both default and develop) makes no sense, should be just install.
• install --deploy should be used instead if no develop packages are wanted.

[januszm]

Totally agree with @joaomcarlos , perhaps it deserves its own GH Issue?

Also, it seems that the behaviour of pipenv install changed recently. The --help says that :

install    Installs provided packages and adds them to Pipfile, or (if no
             packages are given), installs all packages from Pipfile.

which is not true anymore because:

$ pipenv install
Installing dependencies from Pipfile.lock (abc683)…

[lapinvert]

@joaomcarlos If I correctly read the docs, install --deploy is just a regular install that throws an error if the lock file is out of sync. Therefore it still makes sense to use it as well as install --dev.

The other issues raised here are problematic tho and that would be great to see them addressed.

[matteius]

@joaomcarlos

This is extremely confusing.
What I think it should be (and already is ... for the most part):

install should just install dependencies pinned on the pipfile.lock. If no pipfile.lock exists, install latest and create a pipfile.lock. In any other instance, do not touch the pipfile.lock.

What you are describing is pipenv sync. pipenv install is pipenv lock + pipenv sync -- there is no good reason to change this.

lock should be used when we are sure we want the versions currently installed. All other commands should warn us to run it, if we mess up, we have it versioned (in GIT) already anyway.

I disagree; the lock file should always be used to install dependencies, things like --selective-upgrade and --keep-outdated are just asking for problems as they don't respect the results of the Pip resolver. --skip-lock doesn't provide the security guarantees that validating the hashes from the lock file provides.

sync in its current state, makes no sense, in my mind its the same as lock but currently thats not the case.

pipenv sync is what you think pipenv install should be, it does not lock. It will not lock in the future.

install --ignore-pipfile makes no sense.

Agreed. I think it was born from people want to be able to install from just a lock file in their CI and not provide the Pipfile, but I am not sure what it does in this case, it couldn't possibly relock right?

update should get the latest version and warn us about updating pipfile.lock.

As you mentioned prior you should have your lock file checked in to source control, so it shouldn't be a surprise that installing or updating a requirement will result in a lock file change. I never really use update so I cannot comment too much here on the current implementation of it.

update should update get the latest of a single package and warn us about updating pipfile.lock.

Again, I don't think we should warn about updating the lock file, it should be expected that anytime a package updates, that the whole lock file should be recalibated based on the pip resolver results, and the appropriate updates to the lock file should be made. Otherwise there is no updating a package without adjusting the lock file.

install --dev (install both default and develop) makes no sense, should be just install.

I argue this does make sense, install should not default to installing development depndencies you need locally but not in production. Plus now with named package categories it does make sense because you can have arbitrarily many package groups, and your intentions for them are generally to isolate things by platform, or environment and so it would never make sense to have install trigger lock and installation of all groups, in fact it would be quite problematic.

install --deploy should be used instead if no develop packages are wanted.

install --deploy only validates that the _meta hash is correct, which signifies the lock file is up to date with the Pipfile. So again, I think this is a misunderstanding of the pipenv interface and what it does.

Overall though this is quite valuable and shows that we need to do a total overhaul of the documentation.

[joaomcarlos]

@matteius Glad to see this issue summoned from the dead and into the light.

I had totally forgotten I had made a comment to this issue, and largely my mind hasn't changed, but to clarify, the intention of what I said is not to describe what it DOES, but rather what the commands SHOULD do based on the combination of words in them.

Example: Its not very useful to teach to a colleague something like, "oh, use sync instead" when you want to install because pipenv install does not do an install but rather a "lock" + "install" (called sync for some reason).

Hopefully the word install does not mean different things to different people, and to clarify: my understanding of "install" is when you got a set of well defined (locked) files (code or other deliverables) and you simply want to put them into the system so they are available. Which is roughly what happens when you "install" something on a computer using an installer.

Sync on the other hand, means, to me, that you want to synchronize with a remote system, akin to "Dropbox sync" or a "git fetch" (sync repo remote state - not files - with the server).

Your example of install --deploy only doing a validation is probably the most confusing one. To me, what I would expect that command to do is "perform an install tailored for deployment", which could be read as "skip development packages when installing since we are deploying anyway"

Ultimately, the commands can be whatever anyone wants, pipenv potatoe can lock+sync or any other combination, the machine wont care (TM) , but might be confusing to a human :D

Edit: judging from the amount of upvote emojis on my other comment, I am pretty sure I am not alone. This could also be because other package managers from other languages follow the naming convention I suggested for the commands and people are used to them? I don't know.

[matteius]

the intention of what I said is not to describe what it DOES, but rather what the commands SHOULD do based on the combination of words in them.

@joaomcarlos What it does and what it should do are perhaps two different things, but believe I am not the original author of pipenv. I have been improving pipenv for a year now, and we still aren't to the point where we want to re-write the interface and make it not backwards compatible with what it does. I've spent a year studying what it does and fixing bugs in the space, so I was simply explaining it because I think the documentation doesn't make things totally clear.

The install phase is lock + install, I don't think that is wrong and I am not going argue that install doesn't imply installation, it does, but why think that install does not imply re-locking based on your Pipfile specifiers? I also won't argue for or against any of the core command names, because I didn't come up with them, I just support what it is that they do and try to improve things in non breaking ways. For example, we worked to factor the requirements functionality out into its own command instead of being a flag passed to lock for example.

I don't really have a lot more to say on this topic, other than I feel strongly there is more code cleanup and refactor to do before we can really consider what a command interface rewrite might be like.

[joaomcarlos]

@matteius Totally understand your position. And i commend you for helping pipenv moving forward. We need pipenv to keep moving forward, and the fact that we are still using it speaks volumes!

As you said, documentation could be a little bit more clear. I understand not wanting to break backwards compatibility (thanks btw, makes my life easier), however if things should be changed, then breaking backwards compatibility most sometimes be done in order to guarantee a healthy future. You aren't there yet ... totally understand! That bridge will be crossed when we get there.

About "install" meaning "lock + install", well, if you have to explain that "install" is actually "install + lock" perhaps thats not the best name. Or maybe, that is just implicit WHEN a few factors happen, thats totally understandable, if pipfile has new packages for example it could make sense also install those if an optional parameter is added (this parameter could be instructed by a warning like "New packages found in pipfile, install them using --install-new" ... or something similar, its just an idea.

Anyway, I do not aim to be a downer but rather perhaps try and contribute by pointing it out.

Thank you for your work!

[b-rodrigues]

Hi all

so I’ve followed this discussion, and agree with the general sentiment: as a new user of pipenv I am a bit unsure how I should proceed. I would like to validate my understanding with you if you don’t mind: suppose I’m developing a project and use pipenv to handle the env and dependencies. So I have these Pipfile and Pipfile.lock files: now suppose I want to give them to a colleague, so that he or she can regenerate that environment. Here’s what I would tell him/her:

• Start by using pyenv to install the required Python version. You can find the Python version in the Pipfile.
• Use pyenv local 3.10.5 to make this version the one that gets used for that project (assuming 3.10.5 is the version I used for that project back then)
• Run pipenv install to install the dependencies that I’ve used for that project, and start working on the project by calling pipenv shell to drop into that environment.
• Enjoy!

Am I missing something?

[joaomcarlos]

@b-rodrigues Hi, what you would want is instead pipenv sync because pipenv install does not just install.

Cheers.

[b-rodrigues]

Thanks for clarifying @joaomcarlos ! Yeah I definitely agree that I was not expecting install to, well, do more than install stuff as specified in the lockfile.

In R there's the renv package that does something similar to pipenv (albeit just focusing on packages and not interpreters) and the command there is called restore: so restore installs the dependencies exactly as specified in the lockfile, that's it.

Perhaps restore could be provided to do exactly that, it would be an alias of sync I guess, but restore, at least to me, conveys its intent better than sync.

[b-rodrigues]

Sorry, but there’s still something that confuses me. If I install packages by specifying their versions (as well as a python version,
like so pipenv --python 3.10.5 install beautifulsoup4==4.12.2 pandas==2.1.4 plotnine==0.12.4 skimpy==0.0.11), I get a Pipfile that looks like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
beautifulsoup4 = "==4.12.2"
pandas = "==2.1.4"
plotnine = "==0.12.4"
skimpy = "==0.0.11"

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.5"

In this case, wouldn’t pipenv install always install exactly the same packages? Or should I still use pipenv sync? My current understanding is that if instead I don’t specify package versions when installing them, I will get a Pipfile with * for the package versions. In that case, running pipenv install sometime in the future would install the packages latest versions and regenerate a Pipfile.lock. Hence why one should use pipenv sync to install the versions as they were (so using the versions listed in Pipfile.lock). But in the case described above, where the Pipfile lists exact versions, there would be no difference between pipenv sync and pipenv install... right?

[januszm]

But in the case described above, where the Pipfile lists exact versions, there would be no difference between pipenv sync and pipenv install... right?

We've talked about this in the comments for other issues, for many people (including me) the behaviour of pipenv install is counterintuitive, because it actually performs an upgrade

[matteius]

@januszm I believe now that we are planning semver releases, the next major release we are hoping to switch the behavior of pipenv install to be what every non-seasoned user would expect.

[matteius]

Laugh all you want, but I've really been doing my best to drive this project forward in my spare time. I have been using pipenv for about 2-years and 2-months and helping as one of 2 maintainers for just about 2 years. I'm not going to lie and say everything has been a perfect release with no regressions, but I hope that if you have been following along and look at the code base 2+ years ago to now you would actually see big improvements and actually be able to trace through the code and understand better what its doing. Were any of the original authors around to explain their rationale or help out with the changes? Not really. I'm not going to enumerate every accomplishment in this thread, but I think its totally reasonable that we have taken the position of not changing things like this until we have gained enough confidence and understanding to do so. It also makes sense why some things have broken in releases, but we have always done our best to patch them in some kind of timely fashion. I encourage everyone else to contribute more because I am not going to magically end up with more time to solve the world of pipenv overnight.

[januszm]

I respect the time and effort you put into it, really. The smiley is related to "non-seasoned" thing, it's all good man! I'm glad there'll be a change in the next major release as me and my team are using pipenv all the time for all Python projects!

[jperras]

@matteius I hear you, and I appreciate the time and effort you've put towards this and other OSS projects. I'm sorry that other people feel that they own your time. They don't. It's easier to complain than to act, and they're the ones complaining, while you're the one acting. Thank you.

[matteius]

Thanks for understanding, its been a long few months to deliver a major project this week on my end.

[b-rodrigues]

Hi @matteius , thanks for all your hard work on this! It’s truly appreciated.