Feature request: pipenv should have a native way to generate requirements.txt directly from the lockfile

[ImreC]

When you just run pipenv install [package] on your project the version in the Pipfile is not pinned, but the lockfile contains the installed package version. Unfortunately not all tooling around Python supports Pipfiles, which means that you very often need to generate a requirements.txt file based on your Pipfile.lock. I think all approaches for this are well described here: #3493, but for simplicity I will outline them below:

1. Run pipenv lock -r. Why this does not work is that it runs pipenv lock, which generates a new lockfile and then generates the requirements. This leads to problematic cases if you want to keep the requirements exactly the same.
2. Run pipenv run pip freeze > requirements.txt (potentially with pipenv sync before that to make sure that all the packages from Pipfile.lock` are installed). Why this is problematic is because running pip freeze very often leads to too many strict package dependencies being included which leads to dependency mismatches, rendering the whole file unusable.
3. Some methods based on jq or some similar tool that literally parse the lockfile and then output a requirements.txt

I think the problem with option 1 and 2 is that they both do too much. Option 3 is closest to what most people need, but introduces a new dependency on jq which is also annoying to enforce across a larger group of developers with multiple environments. Which lead to that I am now including a simple python file containing this in almost every repo:

import json

with open('Pipfile.lock', 'r') as f:
    reqs = f.read()

with open('requirements.txt', 'w') as f:
    f.writelines(f"{req_name}{value['version']}\n" for req_name, value in json.loads(reqs)['default'].items())

I feel that this is so mindbogglingly simple that it should be included somewhere in the pipenv package. My proposition is to include a command pipenv requirements which reads the Pipfile.lock json file and outputs a requirements.txt based on the default key. Adding --dev as an option also adds the requirements from the develop key. No locking, no syncing, no pip freezing. Just this.

I am not only asking for this, but also happy to take some time to implement/maintain this if other people support this idea.

Overall, I think pipenv is a vital tool in the Python ecosystem and will continue to use it in any case. Thanks for the work that went into this so far! I think adding this seemingly trivial extra command would resolve some confusion around the requirements.txt (f.e. #4731) and therefore would be a relatively low effort high value feature for pipenv.

[matteius]

@ImreC I am wondering if what you need is actually the --keep-outdated flag. Since the pipenv lock -r --keep-outdated will only add any new updates from the Pipfile, it will not generate a totally new lock file. It does require that you have the Pipfile in addition to the Pipfile.lock but from the sound of it you do, and are just trying to generate a requirements.txt without changing your locked dependencies, which I think should already be possible.

[matteius]

Run pipenv run pip freeze > requirements.txt (potentially with pipenv sync before that to make sure that all the packages from Pipfile.lock` are installed). Why this is problematic is because running pip freeze very often leads to too many strict package dependencies being included which leads to dependency mismatches, rendering the whole file unusable.

I am not sure I follow the argument here that the strict package dependencies being included lead to dependency mismatches, if you are generating the requirements.txt from the pip freeze then you are pinning to what is currently an installable environment. When you run pipenv lock -r --keep-outdated you also will be getting the rigidly defined versions from the lock file which should be ultimately very similar to pip freeze version.

Here is an experiment. Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
black = "<22.1.0"
requests = "*"

[dev-packages]

[requires]
python_version = "3.10"

[pipenv]
allow_prereleases = true

pipenv lock -r --keep-outdated generates the following requirements.txt:

-i https://pypi.org/simple
black==21.12b0
certifi==2021.10.8
charset-normalizer==2.0.12; python_version >= '3'
click==8.0.4; python_version >= '3.6'
idna==3.3; python_version >= '3'
mypy-extensions==0.4.3
pathspec==0.9.0
platformdirs==2.5.0; python_version >= '3.7'
requests==2.27.1
tomli==1.2.3; python_version >= '3.6'
typing-extensions==4.1.1; python_version >= '3.10'
urllib3==1.26.8; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'

pipenv sync and pipenv run pip freeze > requirements.txt yields:

black==21.12b0
certifi==2021.10.8
charset-normalizer==2.0.12
click==8.0.4
idna==3.3
mypy-extensions==0.4.3
pathspec==0.9.0
platformdirs==2.5.0
requests==2.27.1
tomli==1.2.3
typing_extensions==4.1.1
urllib3==1.26.8

Which is basically identical to the one generated by pipenv lock -r --keep-outdated except that one also has annotations about python versions.

[ImreC]

Hi @matteius, thanks for responding! I can give an example of where pipenv run pip freeze > requirements.txt does not work well. For example this Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"
requests = "*"

[packages]
fastapi = "*"
uvicorn = "*"
mangum = "*"
pyjwt = "*"

[requires]
python_version = "3.8"

This leads to the following requirements.txt

anyio==3.5.0
asgiref==3.5.0
attrs==20.3.0
certifi==2020.11.8
chardet==3.0.4
click==7.1.2
fastapi==0.61.2
h11==0.11.0
idna==3.3
iniconfig==1.1.1
mangum==0.10.0
packaging==20.4
pluggy==0.13.1
py==1.9.0
pydantic==1.7.2
PyJWT==1.7.1
pyparsing==2.4.7
pytest==6.1.2
requests==2.24.0
six==1.15.0
sniffio==1.2.0
starlette==0.13.6
toml==0.10.2
typing-extensions==3.7.4.3
urllib3==1.25.11
uvicorn==0.12.2

which pins idna at 3.3, but I think requests depends on idna <3 so this errors out when you try to install with normal pip from requirements.txt (or at least I confirmed this when building with AWS SAM).

Running pipenv lock -r --keep-outdated > requirements.txt works in this case and I haven't tried to use that in a long time, but from memory, I think it still includes the Pipfile in the chain, meaning that packages will on some occasions still get updated. If you really just want a clean requirements.txt file from your Pipfile.lock, and be guaranteed that there is no change, I remember that this was not working in several cases. I think pipenv having a way to just generate a clean requirements.txt from the Pipfile.lock without any interactions with other pipenv components would benefit pipenv in this case.

[ImreC]

I just went through the pipenv repo real quick and from what I can tell (in 5 minutes) the intended behavior of --keep-outdated is indeed to still process changes to the Pipfile while keeping outdated packages. What I am looking for specifically (along with the people who use jq in my previous outline) is to generate a requirements.txt based on Pipfile.lock without any changes. Regardless of Pipfile or something else, for the sole purpose of using the current locked requirements in a different tool chain that does not support Pipfile. The most obvious way to me is to parse the json from the lockfile

[matteius]

Running pipenv lock -r --keep-outdated > requirements.txt works in this case and I haven't tried to use that in a long time, but from memory, I think it still includes the Pipfile in the chain, meaning that packages will on some occasions still get updated. If you really just want a clean requirements.txt file from your Pipfile.lock, and be guaranteed that there is no change, I remember that this was not working in several cases.

Yes it still includes the Pipfile in the chain, but if it doesn't differ from the lock file, it will not update anything. What version of pipenv are you running with?

the intended behavior of --keep-outdated is indeed to still process changes to the Pipfile while keeping outdated packages.

Yes but if you haven't modified the Pipfile, there will be nothing to update to the lockfile. Since you didn't include the lock file in your example, it would have to be generated from the Pipfile you provided.

Regardless of Pipfile or something else, for the sole purpose of using the current locked requirements in a different tool chain that does not support Pipfile.

Your project is using pipenv, therefore it has a Pipfile, you can generate a requirements.txt from the Pipfile and Pipfile.lock -- this doesn't require the Pipfile after that, just for generating the requirements.txt.

It may be possible to have a command that generates the package requirements from just the lockfile, but I am not sure I really understand the use case and it feels a little bit like duplicating existing functionality (that currently also looks at the Pipfile) in order to get the same result.

pipenv run pip freeze > requirements.txt

This just puts what you have installed in your virtualenv into requirements.txt not looking at the Pipfile or the lock file.

which pins idna at 3.3, but I think requests depends on idna <3 so this errors out when you try to install with normal pip from requirements.txt (or at least I confirmed this when building with AWS SAM).

That means you have idna 3.3 installed somehow when you really depend on idna <3. I'd start by trying to figure that one out.

[matteius]

@ImreC Another thing about your latest example Pipfile is you have requests = "*" in the dev packages so its not going to force idna to be a specific version when just generating the lock.

[matteius]

I moved requests = "*" to the packages section but still got idna==3.3; python_version >= '3' which matches the setup.py restrictions for requests. Maybe you were talking about an older version of requests or something that is still python2?
https://github.com/psf/requests/blob/main/setup.py#L46-L47

[ImreC]

@matteius I tried to install the generated requirements.txt from pipenv run pip freeze > requirements.txt in a fresh manually created venv and this worked. This leads me to the conclusion that the package mismatches are not a pip problem.

This was the output of pipenv graph which led me to the original idea that the package mismatch was caused by requests:

fastapi==0.74.0
  - pydantic [required: >=1.6.2,<2.0.0,!=1.8.1,!=1.8,!=1.7.3,!=1.7.2,!=1.7.1,!=1.7, installed: 1.9.0]
    - typing-extensions [required: >=3.7.4.3, installed: 4.1.1]
  - starlette [required: ==0.17.1, installed: 0.17.1]
    - anyio [required: >=3.0.0,<4, installed: 3.5.0]
      - idna [required: >=2.8, installed: 3.3]
      - sniffio [required: >=1.1, installed: 1.2.0]
mangum==0.12.4
  - typing-extensions [required: Any, installed: 4.1.1]
PyJWT==2.3.0
pytest==6.1.2
  - attrs [required: >=17.4.0, installed: 20.3.0]
  - iniconfig [required: Any, installed: 1.1.1]
  - packaging [required: Any, installed: 20.4]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - six [required: Any, installed: 1.15.0]
  - pluggy [required: >=0.12,<1.0, installed: 0.13.1]
  - py [required: >=1.8.2, installed: 1.9.0]
  - toml [required: Any, installed: 0.10.2]
requests==2.24.0
  - certifi [required: >=2017.4.17, installed: 2020.11.8]
  - chardet [required: >=3.0.2,<4, installed: 3.0.4]
  - idna [required: >=2.5,<3, installed: 3.3]
  - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.11]
uvicorn==0.17.5
  - asgiref [required: >=3.4.0, installed: 3.5.0]
  - click [required: >=7.0, installed: 8.0.4]
  - h11 [required: >=0.8, installed: 0.13.0]

but I see that the idna requirement that is printed out is indeed the requirement for Python 2, so I don't really know why this was printed out. On top of that I am currently unable to reproduce my original error even though I was able to this morning. I am genuinely confused by this.

I am using pipenv version 2022.1.8. Also, I know what pipenv lock and pipenv run pip freeze are supposed to do. I appreciate the deep dive though.

I just want to stress that I am not pushing to put something into pipenv that has no value. If pipenv lock -r --keep-outdated is what is supposed to work in this case, I'll happily give that a spin. I remember from 1,5 - 2 years ago that something about it was not working and we looked at it quite extensively at the time. However, I realize that this is a long time ago and that whatever issue we had with it back then may very well be fixed.

May I suggest we update the documentation here https://pipenv.pypa.io/en/latest/basics/#pipfile-lock-security-features to also mention the --keep-outdated flag? Even though I am not sure anymore about my original case, I am sure that people want to be able to generate a requirements.txt file and that this issue #3493 is still very high in the search results, which could lead people to all sorts of solutions which are not --keep-outdated.

Thanks for taking the time to look into this. Really appreciate it!

[matteius]

I just want to stress that I am not pushing to put something into pipenv that has no value.

I know, I was just trying to see if there is an existing solution that would work for you now, rather than wait for something to get accepted and published in a new release. There might be value in having a command that does such a thing when a Pipfile is lacking or has been changed and the history is gone. I just want to be careful before supporting too many new commands, if they are truly required. I am hoping others will weigh in on their uses cases, and also I'll say what I said to the developer that wrote the new verify command. If you want to have this change, and have time to make a PR for it, it will definitely be considered further. I personally don't have time right now to pursue creating such a PR as I am working on other things related to the project.

[ImreC]

I have a bit of a busy period, but plan to revisit this in a couple of weeks. Also will look into creating a PR at that time. If people are reading this and feel this is a useful addition, please add a thumbs up.

[ImreC]

I have created a PR for this. Would appreciate a look.

[matteius]

This feature has been merged.