Importing/ exporting from poetry in pip format.

[buriy]

Hi Sébastien,
Now the poetry works great as a package manager (I'm happy to use it!).
So, I think, maybe it's time to revisit export/import to requirements.txt for the projects that can't rely on poetry being installed on the target environment, but still wanted to be managed by poetry (some people prefer pipenv utility, other people use plain pip, etc).
Also, I'm considering use cases when you need to import the pip list of the existing packages to poetry.

The export workflow looks pretty simple:
you just export the requirements, similar to how "pip freeze" does (now this can be only done with poetry run pip freeze ) , or add some option, maybe the good syntax is poetry show --requirements. A proposed implementation is here: #100 (comment)

As for the import workflow, right now the challenges are:

1. requirements.txt format is different from one that poetry uses
2. "poetry add" might break if one of the dependencies won't install

I consider the following command line options (naming can be different, of course):
poetry add --requirements requirements.txt and
poetry add --pip poetry>0.10, poetry add --pip lxml<4.2.6 poetry>0.10
(Or maybe you can just support this format without this option)
I'm not sure how complex is to handle properly the pip format, but even the minimal support would cover 99% of the requirements: "package", "package{comparison op}version". They now work if you add a colon, like this: "package:{comparison op}version". So I believe it's easy to support those -- that would be a great help!
Also, to prevent poetry add from breaking, I would consider adding:
poetry add --save which only saves the updates, then, poetry add --lock which updates the lock file, but doesn't install the package(s) (similar to how poetry update --lock). I would also add poetry remove --save and poetry remove --lock to have the inverse operations.
I would also change poetry add behavior to ignore already added requirements (though a warning might be displayed). Otherwise after poetry add package1 package2 package3, if it breaks at package2 you not only need to change package2 options but also remember to remove package1 from the list -- otherwise, you'll get a "Package package1 is already installed" error.
Currently I have to use the following script to import the requirements from a file:
cat requirements.txt | perl -pe 's/([<=>]+)/:$1/g' | xargs -n 1 echo poetry add
Exporting the list is also non-trivial now, so it's great that pip freeze works fine.

I believe these features might significantly increase poetry adoption speed.

• I have searched the issues of this repo and believe that this is not a duplicate.

[pikeas]

+1 for the ability to export to requirements.txt. My specific use case is Github's automated vulnerability scanner, which currently only supports requirements.txt.

[sdispater]

The PR for exporting to a requirements.txt file is here: #675.

As for importing, I am not sure this is a direction I want to go in. I understand that if you have a lot of dependencies this might be somewhat a painful process but I don't want to blot the CLI for cases that can be handled by hand or via a simple script.

[buriy]

@sdispater please at least support pip format for poetry add, and implement skipping existing packages in poetry add, then, this could be handled by a simple script. Right now, this is a very painful process.
( I'm going to make a PR for these two changes for poetry add )
A lot of projects adopted requirements.txt / pip format, so you're basically telling that supporting mainstream options used in python community is "not a direction to go in".
Reg "handled by hand" -- all automation is about helping users, instead of "handling by hand". All your project is made to solve "handled by hand" situation.
So I think the only real concern not to add a piece of functionality is that it will be costly for you to develop and support it and it won't be often needed. This is ok, I can help you with development and support. I can write a pull request for that. Reg "not needed" concern -- we can discuss this specifically or at least state that this is the real culprit rather than diplomatical "not a direction to go in".
I think #675 is looking fine and, according to the comments, is the most important thing and a very popular request.
But I also have seen several PR issues from people that want to do a requirements.txt import, also I see a clear benefit of this step for your project adoption.
In general, I think, pip format isn't going away this or next year (like python2 didn't disappear immediately after python3 creation ), so supporting it is useful for a package manager in 2018.

[meshy]

I'm looking to move a project from pip with requirements.txt to poetry. I see from the discussion here that there is no import script.

Is there another onboarding/migration process that will make this move easier?

[buriy]

@meshy and what's about the one-line import script I provided?

[meshy]

@buriy it gets me some of the way there, thanks.

It doesn't work for comments, or places where we've done -r some_related_requirements.txt, but that can be worked around.

[floer32]

one-liner import script

Just quoting piece of OP @buriy 's post, in freestanding comment.

the one-line import script I provided

👇

Currently I have to use the following script to import the requirements from a file:
cat requirements.txt | perl -pe 's/([<=>]+)/:$1/g' | xargs -n 1 echo poetry add

[floer32]

I will see if I can use pip's requirements parsing like this other post to make a script that handles edge-cases too. Although pip maintainers have moved parse_requirements to a private submodule and discourage using it, their rationale emphasizes that pip should not be clashing with another package manager.

That's totally reasonable looking at a project continuing to use pip (don't mix package managers). For migrations I think we are in a special enough case, to justify using pip parse_requirements; at least in an ad-hoc script example. (I also understand and support excluding this feature from Poetry itself, for what it's worth. That could lead to a lot of yak-shaving...)

---

UPDATE: Interestingly Pipenv did its requirements-import by subverting those recommendations from pip. They vendorize and patch pip libraries. Spent a long time diving through the source for that, it's interesting. Wondering if I should follow suit, instead of reinventing the wheel for reqs-parsing.

Specifically I may use the pip-shims library because that handles the problems of relying on pip internals. In pip-shims there are shims for each change of pip internals that would break downstream users. and pip-shims is maintained by one of the coredevs on Pipenv; and Pipenv has a feature to import requirements; so we could expect pip-shims to stay maintained!

---

Again I still agree with thread above that it makes sense for poetry to stay out of promising support for requirements import. Avoid those yak-shaves, avoid those opportunity-costs. But I could complete an 70%-working solution that, together with a human overlooking things, does the job well enough to help increase adoption. and that could just be in a separate repo or gist.

will report back soon

[floer32]

Hmm. I had some false assumptions and they were making me think this was more important than it really is. poetry add 'requests<=3.0' etc would not work so I thought poetry add did not support operators besides equality. Now I see I just needed to add a : and actually @buriy 's script covers that, I just missed that detail at first. i.e. poetry add 'requests:<=3.0'. So that covers most normal cases. (opened #948 for documentation update to make this obvious)

The approach noted above may still be relevant so that various edge cases and more complicated syntax can be dealt with (git/etc) ... but lower urgency since now I realize I/we can translate most things in most requirements.txt files, with that one-liner.

[absassi]

I think this feature request overlooks one important aspect of a requirements file: although useful in some cases, it doesn't make much sense to do a poetry add of a typical requirements.txt, because add is meant to put abstract dependencies into pyproject.toml, while a typical requirements file has concrete dependencies, which should go into poetry.lock.

Converting from the format of a requirements file into the Poetry format for pyproject.toml is straightforward, and it is not that hard to do manually, especially with a good text editor, even though it would be nice to have an option to do this conversion automatically.

However, I can't properly do is to convert a requirements.txt into a poetry.lock, because:

• the poetry.lock, despite being human-readable, it is obviously not human-writable.
• Poetry doesn't import a requirements.txt into its lock file, nor any other format I'm aware of.
• Poetry only installs and locks the latest allowed version of my dependencies. I have not found any command to install and lock some specific version of a dependency.

What I do now is to pin to exact versions from my requirements.txt in pyproject.toml, run poetry lock to generate the lock file, then replace the version with the proper ranges as they should be. But this feels like a workaround to me, not a normal procedure to convert an existing project to Poetry, and requires too much manual work, especially when people who needs to do that usually are new to Poetry.

To import for locking, it doesn't matter that parsing a requirements file is very complex. If Poetry just parses the bare minimal (name==version lines) and prints warnings for all the other unsupported lines, that would already be great. That would work for the great majority of cases, including ability to pipe the output of pip freeze, so we can also lock an existing environment.

In terms of implementation in Poetry, I don't know the code base, but I guess it requires calling the dependency resolver with the constraints specified in pyproject.toml, but replacing the constraints for the packages found in the requirements file with a simple exact-version constraint. The resolver would then just do the right thing.

[paranoidi]

You might think that you could import existing requirements.txt easily with poetry add $(<requirements.txt)

... except that a requirement like requests==2.1.0 will be written into pyproject.toml incorrectly with =2.1.0 instad of ==2.1.0. Poetry is fine with this broken notation but dephell is not.

[tigerhawkvok]

@hangtwenty I found that @buriy 's script was totally failing on my requirements file, so I wrote a short Python script to do the import.

#!python3
sourceFile = "./requirements.txt"

import re
import os

if not os.path.exists("./pyproject.toml"):
    os.system("poetry init")

with open(sourceFile) as fh:
    requirements = fh.read()

noComments = re.sub("^#.*$", "", requirements, 0, re.IGNORECASE | re.MULTILINE)
bareRequirements = re.sub("\n+", "\n", noComments, 0, re.IGNORECASE | re.MULTILINE).strip()

pipPoetryMap = {
    ">": "^",
    "=": ""
}

reqList = list()
for line in bareRequirements.splitlines():
    package, match, version = re.sub(r"^(.*?)\s*([~>=<])=\s*v?([0-9\.\*]+)", r"\1,\2,\3", line, 0, re.IGNORECASE | re.MULTILINE).split(",")
    try:
        poetryMatch = pipPoetryMap[match]
    except KeyError:
        poetryMatch = match
    poetryLine = f"{package}:{poetryMatch}{version}"
    reqList.append(poetryLine)

print("Found dependencies:")
print(reqList)

for req in reqList:
    os.system(f"poetry add {req}")

[pnpnpn]

one-liner import script

Currently I have to use the following script to import the requirements from a file:
cat requirements.txt | perl -pe 's/([<=>]+)/:$1/g' | xargs -n 1 echo poetry add

The > was redirecting to files for me. I had to add quotes to make it work:

cat requirements.txt | perl -pe 's/([<=>]+)/:$1/g' | xargs -n 1 -I {} echo "poetry add '{}'"

[pgilad]

I'd use @pnpnpn suggestion, with a minor differences (Notice I removed the /g option from perl:
cat requirements.txt | perl -pe 's/([<=>]+)/:$1/' | xargs -t -n 1 -I {} poetry add '{}'

[sdispater]

The beta releases of the 1.0.0 version now have an export command that can export to a requirements.txt format.

[dz0ny]

If someone comes around and wants to convert from Pipenvs Pipfile

from pathlib import Path

import json
import toml

Pipfile = toml.load(Path("Pipfile").open())
Pipfile_lock = json.load(Path("Pipfile.lock").open())

for package in Pipfile["packages"]:
    try:
        version = Pipfile_lock["default"][str(package)]["version"]
        print(f"poetry add {package}={version.replace('==', '')}")
    except KeyError:
        pass

for package in Pipfile["dev-packages"]:
    try:
        version = Pipfile_lock["develop"][str(package)]["version"]
        print(f"poetry add --dev {package}={version.replace('==', '')}")
    except KeyError:
        pass

[danihodovic]

As for importing, I am not sure this is a direction I want to go in. I understand that if you have a lot of dependencies this might be somewhat a painful process but I don't want to blot the CLI for cases that can be handled by hand or via a simple script.

@sdispater I think it would increase the adoption rate if an import command was built into Poetry. I like using Poetry, but having to do manual work (or rely on scripts) when moving every legacy project to Poetry is an adoption barrier.

If export already exists, I think it'd be consistent in the CLI if import existed as well.

[bolinocroustibat]

I'd use @pnpnpn suggestion, with a minor differences (Notice I removed the /g option from perl:
cat requirements.txt | perl -pe 's/([<=>]+)/:$1/' | xargs -t -n 1 -I {} poetry add '{}'

This one is working very fine.
I really think it should be a Poetry command, though.

[hughesadam87]

Windows user here - can't run these commands natively, and don't have WSL2 or working linux env. on my home pc. Not a huge deal for a small project, but most companies still make devs use windows, so shell scripts are a high barrier for us.

Although it wasn't obvious at a glance but easy enough to figure out that we can just add packages as a list instead of one by one:

poetray add pckg1 pckg2 pckg3

This makes the manual conversion pretty easy...

Big fan of poetry by the way.

[bolinocroustibat]

+1 for an export command from poetry.lock to requirements.txt format. Would help a lot for deployments.
+1 also for an import command from requirements.txt, also would greatly help Poetry adoption.

[gpchelkin]

+1 for an export command from poetry.lock to requirements.txt format. Would help a lot for deployments.

@bolinocroustibat Export is already supported: https://python-poetry.org/docs/cli/#export

[dashdanw]

to do this I have good luck running cat requirements.txt | poetry add --

[0atman]

@sdispater, thank you for your work on Poetry, I love it! I'm always looking out for projects to convert, which means I really would find an import command useful.

It seems to me, that If export is a valid use case, it's obvious that import is also a valid use case. I think this is a feature that many in the community, especially newbies, would really value.

Perhaps this feature should be "up for grabs"? I think that the positives outweigh the cli interface bloat negatives.
Let's get more people converting to the best tool: Poetry 💪

Edit: Wording. I wrote this in a hurry and it sounded rude, apologies. Re-written.

[vshulgin]

it is just one-liner in bash while read in; do poetry add "$in"; done < requirements.txt

[0atman]

@TalyGin does that work if you have pinned your requirements to specific versions in requirements.txt?

Poetry uses a different format, poetry add 'requests:<=3.0'

[sinoroc]

Isn't it something that dephell can do?

https://dephell.readthedocs.io/cmd-deps-convert.html

[vshulgin]

@TalyGin does that work if you have pinned your requirements to specific versions in requirements.txt?

Poetry uses a different format, poetry add 'requests:<=3.0'

Yes, it works, in input file ipaddress==1.0.22 in pyproject.toml I've got ipaddress = "1.0.22"

[buriy]

It works for == but not < nor > , nor for git deps.

…

On Fri, Nov 27, 2020 at 11:55 PM Vitaly Shulgin ***@***.***> wrote: @TalyGin <https://github.com/TalyGin> does that work if you have pinned your requirements to specific versions in requirements.txt? Poetry uses a different format, poetry add 'requests:<=3.0' Yes, it works, in input file ipaddress==1.0.22 in pyproject.toml I've got ipaddress = "1.0.22" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#663 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFFBDR7XEIMXE2ZKF3UXLSR7KZNANCNFSM4GF5PDLQ> .

-- Best regards, Yuri V. Baburov, Skype: yuri.baburov

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.