Installation from path or url cannot be constrained to a version

[jackmen]

Dear cjolowicz,

first of all thank for a very useful package. It works like a charm for many of my applications as long as I'm only using official dependencies.

Problem:
When having URL-based dependencies in the pyproject.toml, the approach fails with the below error:

Processing ./dist/temporality_pipeline-0.1.0-py3-none-any.whl
ERROR: Could not satisfy constraints for 'my-package': installation from path or url cannot be constrained to a version

Code applied:
@nox.session(python=3.8, name="build-environment")
def build_environment(session: Session):
nox_poetry.install(session, nox_poetry.WHEEL)

Any suggestion on how to use my URL dependencies with your tool?

Thanks a lot!

[jackmen]

OK, I repeated the steps that your tool does under the hood and stopped after the creation of the requirements.txt, removed the '-e' before my URL dependencies and executed the remaining steps. That created the environment just fine. Could you add an option to enable this in your workflow? I'll see at my end if I find the time to do a PR.

Thanks!

[cjolowicz]

Thank you for taking the time to report this! 👍

Could you provide a series of steps for me to reproduce your issue? A link to your project, or its noxfile.py, pyproject.toml, and poetry.lock, would also be great.

In the past, I found it helpful to use a URL with an embedded hash. Here is an example using the ZIP archive for the GitHub repository of a Python package, with an embedded SHA256 hash:

[tool.poetry.dependencies]
foo = {url = "https://github.com/user/foo/archive/master.zip#sha256=43f4fa8d8bb313e65d8323a3952ef8756bf40f9a5c3ea7334be23ee4ec8278b6"}

You can determine the hash like this:

curl -fsSL https://github.com/user/foo/archive/master.zip | sha256sum

Thanks for offering to submit a PR! Contributions are always welcome and appreciated. I feel like I need to get a better understanding of the precise problem you're facing here. So it may be better to wait until we've talked over some options. One thing that comes to mind is a setting to generate the constraints file without hashes (poetry export --without-hashes).

[jackmen]

Dear cjolowicz,
thanks for your answer. Using the --without-hashes option does not solve the issue, because the URLs would still be entered in requirements.txt with -e.

I cannot send you an example as it is company stuff and the dependencies are not public. If needed, I can come up with an example repo....

Your suggestion creating hash might work, but that would require to add additional complexity to the creation of such repos etc.

I did do this to get what I want. No idea if this is something that you would consider a solution worth adding. Basically remove the '-e' before the URL dependencies:

@nox.session(python=PYTHON, name="build-environment")
def build_environment(session: Session):
    session.run("poetry", "build", external=True)
    session.run("pip", "uninstall", "--yes", "./dist/temporality_pipeline-0.1.0-py3-none-any.whl", silent=True)
    session.run("poetry", "export", "--format=requirements.txt", f"--output={ENVDIR}/requirements.txt", "--dev",
                "--without-hashes", external=True)
    session.run("sed", "--in-place=.bak", "-E", "s/^-e[[:space:]]+//", f"{ENVDIR}/requirements.txt", external=True)
    session.run("pip", "install", "-r", f"{ENVDIR}/requirements.txt",  external=True)
    session.run("conda", "pack", "-p", f"{ENVDIR}/build-environment", "-o", f"{ENV_ZIPDIR}/venv.tar.gz")

Cheers!

[cjolowicz]

Hi @jackmen is this still an issue with a current Poetry version?

Using Poetry 1.1.4, if you add a URL dependency and export to requirements.txt, the dependency is not editable:

$ poetry new example && cd example
$ poetry add git+https://github.com/pypa/packaging
$ poetry export -f requirements.txt | tee requirements.txt
packaging @ git+https://github.com/pypa/packaging@master ; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
pyparsing==2.4.7; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" \
    --hash=sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b \
    --hash=sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1
$ virtualenv venv
$ venv/bin/pip install -r requirements.txt 
Collecting packaging@ git+https://github.com/pypa/packaging@master
Collecting pyparsing==2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
ERROR: Can't verify hashes for these requirements because we don't have a way to hash version control repositories:
    packaging@ git+https://github.com/pypa/packaging@master from git+https://github.com/pypa/packaging@master (from -r requirements.txt (line 1))

To me, it looks like the problem is the missing hash, not the dependency being editable.

[cjolowicz]

Worth noting that paths and URLs are no longer supported in constraints files as of pip 20.3, which enabled the new dependency resolver by default. nox-poetry uses constraints files to translate Poetry's lock file into a format pip understands.

[jlujan]

So the solution is to downgrade pip <20.3? This also seems to be the case with local packages, i.e.

[tool.poetry.dependencies]
foo-bar = { path="./lib/foo_bar", develop=true }

It is not clear from the docs why the constraint file is used.

When installing third-party packages, Poetry is used to export a constraints file,

For the local package use case, where would I look in nox-poetry to have it wheel and install the dependency? Willing to create a PR for the local package support with some guidance.

[cjolowicz]

Hi @jlujan first off, thanks for providing your use case!

It is not clear from the docs why the constraint file is used.

The README says this in the section titled Why?:

When installing third-party packages, Poetry is used to export a constraints file, which is passed to pip along with the packages. The constraints file ensures that package versions are pinned by the lock file, without forcing an installation of every listed dependency and sub-dependency.

Do you find that this describes the reason for using a constraints file clearly? Should the explanation be more prominent in your opinion?

So the solution is to downgrade pip <20.3?

No, pip's behavior is not going to change any time soon.

This also seems to be the case with local packages [snip] For the local package use case, where would I look in nox-poetry to have it wheel and install the dependency? Willing to create a PR for the local package support with some guidance.

That would be out of nox-poetry's scope. Like Nox, nox-poetry does not attempt to build and install dependencies itself.

One thing nox-poetry could do though is to prune dependencies from the constraints file when they are not valid constraints under the new pip resolver, like URLs and paths. Your dependency should still get installed from the right location, because it is also a part of the package metadata of your main package. You can check if this approach would work for you by editing out the foo-bar dependency from the file .nox/<your-session>/tmp/requirements.txt, and running the session again.

Unfortunately, this approach will not cover all use cases (although I hope that it would cover the one you presented here). In particular, replacing one of your indirect dependencies by a URL or path cannot be translated into a constraints file under the new resolver, and as its also not part of your package metadata, it will simply be ignored when installing with nox-poetry.

Finally, I should mention that you may want to look into setting up a private package repository, instead of referencing local source trees. You can then install the dependency from your private repository as described in the Poetry documentation. Alternatively, if the dependency is open-source, you could also publish pre-releases or development releases to PyPI.

[jlujan]

setting up a private package repository, instead of referencing local

To clarify on the use case, these are local editable packages that share the same repo and are developed in unison so a package repository isn't viable.
I will mess around with the poetry generated requirements and see what I can figure out when I get a chance.
Thanks for the help and I will report back if I work something out.

[cjolowicz]

Hi @jackmen and @jlujan , a fix was released in nox-poetry 0.8.2.

[jackmen]

Nice, I'll have a look!

[brechtm]

Hi @jackmen and @jlujan , a fix was released in nox-poetry 0.8.2.

Mmm, If I understand the change correctly, this breaks pinning a dependency to a specific URL since package @ https://url.to/dir/package.zip (with out without hash) lines are removed from the constraints file.

EDIT: I tested with nox-poetry 0.8.1, which does keep the package @ url line in the constraints file and thus installs the pinned version in the nox env, as expected.