Pre-release version of a package not found even with `--allow-prereleases` #3538

kubami · 2021-01-04T10:09:38Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: archlinux
Poetry version: 1.1.4
Python version: 3.7.7

Issue

I am trying to install a pre-release verion of https://github.com/elastic/enterprise-search-python

Their recommended installation method is

python -m pip install --pre elastic-enterprise-search
which works fine.

When trying to install it with poetry using

poetry add --allow-prereleases elastic-enterprise-search
I am getting an old version: 0.2.1 (the same as the one without the --allow-prereleases switch.

Am I doing something wrong? Or is it that this package is configured incorrectly?

The text was updated successfully, but these errors were encountered:

kubami · 2021-02-10T16:42:07Z

when trying to install as above it doesn't work.
however when I specify the version explicitly the page installs correctly

# pyproject.toml
[tool.poetry.dependencies]
elastic-enterprise-search = {version = "7.10.0b1", allow-prereleases = true}

markf94 · 2021-03-31T19:54:56Z

I second this! I'm unable to install pre-release versions with the --allow-prerelease flag using either Poetry 1.1.4 or 1.1.5. Even did a fresh reinstall of poetry on my system and still have the same problem. However, it does work for my colleagues with the same/similar setup.

Also verified that I can install the pre-release version when explicitly defining its version in pyproject.toml and it works in that case. However, I would like it to always pick the latest pre-release...

ClementWalter · 2021-04-14T08:39:27Z

I have the same problem with poetry 1.1.5. All private repos so I can't point out to the given packages, but essentially:

mypackage = { version = "^3.0.0", allow-prereleases = true}

does not resolve to using the 3.0.0.dev0 version, while if I write directly:

mypackage = { version = "^3.0.0.dev0", allow-prereleases = true}

poetry works fine.

But maybe there is a standard way of defining prerelease versions I am missing?

alexporter8013 · 2021-05-19T01:39:21Z

looks like doing:

mypackage = { version = "^3.0.0*", allow-prereleases = true}

works fine for poetry 1.1.5 works just fine for me. If allowing pre-releases just adding a wildcard * is probably the best bet to get the latest pre-release.

Fix from python-poetry/poetry#3538 (comment)

bushshrub · 2022-05-18T04:45:40Z

I know this is an old issue but this still seems to be a problem.

abn · 2022-05-19T15:38:37Z

@xiurobert did you try the version on master? This might have been fixed by #5500.

dimbleby · 2022-05-19T16:39:51Z

I think that poetry takes the view that

^3.0.0 is asking for something >= 3.0.0
3.0.0b1 is strictly before 3.0.0

in which case I wouldn't expect #5500 to make a difference. But also it's not clear to me that this is wrong!

For instance today (poetry 1.2.0b1 exists but 1.2.0 does not):

poetry = { version = "^1.0.0", allow-prereleases = true} does allow version 1.2.0b1 to be installed
but poetry = { version = "^1.2.0", allow-prereleases = true} fails, finding no matching versions

which is probably correct, there is no available version that is >= 1.2.0.

Possibly this should be documented or documented better, I haven't checked what the docs do say but clearly this is confusing to lots of people.

abn · 2022-05-19T18:37:06Z

^3.0.0b1 == { version = "^3.0.0", allow-prereleases = true } -> this includes all pre-releases in the lowerbound. Unless the upperbound is <= pre-releases there are not included (iirc).

dimbleby · 2022-05-19T19:02:42Z

@abn I'm struggling to parse that! Really not sure what you mean?

abn · 2022-05-19T19:13:57Z

Should have explained it clearer. Effectively when searching for a compatible version ^3.0.0 with allow-prereleases = true behaves the same as ^3.0.0b1. The behaviour being, 3.0.0a1 or earlier is disallowed, 3.0.0b1 | 3.0.0b2 | 3.0.0rc1 | 3.0.0 etc. are allowed.

dimbleby · 2022-05-19T19:24:21Z

You mean: that's what you think the behaviour should be? Per my example - and indeed this issue - that's not what the behaviour is.

(Isn't 3.0.0a1 just as good a pre-release version as 3.0.0b1 - why should one but not the other be disallowed?)

abn · 2022-05-19T19:47:00Z

When selecting packaegs if dependency allows for prereleases, they are selected else discarded.

poetry/src/poetry/repositories/repository.py

Lines 59 to 63 in c880cd7

    
           if constraint.allows(package.version) or ( 
        
               package.is_prerelease() 
        
               and constraint.allows(package.version.next_patch()) 
        
           ): 
        
               packages.append(package)

When finding the best candidate, we select the prerelease if there are only prereleases available and/or preleases are explicitly allowed.

poetry/src/poetry/version/version_selector.py

Lines 47 to 51 in c880cd7

    
           if ( 
        
               candidate.is_prerelease() 
        
               and not dependency.allows_prereleases() 
        
               and not only_prereleases 
        
           ):

(Isn't 3.0.0a1 just as good a pre-release version as 3.0.0b1 - why should one but not the other be disallowed?)

All pre-releases are not made equal. This is more about PEP 440 that poetry.

Pre-releases for a given release are ordered first by phase (alpha, beta, release candidate) and then by the numerical component within that phase.

Ref: https://peps.python.org/pep-0440/#pre-releases

dimbleby · 2022-05-19T19:56:12Z

Yes, I found the same code - but I don't find any code in the solver that would pay attention to allow_prereleases. ie my interpretation is that poetry today does allow pre-releases to be found in repositories, but the flag does not make any difference to how it interprets constraints.

This is consistent with the observed behaviour - it'll use poetry 1.2.0b1 if the constraint is ^1.0.0, but not if the constraint is ^1.2.0.

My assertion is that this is reasonable, if confusing, it sounds as though you are more willing to call this a bug.

(The a1 vs b1 thing is a bit of a side-issue, perhaps I should drop it. But the flag is called allow-prereleases, not allow-prereleases-except-not-alpha-prereleases!)

abn · 2022-05-19T20:14:42Z

(The a1 vs b1 thing is a bit of a side-issue, perhaps I should drop it. But the flag is called allow-prereleases, not allow-prereleases-except-not-alpha-prereleases!)

Oh what i meant was if you said ^3.0.0b1 this will exclude since b1 is the lower bound and it implicitly allows pre-releases. Think that was miscommunicated.

the flag does not make any difference to how it interprets constraints.

This is correct. The solver works only on the pre-selected candidates iirc. And since the allow-prereleases flag is a Poetry feature, it is not propagated to the candidate dependencies.

My assertion is that this is reasonable, if confusing, it sounds as though you are more willing to call this a bug.

I not saying you are wrong here. I am indicating that the there is a bug in the candidate selection not the solver - ie. we were not selecting pre-releases when searching for package candidates.

bushshrub · 2022-05-25T17:16:00Z

@xiurobert did you try the version on master? This might have been fixed by #5500.

Sorry, didn't get the chance to test it. I ended up returning back to virtualenvs. Will give it a shot again when I work on my personal projects.

dbrennand · 2022-06-01T17:20:03Z

I'm having some trouble installing python-telegram-bot pre-release version 20.0a0:

Poetry version 1.1.13

# pyproject.toml
[tool.poetry.dependencies]
python = "^3.10"
# 1 - Installs version 13.12
# python-telegram-bot = {version = "^13.12", allow-prereleases = true}
# 2 - SolverProblemError
# python-telegram-bot = {version = "20.0a0", allow-prereleases = true}
# 3 - SolverProblemError
# python-telegram-bot = {version = "^20.0a0", allow-prereleases = true}

I'm quite confused and not sure how I can install the pre-release 😞

dimbleby · 2022-06-01T22:11:56Z

2 installs python-telegram-bot 20.a0 just fine for me at both 1.1.13 and master.

dbrennand · 2022-06-03T15:12:36Z

2 installs python-telegram-bot 20.a0 just fine for me at both 1.1.13 and master.

Huh, you're right. Not sure what I was doing wrong earlier 😅 😆 Thanks 👍🏻

user@computer:~/RSS_Feederbot$ poetry --version
Poetry version 1.1.13
user@computer:~/RSS_Feederbot$ cat pyproject.toml
[tool.poetry]
name = "rss_feederbot"
version = "1.0.0"
description = "An RSS feed reading Telegram bot"
license = "MIT"

[tool.poetry.dependencies]
python = "^3.10"
reader = "^2.12"
loguru = "^0.6.0"
python-telegram-bot = {version = "20.0a0", allow-prereleases = true}
emoji = "^1.7.0"

[tool.poetry.dev-dependencies]
black = "^22.3.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
user@computer:~/RSS_Feederbot$ poetry install

TBBle · 2022-06-07T10:58:12Z

This looks like the original request is working now.

Testing on Poetry (version 1.2.0b2) (which includes #5500), in a trivial project using poetry-1.2 init -n and then poetry-1.2 lock. (poetry-1.2 is because I'm using pipx, and I already have Poetry 1.1.13 as poetry)

poetry-1.2 add --lock --allow-prereleases "elastic-enterprise-search<8.2.0"

chose 8.1.0a1, which is the latest pre-release before 8.2.0. Without --allow-preleases, it chooses 7.17.0.

poetry-1.2 add --lock --allow-prereleases "opentelemetry-api"

chose 1.12.0rc1 (it guessed ^1.12.0rc1 for the version match), which is a newer pre-release than the latest release. Without --allow-prereleases it chooses 1.11.1, with guessed version match ^1.11.1.

poetry-1.2 add --lock "opentelemetry-semantic-conventions" --allow-prereleases

chose 0.31b0 (it guessed ^0.31b0) as this is the latest pre-release, and there has never been a non-pre-release. It does the same thing without --allow-prereleases. I note that the pyproject.toml doesn't record the allow-prereleases flag in this case, because it wasn't provided, but a pre-release version was selected. I'm not sure if there's any scope for this to be an issue though, since once we have the guessed version match, that will accept pre-releases anyway, irrespective of the flag.

i.e.

poetry-1.2 add --lock "opentelemetry-api=^1.12.0rc0"

happily selected 1.12.0rc1.

So perhaps --allow-prereleases could be slightly-better documented that it only applies when guessing a constraint for names provided without constraints (which is probably the common use-case).

Although if that's true, I'm not sure why it's recorded in pyproject.toml, since that last command, with or without --allow-prereleases has only the following, non-meaningful differences:

diff --git a/poetry.lock b/poetry.lock
index 400fdf9..07b5111 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -48,7 +48,7 @@ python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7"
 [metadata]
 lock-version = "1.1"
 python-versions = "^3.10"
-content-hash = "1ede8f5bb37ee43871f3bfbaee39e1abd027d0443aa65b5d7319abb2c0aac657"
+content-hash = "eb1e42b0bf10677fc70e389d22034d17f0c2ecfa35f508d8f6f3997a03a7777a"

 [metadata.files]
 deprecated = [
diff --git a/pyproject.toml b/pyproject.toml
index 16d3cef..609e097 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -7,7 +7,7 @@ readme = "README.md"

 [tool.poetry.dependencies]
 python = "^3.10"
-opentelemetry-api = "^1.12.0rc0"
+opentelemetry-api = {version = "^1.12.0rc0", allow-prereleases = true}


 [build-system]

i.e. in both cases it guessed the same constraint, locked the same packages, etc.

wolph · 2022-10-06T10:14:11Z

Previously I was able to use ^3.0.* to select the latest pre-release, but it seems that's broken now. Is there a new syntax that works for the latest pre-release?

What doesn't work:

aiogram = { version = "^3.0.*", allow-prereleases = true}
aiogram = { version = "3.0.*", allow-prereleases = true}
aiogram = { version = "3.*", allow-prereleases = true}
aiogram = { version = "^3.0.0", allow-prereleases = true}
aiogram = { version = "^3.0", allow-prereleases = true}
aiogram = { version = "^3", allow-prereleases = true}

dimbleby · 2022-10-06T10:54:30Z

all of the above are asking for a version that is equal to or later than 3.0.0. A pre-release is before 3.0.0.

Provide a version bound that does not rule out the pre-release you want eg <4

neersighted · 2022-10-06T15:36:47Z

This issue is a bit of a jumble -- as far as I can tell however, things are currently as they should be. That is to say, --allow-prereleases works as expected during poetry add, and interacts properly now with version constraints.

If you want to use ^ with a beta release, you need to realize the part that comes after ^ sets the lower bound. So ^3.0.0a1, with allow-prereleases = true will take the latest prerelease greater to alpha 1 or later. ^3 with allow-prereleases = true will work for 3.0.1a1, but that is because it comes after the lower bound of 3.0.0.

I'm closing this for now -- if you have a request/idea for docs improvement, please open a docs issue. If you think you are having trouble with allow-prereleases itself (as opposed to figuring out the correct version specifier for your use case), please feel free to chime in here -- but I would make sure your usage of the version specifier is correct first.

wolph · 2022-10-06T19:03:46Z

all of the above are asking for a version that is equal to or later than 3.0.0. A pre-release is before 3.0.0.

Provide a version bound that does not rule out the pre-release you want eg <4

That makes sense... silly of me. I just find it odd that ^3.0.* worked earlier and stopped working now

Only encountered this in CI, where poetry 1.2 started getting installed (https://github.com/sjdemartini/graphene-django-permissions/actions/runs/3284716438/jobs/5411001549), since originally I was using Poetry 1.1 locally: ``` poetry.core.constraints.version.exceptions.ParseConstraintError: Could not parse version constraint: ^3.0.0* ``` Poetry 1.2 (released August 2022) contained a backwards incompatible change for `allow-preleases` logic, which led to this breakage (and difference from 1.1). See python-poetry/poetry#3538 (comment) for the new preferred approach.

ozancaglayan · 2023-09-01T15:28:13Z

Hello,

Is this supposed to work if we keep the version the same (e.g. rc1) but force re-releases into a development PyPI? This changes the buildID of the underlying package but I can't make poetry to fetch the new one unless I append the buildID to the dependency in pyproject.toml

github-actions · 2024-02-29T07:06:24Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

kubami added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Jan 4, 2021

honzajavorek added a commit to juniorguru/junior.guru that referenced this issue Jul 14, 2021

fix pre-releases for gql

d1e5e23

Fix from python-poetry/poetry#3538 (comment)

joukos mentioned this issue Jan 22, 2022

Installing RPi.GPIO fails with Python 3.9 joukos/PaperTTY#87

Open

turbomam mentioned this issue May 9, 2022

linkml/linkml PR testing new common metadata in linkml-runtime 1.3.0rc1 linkml/linkml#756

Closed

TBBle mentioned this issue Jun 7, 2022

poetry won't check for pre-releases during dependency resolving of non-direct dependencies #4405

Closed

3 tasks

neersighted closed this as completed Oct 6, 2022

neersighted removed the status/triage This issue needs to be triaged label Oct 6, 2022

neersighted added the area/solver Related to the dependency resolver label Oct 6, 2022

github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pre-release version of a package not found even with `--allow-prereleases` #3538

Pre-release version of a package not found even with `--allow-prereleases` #3538

kubami commented Jan 4, 2021

kubami commented Feb 10, 2021

markf94 commented Mar 31, 2021 •

edited

Loading

ClementWalter commented Apr 14, 2021

alexporter8013 commented May 19, 2021

bushshrub commented May 18, 2022

abn commented May 19, 2022

dimbleby commented May 19, 2022 •

edited

Loading

abn commented May 19, 2022 •

edited

Loading

dimbleby commented May 19, 2022

abn commented May 19, 2022

dimbleby commented May 19, 2022 •

edited

Loading

abn commented May 19, 2022

dimbleby commented May 19, 2022

abn commented May 19, 2022

bushshrub commented May 25, 2022 •

edited

Loading

dbrennand commented Jun 1, 2022 •

edited

Loading

dimbleby commented Jun 1, 2022

dbrennand commented Jun 3, 2022

TBBle commented Jun 7, 2022 •

edited

Loading

wolph commented Oct 6, 2022

dimbleby commented Oct 6, 2022

neersighted commented Oct 6, 2022

wolph commented Oct 6, 2022

ozancaglayan commented Sep 1, 2023

github-actions bot commented Feb 29, 2024

Pre-release version of a package not found even with --allow-prereleases #3538

Pre-release version of a package not found even with --allow-prereleases #3538

Comments

kubami commented Jan 4, 2021

Issue

kubami commented Feb 10, 2021

markf94 commented Mar 31, 2021 • edited Loading

ClementWalter commented Apr 14, 2021

alexporter8013 commented May 19, 2021

bushshrub commented May 18, 2022

abn commented May 19, 2022

dimbleby commented May 19, 2022 • edited Loading

abn commented May 19, 2022 • edited Loading

dimbleby commented May 19, 2022

abn commented May 19, 2022

dimbleby commented May 19, 2022 • edited Loading

abn commented May 19, 2022

dimbleby commented May 19, 2022

abn commented May 19, 2022

bushshrub commented May 25, 2022 • edited Loading

dbrennand commented Jun 1, 2022 • edited Loading

dimbleby commented Jun 1, 2022

dbrennand commented Jun 3, 2022

TBBle commented Jun 7, 2022 • edited Loading

wolph commented Oct 6, 2022

dimbleby commented Oct 6, 2022

neersighted commented Oct 6, 2022

wolph commented Oct 6, 2022

ozancaglayan commented Sep 1, 2023

github-actions bot commented Feb 29, 2024

Pre-release version of a package not found even with `--allow-prereleases` #3538

Pre-release version of a package not found even with `--allow-prereleases` #3538

markf94 commented Mar 31, 2021 •

edited

Loading

dimbleby commented May 19, 2022 •

edited

Loading

abn commented May 19, 2022 •

edited

Loading

dimbleby commented May 19, 2022 •

edited

Loading

bushshrub commented May 25, 2022 •

edited

Loading

dbrennand commented Jun 1, 2022 •

edited

Loading

TBBle commented Jun 7, 2022 •

edited

Loading