Git Dependencies do not respect lock file

[wakemaster39]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS, RedHat 7.6, Fedora 31

• Poetry version: 1.0.5

[tool.poetry]
name = "test2"
version = "0.1.0"
description = ""
authors = ["Cameron Hurst <cahurst@cisco.com>"]

[tool.poetry.dependencies]
python = "^3.8"
test-abc = {git="git@xxxx:xxxxxxxx/test-abc.git"}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Issue

I have created two test projects so I could verify this, simple contain no code just a git repo where one of them lives. The expectation I had, is what when I do a poetry install for the first time on my test2 repo that is shown above, a poetry lock file would be generated which contains the revision that the git should be locked against. This currently happens

[package.source]
reference = "6795f40dbe1f9a28dee1c3c5b2fae09763afa0be"
type = "git"
url = "git@xxxxx:xxxxx/test-abc.git"

I then go and push a change to test-abc and run poetry install again in test 2. My expectation here is that nothing changes and life is good. This is where I believe to be a Critical bug occurs:

Package operations: 0 installs, 1 update, 0 removals

  - Updating test-abc (0.1.0 6795f40 -> 0.1.0 a10f113)

The contents of the poetry.lock file remain unchanged, but I now have an updated dependency.

Now rev locking to that specific revision works, it is just the default does not respect the poetry.lock file defeating the purpose of the lock file here in my opinion.

There are a couple other issues I found on here which reference resolving tags and branches to specific hashes in the lock file, so it seems like the intent here is for git to install the specific hash and not latest by default.

[wakemaster39]

I have done more digging and made a PR, but I have run into one problem

poetry/tests/puzzle/test_solver.py

Lines 1276 to 1309 in 288d997

	def test_solver_git_dependencies_update(solver, repo, package, installed):
	pendulum = get_package("pendulum", "2.0.3")
	cleo = get_package("cleo", "1.0.0")
	repo.add_package(pendulum)
	repo.add_package(cleo)
	demo = get_package("demo", "0.1.2")
	demo.source_type = "git"
	demo.source_url = "https://github.com/demo/demo.git"
	demo.source_reference = "123456"
	installed.add_package(demo)
	package.add_dependency("demo", {"git": "https://github.com/demo/demo.git"})
	ops = solver.solve()
	check_solver_result(
	ops,
	[
	{"job": "install", "package": pendulum},
	{
	"job": "update",
	"from": get_package("demo", "0.1.2"),
	"to": get_package("demo", "0.1.2"),
	},
	],
	)
	op = ops[1]
	assert op.job_type == "update"
	assert op.package.source_type == "git"
	assert op.package.source_reference.startswith("9cf87a2")
	assert op.initial_package.source_reference == "123456"

I disagree with this unit test and the expectations it sets which has led me to file this bug but now looks like it is an intentional feature. The solver solves the problem based on the constraints it is given, but it seems like in this situation it is making assumptions about how constraints should be treated and ignored.

On an install, I would expect that the constraints of a lock file are respected no matter the condition and on an update a new set of constraints are generated and evaluated on the course of action that should be taken.

The installation process already has the update capability which generates a new lock file, putting in the new revision to lock the git repo to. But then the solver ignores that and just always updates. This seems like a bad situation and not respecting the poetry.lock file.

[al-muammar]

We as well were just affected by this bug, when a breaking change happened in one of our git dependency repositories, which broke our trunk and CI didn't catch it.
The lock file should be respected no matter what during the poetry install.

[fredrikaverpil]

On an install, I would expect that the constraints of a lock file are respected no matter the condition and on an update a new set of constraints are generated and evaluated on the course of action that should be taken.

+1 on this.

Although it may be a corner case, I have a scenario where I need poetry to install the latest and greatest from a git master on every poetry install. But then this behavior vs the one locked to a commit should have different declarations in pyproject.toml, and should also be supported from the CLI.

You can argue that "you should not do this" and so on, but it serves a real world scenario. I can agree that this behavior should perhaps not be the default though. So whatever happens, at least give the option to be able to fetch the latest and greatest and not remove this current functionality.

[hagaika]

i agree with @fredrikaverpil , as i have a scenario currently needs this behavior.
with respect to pyproject.toml my expected behavior would be:

[tool.poetry.dependencies]
...
private-repo-latest = {git = "ssh://git@bitbucket.org/<my-user>/<my-repo>.git", branch = "specific-branch"}
private-repo-specific-from-branch = {git = "ssh://git@bitbucket.org/<my-user>/<my-repo>.git", rev = "specific-branch"}
private-repo-specific-from-tag = {git = "ssh://git@bitbucket.org/<my-user>/<my-repo>.git", tag = "1.2.3"}

upon poetry install:
private-repo-lastest - would be latest and greatest from specific-branch
private-repo-specific-from-branch - would be a specific commit specified in poetry.lock file (the commit which was latest when we installed the package, or an updated commit if someone re-synced poetry.lock with poetry update or poetry lock)
private-repo-specific-from-tag - would always be a specified commit marked with that tag (obviously..). this is also the behavior if specified rev with a concrete commit hash.

i believe that this will cover all needed behaviors with the most simplistic and intuitive syntax.
would also be happy to work on that with a PR if core-contributors agree with that approach!

[samize]

Has any progress been made on this yet?

I have an issue where I'm trying to do like hagaika mentioned with:

private-repo = {git = "path/to/my.git", branch="specific-branch"}

I'm hoping to get it to install the latest version of that branch, but it doesn't detect it. Has this been resolved? If not, is there a temporary work around?

[mprudek]

Is this an issue anymore? I tried to reproduce the bug using poetry 1.1.13 and it seems that poetry install respects poetry.lock in all cases.

@wakemaster39 could you please confirm that the bug is still relevant in 1.1.13 and later?

[wakemaster39]

@mprudek this issue is 100% still a problem in the 1.1.x branch. My changes got merged into master (1.2.x) and never got the back port.

[mprudek]

@wakemaster39 I can confirm that I made a mistake during reproducing of the bug. I basically swapped words reference and resolved_reference.

During the first reproducing, my poetry.lock file looked like this:

 [package.source]
 type = "git"
 url = "ssh://git@github.com/user/repo.git"
 reference = "master"
 resolved_reference = "60997aafc40b9e652ba1414ceafd51b440ca672f"

And with this settings, poetry install really didn't installed anything.

But I overlooked that your poetry.lock was rather like this:

[package.source]
reference = "6795f40dbe1f9a28dee1c3c5b2fae09763afa0be"
type = "git"
url = "git@xxxxx:xxxxx/test-abc.git"

so I changed my poetry.lock so it looked like this:

 [package.source]
 type = "git"
 url = "ssh://git@github.com/user/repo.git"
 reference = "60997aafc40b9e652ba1414ceafd51b440ca672f"

... and then I updated & pushed the dependency repo, poetry install then definitely behaved weirdly and did install the latest version.

Thanks for patience 🙂 . I had the feeling that these observations could help also others.

[wakemaster39]

just incase anyone else also run into this. Another side effect going on here without my changes merged in, doing a poetry install or poetry update is not updating the installed version because it views it as a local package and not a git install to update. This means you need to wipe the virtual environment to get the latest after the initial install.

This one burns me alot as I forget that this occurs on anywhere I didn't apply my patch

[veselyvojtech]

My changes got merged into master (1.2.x) and never got the back port.

@wakemaster39 Which PR is it please? I would like to know what exactly is the syntax of the github dependency in pyproject.toml to make it work.

[wakemaster39]

Oh i didn't realize I failed to link it back here: #3867

[wakemaster39]

@veselyvojtech

Its not an issue with the pyproject, no syntax will make this work. Its an issue with the resolver and detection of it being a git package. The resolver needs the patch or it will just treat it like a local package and it will never function right.

[veselyvojtech]

Ok, thanks. So it is related to the git installation vs. directotry installation issue. The other issue about not respecting the resolved commit hash in poetry.lock is still there, right?

[wakemaster39]

#3875 should actually resolve the gitlock. That came out of my original concern. But again, stuck in 1.2.x branch only so still an issue in 1.1.x

[dimbleby]

Fixed at #3875, this can be closed

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.