Glide does not respect glide.lock for transitive dependencies

[silasdavis]

My project eris-db depends on a certain project in my glide.yaml:

- package: github.com/tendermint/tendermint
  version: 55ef4b225fb0aa5637a96d36a8c0d030a59dc21d

That project (https://github.com/tendermint/tendermint) has the following glide.lock: https://github.com/tendermint/tendermint/blob/55ef4b225fb0aa5637a96d36a8c0d030a59dc21d/glide.lock

This contains many transitive dependencies for my project, for example:

in its glide.yaml (@55ef4b225fb0aa5637a96d36a8c0d030a59dc21d):

- name: github.com/tendermint/tmsp

in its glide.lock (@55ef4b225fb0aa5637a96d36a8c0d030a59dc21d):

- name: github.com/tendermint/tmsp
  version: 7ffd2899092f47110a5ffebe20247a9b7f80f4ad

When I run glide up from my project...

What I expect to happen is for the transitive dependencies to be collated according to the glide.lock file of my direct dependencies, failing that their glide.yaml, failing that their Godeps.json. And the same for each of my transitive dependencies. So in my example I should get version 7ffd2899092f47110a5ffebe20247a9b7f80f4ad of github.com/tendermint/tmsp.

but what actually happens is glide uses the glide.yaml of github.com/tendermint/tendermint and fetches master of github.com/tendermint/tmsp

this is bad because glide has effectively forced a glide up upgrade of all my transitive dependencies, which may happen to be incompatible with my direct dependencies which I have pinned to a particular version.

the solution would be to give preference to glide.lock dependency versions of my dependencies unless I explicitly override them in my glide.yaml.

[silasdavis]

If a project baz has the following glide.yaml:

- name: foo
- name: bar
  version: 1.0.0

It occurs to me you could interpret this as "baz will work with any future version of foo that becomes available."

However, I think this is not a very useful interpretation as it is rather optimistic and assumes that glide up is not a destructive/dangerous action, which seems contrary to the core design. When I run glide up I know I have to make my project work again before committing my glide.lock.

So for me respecting the glide.lock makes more sense, and allows you to grab an entire transitive tree of dependencies that will work together (as they were checked in).

[silasdavis]

Another data point (before I stop talking to myself) is that I'm pretty sure the comparable Gemfile.lock of Ruby's Bundler would work in this way (building the dependency graph using the lockfile)

[mattfarina]

This is where deps start to get hard. Dependency management is sometimes called dependency hell (link is to wikipedia).

Top level applications are the only thing that really should have a lock file because of fluidity in dependency trees. For example, if you have dependencies A and B that both depend on C. If A and B depend on different versions you have a problem. They way bundler and others handle is it so use supported ranges. So, you can same A depends on >= 1.2.3, < 2 for C. And B can set >= 1.0.0, < 1.4.0. The goal of the dependency manager is to try and find the latest version that meets constraints.

If you pin to revision ids this causes a problem.

Right now Glide is strong in it's usage of outside metadata and will lesson over time as folks switch to version ranges and we improve the resolver (which is in development).

Glide uses versions to try and figure out the best and latest versions to fulfill the need. This is the same way bunder, npm, composer, and the others do it as well. When no version is specified we assume the latest commit on the default branch.

Does this make sense?

[sdboyer]

@silasdavis - just to add a bit to what @mattfarina said (which is all spot on)

the solution would be to give preference to glide.lock dependency versions of my dependencies unless I explicitly override them in my glide.yaml.

The new engine (#384) will - though perhaps not right away - allow using locked versions expressed in a transitive dep's lock file as a "preferred" version. Basically, that means it'll try to use the locked version, unless another constraint makes that impossible. This has been my planned solution to the issue you're raising for several months now; we'll have to see how it goes in practice, but I think it'll cover the issues you're concerned about.

[sdboyer]

Issue for lock-preferred versions is sdboyer/gps#16

[silasdavis]

Thanks for responses. Most projects I am depending on are not using version ranges in their glide.yaml, perhaps part of the answer is to see more tagging by dependencies, but there is still is a hangover from 'just depend on master' 'google does it' and there's not that much semantic versioning. For running such projects glide install usually gives you a working version, but this won't work for transitive dependencies. For these projects I need to manually manage my transitive dependencies, which glide can help with a bit, but then I'll need to manually fix.

I do see the generality though - if the glide.yaml capture all working versions with ranges then perhaps we can use that to find a mutually compatible version. However if this was the case, why bother having the lockfile? These approaches are slightly at odds. Though i can see myself wanting both, but the lockfile preferring one much more often in my case. The lockfile represents the set of dependencies that the person pushing their code has committed as working. When versions conflict you'll only be able to help some of the time anyway, so it seems better to try and use the locked version where possible.

Perhaps the general answer would be to allow to specify conflict resolution strategies. This adds complexity but probably necessary complexity. Gradle (which I'm sure you don't want to be as complex as!) allows you to specify default strategies or handle individual cases. @sdboyer the prefer lockfile functionality sounds like it will scratch my itch so I look forward to that.

[sdboyer]

Most projects I am depending on are not using version ranges in their glide.yaml, perhaps part of the answer is to see more tagging by dependencies, but there is still is a hangover from 'just depend on master' 'google does it' and there's not that much semantic versioning.

Sadly, this is the very large-scale catch-22 in which the Go community is currently caught. People do not use version ranges/semver - indeed, often no tagging at all - because there is no tooling to support it (c.f., golang/go#12302); at the same time, there is no tooling to support ranges and general matching because no one uses it. The only thing we can really do to break that up is to build a tool that supports version ranges/semver without hampering other use cases, and hope that then unblocks the other side of the equation.

glide aims to do just that, and once we get vsolver in, it'll do so, transitively and completely. (right now, glide still relies on "first dep to state a version requirement wins")

However if this was the case, why bother having the lockfile? These approaches are slightly at odds.

They're complementary, and used at different stages in the process. Manifests describe constraints with many possible solutions; lock files describe a complete, repeatable build, more or less as you noted. If you want a ludicrously-much longer explanation, I wrote this 😄 .

[sdboyer]

Oh, and...

Perhaps the general answer would be to allow to specify conflict resolution strategies. This adds complexity but probably necessary complexity. Gradle (which I'm sure you don't want to be as complex as!) allows you to specify default strategies or handle individual cases.

This is something we've started to explore. I have one pet strategy I'm particularly hot to try out, but a number are possible, and they can probably be mixed together. There's also benefit to be had from Go's easy, fast static analysis - we can do a lot of inference and possibility narrowing without needing user intervention.

IMO, though, we start dealing with conflict resolution strategies later - it's less of a priority than getting the community to take up some sane versioning strategies.

[silasdavis]

Actually I have read and shared your blog post before!

I do see the approach of having glide up upgrade everything including transitive dependencies using manifests, particularly when you have multiple paths to different versions of the same dependency -- perhaps you want to upgrade to the latest available for stability.

But the argument for glide uping upgrading your direct dependencies and then trying to use lockfiles for their transitive dependencies is one of repeatability.

I think supporting both of these is a good idea, and probably requires some sort of lightweight 'conflict resolution strategy' if you want it to work per dependency. Perhaps:

- name: github.com/tendermint/tmsp
  version: 7ffd2899092f47110a5ffebe20247a9b7f80f4ad
  transitive:
    - lockfile
    - manifest
    - godeps

or:

- name: github.com/tendermint/tmsp
  version: 7ffd2899092f47110a5ffebe20247a9b7f80f4ad
  transitive: ignore

The transitive directive gives a priority list of which strategy to use when resolving the chain of transitive dependencies (unless a link in the chain is overriden to use a different transitive dependency strategy). It seems like this would not be too hard to implement. Any transitive dependency conflicts can be resolved as they are now, by putting them explicitly in the glide.yaml.

[sdboyer]

Actually I have read and shared your blog post before!

yay! 😃 🎉 🎉

I do see the approach of having glide up upgrade everything including transitive dependencies using manifests, particularly when you have multiple paths to different versions of the same dependency -- perhaps you want to upgrade to the latest available for stability.

So, the way vsolver works right now (as this discussion is increasingly not applicable to glide in its current form) is to allow you to specify that all dependencies should be updated - that is, their locked versions (if any) should be disregarded - or that only certain specific dependencies should be updated. Neither of these let us bypass version constraints specified in manifests, of course - it just lets us explore the full range of version options allowed by those constraints.

But the argument for glide uping upgrading your direct dependencies and then trying to use lockfiles for their transitive dependencies is one of repeatability.

Indeed, there is additional stability afforded by relying on versions of transitive deps that your dep has actually tested with (we assume), as opposed to "should work with" as indicated by some version range constraint. And there's a strong argument to be made for erring on the conservative side, which would mean keeping those transitively-locked versions even when the "upgrade all" flag is passed (assuming they still satisfy version constraints).

I think supporting both of these is a good idea

Yeah, I think it might be a saner default to have things operate in this conservative fashion. I'll keep it in mind - I haven't really dealt with this specific question yet, as I haven't implemented the preferred versions logic. That said...

and probably requires some sort of lightweight 'conflict resolution strategy' if you want it to work per dependency.

It's important to clarify one thing here: there's two different senses of 'conflict'. One is simply where a version of a dep (e.g. one from a lock), doesn't satisfy a version constraint. This is a normal occurrence, and not really a "conflict" per se; when it happens, we just move along to the next available version and try again.

Another instance of this first type of conflict occurs if we encounter two separate project's constraints on a third project that are mutually exclusive. This may seem like a stronger class of conflict, but we still handle it the same way: keep trying other versions to see if any of them DON'T have mutually exclusive constraints.

The second type of conflict - when we might want to consider a conflict resolution mechanism - only occurs when we've exhausted the queue of possible versions for a given project, with none of them satisfying all the various constraints. Right now in the solver, the only option we have is to trigger backtracking - essentially, walking even further back up the versions we've selected and trying new combinations, to see if some other combination of versions can work everything out.

In the future, however, we could also choose to sidestep such type-2 conflicts. My aforementioned pet strategy is to see if it's safe to just allow the conflict, which we could do by putting one version of the conflicted transitive dep under a nested vendor dir, and keeping the other version in the top-level vendor dir. "Seeing if it's safe" is quite hard, though, and this is an approach with serious possible side effects. Not something I'm approaching lightly.

Per-dependency is...even harder, and presents a level of choice to the user that I think would be more hindrance than help. To that end...

- name: github.com/tendermint/tmsp
  version: 7ffd2899092f47110a5ffebe20247a9b7f80f4ad
  transitive:
    - lockfile
    - manifest
    - godeps

or:

- name: github.com/tendermint/tmsp
  version: 7ffd2899092f47110a5ffebe20247a9b7f80f4ad
  transitive: ignore

Hmm...how to explain this.

If we were to allow these additional properties, I think we actually make the overall problem worse. What happens when two different projects, both relying on github.com/tendermint/tmsp, specify different directives for transitive? What does it even mean to disagree on that property? Is there a way to determine equivalency, like we do with versions by transparently looking at the underlying revision?

The only real answer is that we have to consider that disagreement itself a type-1 conflict. Maybe, maybe, we could say, "if one is root then override the other," but that just solves one problem. What we've really done here is just open up more surface area for disagreement between projects, and on a fairly abstruse point that, I'm pretty confident, almost no one will really know WTF's going on when the solver fails because of a conflict like this. Inscrutable solver failures are the path to dependency hell :)

We have to consider them a disagreement because those directives would necessarily have to control the way that information itself flows into the solving process, rather than just being different information within the structured flow. I'm honestly not even sure what effect such changes in the information flow would have on the algorithm's consistency or correctness; that's a level of meta-control that I have not yet seen the need to allow in vsolver (which, to be clear, is generally already quite abstracted and flexible).

For reference, the way vsolver works is to have a single ProjectAnalyzer which the tool relying on vsolver injects. The ProjectAnalyzer is responsible for deciding how to interpret all types of manifests and/or locks. What that means should be reasonably evident from the WIP analyzer for glide. So, a tool tells vsolver what the version constraints are, but vsolver is still deciding the order in which to do things. (SAT solvers are incremental algorithms, and as such owe most of their mechanics and correctness properties to ordering.)

Now, this might actually be loosely compatible with what you're saying - glide could allow directives like that in the yaml file, and use that to determine how its analyzer works. But the engine is still going to decide the order by which possible solutions are visited and tested, as its fundamental responsibility is dealing with type-1 conflicts in an ordered, reliable fashion.

In general, I think the much simpler solution is the 'lock-preferred' approach with perhaps another SolverOpts property that allows for granular control over that. That, and/or providing overriding control to the root project: if you want a particular transitive dep not to change, then you name an override constraint for it in the root. Overrides - not yet implemented, but more or less trivial - let the root project control the depgraph, but are ignored if the project is not the root. That way, you don't mess up other peoples' flexibility.

[sdboyer]

Just updating to note that sdboyer/gps#16 is now done :)

[silasdavis]

Great, do we need to apply any config to give preference to dependency lockfiles, or will that now be the default?

[sdboyer]

@silasdavis no config will be needed. However, since this is part of the new vsolver engine, the behavior won't be available until the new engine is merged - #384.

[silasdavis]

Any idea of an ETA on this? What's the relevant branch where the dependency work for this is now happening?

[sdboyer]

gps-integration is where the work currently is. things have gotten a bit sidetracked in the last month, and my obligations to the pm committee have cut into the time i'd have for this, as well. so, it's progressing, but i'm sorry, it would be unwise to give an ETA :(

there's also #565 , which is a bit of a checklist of issues.

[silasdavis]

I made a glide plugin to address my problems. It's not hugely general or very beautiful but it works for me: https://github.com/silasdavis/glide-lock-transitive.

@sdboyer informs me that preferred versions will provide a way to do this in golang/dep#622