Handling of transitive dependencies and lockfiles

[silasdavis]

My apologies if this issue is being/has been addressed elsewhere. I did search and read through design documents linked on readme and I couldn't find it.

I have created an issue that concerns this matter in Glide: Masterminds/glide#479 and if @sdboyer is reading this you may recall we spoke before.

The issue I have is that I have direct deps that use glide and when I want to add such a direct dep then glide (and possibly gps) would use the manifest of the direct dep and not the lockfile to determine the version of transitive dependencies. In effect this tacitly updates my dependency's dependencies to versions my dependency was not tested against.

I do however see the counter-argument that the manifest should specify elastic constraints on its dependencies such that any satisfying versions would genuinely work together. However in reality this is rarely the case since people will often use constraints that would allow versions of their dependencies that will not work. And even with the best will in world and semver it is highly unlikely that a package has actually been properly tested over the power set of all its satisfying dependencies.

What I almost always want is for the transitive dependency versions to be set based on whatever is in the direct dependency's lockfile.

Provided no dependency shares any transitive dependencies then this is unproblematic (for me this is often the case). However if they do then you will have a conflict unless the lockfiles specify the exact same version. At this point we may want to fallback to resolution based on the manifests, and if there is still a conflict then there will have to be some notion of resolution strategies of manual overrides. It would still be really nice to support this lockfile based versioning (I'd argue as default first attempt). In my case I'd also like to be able to force it so that I have to deal with any conflicts manually.

Another approach that could work (and I think would work well for me) is to vendor transitive dependencies (recursively) into a direct dependency's repo. In effect this would be like recursively running dep ensure for any dependencies in the vendor tree. This would mean you could support multiple versions of a dependency (as specified by lockfile). I see you have thought about this looking at your 'Committee Features' doc. As you say there this is fine provided types from a dependency do not need to be shared at a 'more global' level (by packages above the current vendor folder depth). This then leads to ideas of vendoring at the most global level possible without getting a conflict.

So I feel your rabbit hole pain, but I really think the basic case of this would be nice to support without feeling like the full complexity needs to be dealt with by the tool.

Will it be possible to version deps from transitive lockfiles?

Here's my current hack around this issue with glide: https://github.com/silasdavis/glide-lock-transitive

[sdboyer]

Hi! i definitely remember that issue. Up front, let me note that this issue can really be treated separately from the question of allowing multiple versions. I was still feeling a lot of this out when we discussed last year, and so conflated those things together.

The short answer: preferred versions are alive and well in gps. The details of preferred version' algorithmic behavior are not exactly what you describe here, but they're as close as we can reasonably get. In the time since we last discussed, I also created some documentation that roughly explains their mechanics.

dep does not currently utilize preferred versions; we discussed it briefly within the committee, and decided to punt on it until later. Fortunately, it's essentially no more difficult than flipping a bool to enable them. That said, there is some complexity in controlling the interaction of preferred versions with update flags that I never got around to dealing with in gps - #440.

To my mind, the pros of preferred versions heavily outweigh the cons - assuming we get the CLI UI for it right. And that's nontrivial - there are some new flags and ideas we'll likely need to introduce, and it could have a bit of a nasty combining effect on some of our current flags.

I'd welcome someone wanting to work on #440, and trying to hash out what the CLI UI should look like. It'd be great to get that sorted out so that we can a) have a more precise discussion about what would be entailed by this, and b) easily flip the switch if we end up opting for this.

However, the discussion component is probably a bit blocked on #277, as we need to think about how this would factor in to the new command set, not the one we currently have. It's generally not productive to ask people to discuss additional, subtle changes to a not-yet-implemented spec for a tool they're still just getting used to.

In my case I'd also like to be able to force it so that I have to deal with any conflicts manually.

The best we'd likely be able to do here is improving result diff output to clearly indicate that a preferred version was not selected, making it easier for you to take corrective action and then re-run dep ensure. That's the use pattern we're optimizing for, as we believe it to be the best balance of many, many tradeoffs.

if @sdboyer is reading this

I AM ALWAYS READING 👀