Must require specifying the type of version constraint

[sdboyer]

So, in working on #384 (holy crap, it's coming along even better than I thought!), I ran into this little problem:

[ERROR] Could not find any versions of github.com/Masterminds/semver that met constraints:
    1.1.0: Could not introduce github.com/Masterminds/semver at 1.1.0, as it is not allowed by constraint >=2.0.0, <3.0.0 from project github.com/Masterminds/glide.
    1.0.1: Could not introduce github.com/Masterminds/semver at 1.0.1, as it is not allowed by constraint >=2.0.0, <3.0.0 from project github.com/Masterminds/glide.
    1.0.0: Could not introduce github.com/Masterminds/semver at 1.0.0, as it is not allowed by constraint >=2.0.0, <3.0.0 from project github.com/Masterminds/glide.
    2.x: Could not introduce github.com/Masterminds/semver at 2.x, as it is not allowed by constraint >=2.0.0, <3.0.0 from project github.com/Masterminds/glide.
    master: Could not introduce github.com/Masterminds/semver at master, as it is not allowed by constraint >=2.0.0, <3.0.0 from project github.com/Masterminds/glide.

This, resulting from the following (abbreviated) glide.yaml for glide itself:

package: github.com/Masterminds/glide
import:
- package: gopkg.in/yaml.v2
- package: github.com/Masterminds/vcs
- package: github.com/codegangsta/cli
- package: github.com/Masterminds/semver
  version: 2.x # <-- this is the problem
- package: github.com/sdboyer/vsolver

I knew this was gonna be an issue in general when I started implementing vsolver's interfaces on glide's Config and Lock types, but I didn't realize it was gonna crop up so soon.

The problem here is that 2.x - a valid semver string - is actually intended to be a literal branch name. But, because we have no guidance as to the type of constraint it's supposed to be, we just have to guess. I assume that right now, glide gets away with this by just checking to see what exists in the target repository, and assuming that if there's a direct string match, that's what the user wants. That's problematic, though - for one, it requires touching network and/or disk in order to verify.

More importantly, though, it means that the pure, locally-controlled inputs (manifest and static analysis of local project files) are no longer sufficient to govern the behavior of the algorithm. For example, in this particular case, if we rely on inferring from the list of branches in semver that 2.x is actually a branch, then that constraint becomes a branch - if, however, at some later time that branch goes away and we solve again, then we'd interpret that constraint as a semver range. Externalities are affecting the way we interpret inputs. And when the meaning of inputs are dependent on external factors, memoizing becomes pointless...so, the hash digest comparison that glide already does becomes kinda pointless.

The only solution here, really, is to add a new field that allows the user to specify the type of constraint they intend to provide. We can make this less onerous by letting the preferred constraint type be an assumed default. The approach that makes the most sense to me is:

1. If no constraint type is specified, assume it's either semver or version - but, either way, tied to a non-floating version. semver's parser can then differentiate between those two.
2. branch must be specified as the constraint type if you want to track a branch
3. revision must be specified as the constraint type if you want to fix to an immutable revision

We could also throw a little magic in there like what I already wrote that tries to infer if it's a revision by seeing if it's 40 hex chars. I think the cost might outweigh the benefit there, though.

[mattfarina]

@sdboyer First, I'm really glad this happened because it means we get to catch the issue and work it out before end users have to deal with this. Thanks for bringing this up explicitly.

Here is the start of me trying to think it through (it's a little stream of thought)...

The first thing we have to do is be pragmatic and build a system with the least about of WTF.

To put it another way, we need to build a system that minimizes the number of support requests (because many users will walk away at issues and not file a request and those that do we don't have piles of time to help).

This makes me relate to the repo and vcs properties. Originally Glide followed the same patterns that go get did for specifying repos and identifying the type. For the cases it couldn't happen automatically from the repo content the vcs property could explicitly hold the type.

But, there were numerous times people expected the repo to be auto-detected when it wasn't. I would get asked, why wouldn't git@example.com:foo/bar be detected as git? Why do that extra work when the tooling can figure it out?

People wanted to automation to detect common use cases.

I think adding an additional property to specify the type of version falls when it comes to making users think. Today it can figure out the type but requires some extra computational work (including talking with the repos). Asking people to add something extra will both not be compatible with glide.yaml files today (possibly breaking some CI toolchains) and be extra non-obvious work.

But, let's continue but digging into the technical...

Glide today checks if something is an existing reference (tag, branch, or commit id). If it's not it then goes down the SemVer path. Internally, a VCS reference is prioritized higher than SemVer.

The systems, like Crates or NPM, that have a central repo of packages are able to enforce SemVer in the packages that are pushed. So, we can't look to them for guidance because we can't control things to the same level.

Packagist is a hybrid solution. It talks to the systems, such as GitHub, for the code while holding a metadata store in the registry. Composer will let me specify branches using the syntax dev-[branch name]. Or, if config maps a branch to a version range it could be 3.0.x-dev. For an example see the versions for html5.

To keep the metadata up to date Packagist does daily polling or allows web hooks to be setup when code is pushed. This allows the central repo to keep up to date. A client side thing handling versions has to pull to get the latest versions.

If there is going to be a local cache of possible values it will need to be derived from repos. If that cache holds the versions why can't it hold all the branches, tags, and versions knowing which is which? Why can't that be calculated automatically and stored in some local cache for the software to use later?

This could be useful because branches or unknown versions would force going out to the network to check for updates.

Thoughts?

[sdboyer]

because it means we get to catch the issue and work it out before end users have to deal with this.

indeed!

Asking people to add something extra will both not be compatible with glide.yaml files today (possibly breaking some CI toolchains) and be extra non-obvious work.

I don't think it'd be incompatible with existing glide.yaml files, at least at the level of parsing them. It'd just be adding a field. The addition of the field could, however, change the meaning of existing files - such as in the case we're seeing right now, where we want it to interpret 2.x as a branch, rather than a semver constraint. So, it's that semantic shift that you're referring to, I'm guessing?

It can certainly be a pain to have to specify extra data. But in this case, we're not actually talking about something the tool can figure out. 2.x, without more information, is ambiguous between a branch (or tag, for that matter) named 2.x, and >=2.0.0 <3.0.0. It doesn't matter how much information we harvest from existing repositories; it's still ambiguous input.

So...

If there is going to be a local cache of possible values it will need to be derived from repos. If that cache holds the versions why can't it hold all the branches, tags, and versions knowing which is which? Why can't that be calculated automatically and stored in some local cache for the software to use later?

It's not that we couldn't do this. We can, and a fair bit of it's already in place. It's that when we rely on the information available from upstream repositories in order to divine the meaning of a local constraint declaration (like 2.x), then the local system is no longer in control.

Here's the not-unlikely nightmare scenario: you're working away on a large project, and you have some dep, A, which in turn has a dep on B constrained to 2.x. Because B has no version literally named 2.x, this gets interpreted as a semver range (which is what A wanted).

In the middle of your workday, the author of B pushes some new branch, called 2.x, to their upstream repository. A few minutes later, you glide get some other dependency, C. The solver runs because we need to incorporate C, and in doing so necessarily visits B, too. When it does, instead of interpreting the 2.x constraint on B as a semver range (as it had the last literally thousand times you'd run a get/up/install/sync/whatever), now the solver interprets that as referring to the branch.

From here, there are several possible outcomes:

1. If there's no other dependency on B, then great! ...sort of. Because the code that's in the 2.x branch may be vastly different than any of the released versions admitted in the >=2.0.0 <3.0.0 range - we don't even have the informal guarantees of semver to suggest otherwise. So, the solver reports success and writes out a new lock. It might be:
   1. You get lucky, and everything just works fine.
   2. The types are compatible, but something not encoded in the type system changes, and now you have bugs that you really didn't expect to have, because...why did your version change in the first place?
   3. There's type incompatibilities, and so it won't compile. Eventually, I may make this a satisfiability condition the solver checks, too - in which case, there'll now be a new, confusing failure totally unrelated to C (but you assume is related to C, because that's all that you changed)
2. Or, maybe, C also has a dependency on B with ~2.0.0, which is semantically equivalent to 2.x. However, ~2.0.0 is still interpreted as a semver range, which means the solver suddenly starts failing with constraint type-level disagreement (branch literal vs. semver) between A and B.
3. The previous problem, except instead of it being C (the project you were geting), it's D, some other project on which you already had a dependency. Again...you'd assume the issue is with C, because that's what you'd changed. But you'd be wrong.

There's some more, too, but that gives a flavor. (Remember, also, that having this issue makes the hash stored in the lock file meaningless. That is a huge loss.) I could easily see people spending hours tearing their hair out over this; these issues could easily be experienced as schroedinbugs, which are...a very special circle of hell.

The only real solution, as I initially suggested, is to make the constraints fully self-contained, without the need to contact any external system in determining their meaning. That's really not hard to do, but its effect on sane system design is enormous. Having/not having a central registry becomes irrelevant (on this specific issue). The drawback of that, as you noted, is the extra work for users. I think, though, that that's quite surmountable. I'll write a second comment for that...

[sdboyer]

When it comes to a solution, I can't see a way around needing to have the user indicate the type of version, as well as the version itself, in an unambiguous way. The goal is to not have any distributed system-like failure conditions:

A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.

• Leslie Lamport

IMO, encoding the type within the version string itself is never really going to be sufficient - e.g. a tag named dev-foo that is itself a tag, but composer's system would view as a branch named foo (that doesn't exist). That's just one example, but trying to encode multiple categories of meanings into a single string space is always going to be awkward. I'd rather have a separate field, where things can be quite clear, albeit more verbose.

With a separate field as the goal, I think there are three issues to address:

1. How do we still make it easy for users to have the tool do autodetection for them, but avoid ambiguity?
2. Do we optimize defaults for the short term or the long term?
3. How do we deal with existing glide.yaml files?

User ease

The big issue here, I think, is not requiring users to manually enter a version type all the time. One way to do this is with autodetection. The change I'd propose from right now is that that autodetection is done once by the initial stages of glide get initial fetching, rather every time we enter a solver run. glide get, then, becomes responsible for putting the appropriate version + version type into glide.yaml, then attempting a solve run.

The advantage of this is that we can go with autodetection most of the time, but when there's an ambiguity like in our 2.x case here, we can ask the user to clear it up then and there:

$ glide get github.com/foo/bar#2.x
[INFO] 2.x is ambiguous for github.com/foo/bar; do you want to track the "2.x" branch, or do you want the semver range ">=2.0.0 <3.0.0" [b/s]?

The user is only confronted with this message if we actually find a branch (or tag) named thusly; otherwise, it proceeds with the semver assumption.

And, for the naked get - $ glide get github.com/foo/bar - we find the name of the default branch and actually fill it in.

Obviously this doesn't cover the case where people are hand-editing the file, but as long as the rules are consistent, then it's a small cognitive leap, and the extra typing is...well, hopefully negligible. At the very least, the way it works would not be non-obvious, which I think is really the biggest risk?

The best default

As I noted in the initial issue, we could take the approach where there's a default vtype that's applied when one isn't explicitly provided. This would be handy for the most common case if you're hand-editing the file. (It does have the drawback of making the behavior here less obvious, though.)

I initially suggested that giving semver/versions the default slot. There are two reasons for this:

• Because we can effectively cover two types (semver and plain version), the default is ostensibly covering more cases
• Because the eventual hope is that the Go ecosystem picks up semver, in which case most deps most of the time will be on semversions.

"Eventual" is the key word there, though. We could also optimize for the shorter term, where people are still tracking branches a lot, and have branch be the default type.

Legacy manifests

This is probably the trickiest part. Having a default vtype actually makes this harder, because it would assign potentially erroneous meaning to existing manifests. It's also hard because we know we have to support whatever exists now in perpetuity - while we can require that the user fix an old manifest in their project, we have to be able to parse in data from old glide.yaml that may show up in dependencies.

TBH, this is its whole own discussion. The cleanest thing to do is not have any defaults, because it lets us draw a line between the new and old type of data. But that default option...well, it seems compelling to me.

We could also entertain introducing a manifest version number into glide.yaml. I don't really like that either, though.

[sdboyer]

Oh, another alternative - instead of always using the version field, we could just name the fields according to the version type:

import:
- package: github.com/foo/bar
  version: ~1.0.0
- package: github.com/masterminds/semver
  branch: 2.x

[mattfarina]

@sdboyer Thanks for going so deep on this.

First, User ease. I have already been planning a bunch of work here. We can do a lot better when it comes to this. I'm glad to see you're thinking about something similar. I have plans to even go beyond this but that's a conversation for another time.

Second, I'm very wary of adding another config field no matter how we try to ease it. A configuration option for every possibility isn't the path to take. That ends up in user confusion and pain. Yet, you do need to have just enough. And we can have automation try to fill in the rest. The goal is to target the 80% and provide sane default solutions (often not exposing them).

If we have to add a configuration option for this we'll have failed our users. I don't want to do that. So, let's exhaust all other options for automation or flexibility.

So, let's look into other options.

1. Start with modifying the glide.lock file. This is auto-generated and managed by Glide. What if a reference or branch property were added to an import. If the version is a tag, branch, or other reference it's set to true. Then, if the property is not present or set to false, the glide.yaml has not changed, and a branch comes in we can detect it.
2. When Glide adds something to the glide.yaml like 2.x and there is no reference for that it makes it ^2x. For 2.0.x it makes it ~2.0.x. If someone manually edits the glide.yaml we tell them if they've done something like 2.x and the possible problem if it's not already in the glide.yaml. This will not completely remove the potential for the problem but will decrease the opportunity for it.

This is by no means exhaustive. But, I don't want to fail to find an option that keeps the UX simple. We need to hide the complexity.

Risk, complexity, and user experience needs to be weighed. My stance is the best user experience possible, hide as much complexity as we can, and keep the risk low. Risk can never be eliminated (if that's even possible) without making the experience painful for the long tail which isn't acceptable.

[sdboyer]

Thanks, these are helpful ideas.

Now, I'm not quite clear from your response if you're suggesting we continue relying on checking against external repositories at solve time (as opposed to get-time) to disambiguate constraints. So, to make sure we're on the same page about this, I'm just gonna reiterate the basic, cut-and-dry decision:

At solve time, either the system can figure out from pure user input what the intended constraints are, or it can't. If it can, we get:

1. Performance benefit during most solves from not having to hit network or disk to figure out what constraints mean
2. A hash in the lock file we can actually trust to avoid a bunch of work
3. By far most important: immunity from upstream changes confusing the tool (one of the circles of dependency hell)

If it can't...well...

User delight is important. Avoiding hitting people with a confusing number of concepts is important. But I'm not convinced it's worth the cost of intentionally allowing the user to get stuck in unresolvable situations. So, I need to be clear on where you're coming from - are you saying:

• That you are wiling to sacrifice those three things for this aspect of UX? or...
• That you aren't willing to sacrifice those three things, but you also don't want the extra fields, so you're looking for another way?

Your feeling on this really informs the rest of my answer. I'm gonna mostly assume your answer is the former - sorry in advance if I misunderstood.

Start with modifying the glide.lock file. This is auto-generated and managed by Glide. What if a reference or branch property were added to an import. If the version is a tag, branch, or other reference it's set to true. Then, if the property is not present or set to false, the glide.yaml has not changed, and a branch comes in we can detect it.

This is already going to be happening - lock data emitted by vsolver always contain a rev, plus a version if one's available. Sadly, nothing done in the lock can help resolve constraint ambiguity; constraints come solely from the manifest.

When Glide adds something to the glide.yaml like 2.x and there is no reference for that it makes it ^2x. For 2.0.x it makes it ~2.0.x.

Sure, that could take care of some of it. But it's still just tiptoeing around the core problem. Someone could just as easily make a ~2.x branch. We have no registry that we can use to constrain the naming of branches and tags.

Now, if what you're saying here is that you'd consider a branch or tag literally named ~2.x to be something on which it's not possible to depend in glide (on the assumption that it'll be rare), then IMO, that's a more reasonable approach. As I previously mentioned, we're trying to encode more information in a string than I think is wise, but it's not unprecedented (composer).

If someone manually edits the glide.yaml we tell them if they've done something like 2.x and the possible problem if it's not already in the glide.yaml.

I assume the latter is supposed to be glide.lock? In truth, this just makes things worse. We'd have added an entire new dimension to an already very large possibility space: the lock is effectively providing constraints, thus becoming an input to itself (in a way quantitatively different from how it is right now). Solver outcomes become potentially significantly different depending on whether you have a lock or not. And, if you rm a lock file, it is impossible for even any theoretical solver to recreate that same lock file again. Or, in practical terms: people would end up treating their lock files the same as their manifests, because they'd learn pretty quickly that the solver can only reliably complete if the original lock file is there.

If we're already detecting that there's a problem, why allow an operation to continue? This is a case where soft failure injects bad data into the system in a way that poisons not only the current user, but other users, later. That's a much higher cost than hard failure.

If we have to add a configuration option for this we'll have failed our users. I don't want to do that.

We're also failing our users right now - see the nightmare case described above, or even just the problem that prompted opening this issue.

I don't particularly want to add another field, either. Especially after having done it provisionally in #384. It's...gross. At this particular moment, I'm more a fan of actually adding the four fields - branch, version, semver, revision - and then validating that only one of them can be present per dep. To that end...

But, I don't want to fail to find an option that keeps the UX simple. We need to hide the complexity.

There's a lot of complexity in this domain to hide, to be sure. But I actually believe this bit is reasonable to ask people to think about: do you want to depend on a branch, a rev, a plain version, or a semver [range]? In my article, I even had my little visual metaphors for this:

This isn't one of those bottomlessly complex problems where new layers just keep on appearing. The choice of constraint type for a dep is meaningful, has easy metaphors, and the basic guidelines can be taught in a few minutes. People can casually disagree about it over beers. It's essential, not incidental, complexity. And, as I pointed out wrt glide get in a previous comment, it's still something where we can mostly automate it away.

Risk can never be eliminated (if that's even possible) without making the experience painful for the long tail which isn't acceptable.

Of course, there'll always be risk. Just like there'll always be bugs. But arguing that we shouldn't try to close off this entire class of bugs because "there will always be risk" is like arguing against seatbelts because car accidents will still happen. This is a single, specific problem, with a single, specific architectural requirement: making pure user input sufficient to determine the meaning of constraints.

We're very close to having this, and it's the kind of thing that could cost us literally months or years of work later (if it's fixable at all); when the ecosystem becomes more complex and interconnected, a popular project deep in the dep chain somewhere doing something wrong in this regard could have immediate, viral ripple effects. Our very own left-pad, but worse.

[sdboyer]

I realize, I should clarify one thing - we could take an approach where glide hits the network prior to calling Solve() (or SolveOpts.HashInputs()) in order to disambiguate any constraints in the glide.yaml. We'd have to do the same from within the analyzer, as well.

This would solve the useless input hash problem, at least...though it wouldn't insulate users from much craziness, and it would (obviously) make any action around the manifest -> lock step pretty slow, too.

also, could use @technosophos's perspective on this

[sdboyer]

just noticed that cargo, when it's interacting directly with git repositories rather than using crates.io, uses something similar to the multiple named-field approach:

You can specify the source of a dependency in a few ways:

• git = "<git-url>" refers to a git repository with a Cargo.toml inside it (not necessarily at the root). The rev, tag, and branch options are also recognized to use something other than the master branch.

[sdboyer]

OK, I've reflected on this since our debate on Tuesday, and I've come up with some things. To help, I looked through six analogous systems for comparison: cargo, bundler/gems, mix (erlang/elixir), pub (dart), npm, and composer.

• Cargo strictly differentiates between source types - there are packages from crates.io, and there's code from git (and only git) repositories. When you use their equivalent of a version field, it's restricted to crates. If you want to use git, then you're given the branch, tag, rev choice (all of which are done via exact string matching). This means there's zero potential for ambiguity between a semver range - which can only apply to packages from crates - and a literal branch or tag name, which can only come from a raw repository.
• Bundler is basically the same as cargo (unsurprising, given the shared authorship). The main difference is that it doesn't enforce semver as crates does, presumably because rubygems predates semver. There may be a narrow possible overlapping space there - if your constraint is 2.x, and a 2.x version exists (I can't tell if it implicitly inserts = or not) - but even in that case, you're still guaranteed to be using a tag, not a branch. I'd also wager a guess that that's the sort of uncomfortable thing that they'd rather NOT have to deal with...judging from the fact that crates has cut out that possibility.
• Mix also strictly differentiates between packages sourced from the central registry (Hex) and things sourced directly from git/github. The rules here are pretty much the same - Hex enforces semver on all published packages, which removes the 2.x ambiguity entirely, because it's not possible for that literal version string to be published.
• Pub is also basically the same as mix and cargo: repository and registry source types are strictly separated. Only git is supported on the raw repo side. Semver is required to publish...I believe. (I can't find 100% confirmation of that in the docs, but if it's not, then the worst case is what I described above for rubygems).
• npm is, by now, a familiar refrain. Strict separation between registry and repository source types, with only git (plus github special sauce) supported. Semver is enforced in the registry. Revs, branches, and tags are smooshed together when specifying from git[hub], but they're all as literal matches - no semver.
• Composer behaves the least like the others, and the most like glide does now (though still, notably differently). There's just the one version field, and they compress a lot of information into it. But the dev- prefix they require (enforced by packagist) when constraining to a branch inherently resolves the potential for mistaking a semver range for a branch - the original case here. They do allow forms of aliasing which would allow a semver constraint to incorporate a branch, though that's to support a specific case.

Looking through all this has helped me to see the real kernel of the problem here. Hopefully, it can help focus discussion.

The base issue is that we have several different namespaces for different types of versions, all of which potentially overlap. That overlap entails that, if all we have is a "version" string from the user, we may not know which type of version they actually intended. Autodetection can tell us what's available, but that's not the same as user intent. Loosely, the three types are what I identified in my article with the three images I dropped in earlier:

• Branches - or, 'floating' versions that we expect to change over time (indeed, that's the express reason to rely on them instead of tags).
• Tags - semver mostly, though not necessarily; the expectation is that these mostly won't ever change (though it is possible)
• Revisions - immutable identifiers. The only reason to specify a constraint this way is if there is no associated tag, meaning that you've basically gone in and plucked out that revision yourself.

All six of the other tools have rules to clearly disambiguate how user input corresponds to these version types. User input alone is sufficient to determine exactly one version type to allow (with only one, specific, carefully thought-out exception). That all six tools adhere hew to this line suggests how important it is that the user be in clear and complete control over whether they're relying on a branch or a version.

Some of these disambiguation rules rely on explicitness from the user - e.g., the branch, rev, tag fields - while others arise from limits on the types of versions offered by a source - e.g., a tool reliant on a registry that offers only semver (crates, pub, hex, npm) can safely enforce that all user input must be a valid semver constraint. For those which allow semver and versions (rubygems), if the input isn't a valid semver constraint, then it can be safely assumed to be a literal match on a plain version (which is still of the same general type).

Composer is an outlier in that it allows slightly more overlap between the branch namespace and the version namespace. (Packagist allows semver tags OR branches). This is all in service of its goal of allowing dependees to specify a constraint that admits EITHER a tag series (e.g. 0.6.x) OR a branch from which the author creates tags in that series. This overlapping can either be done with explicit aliasing (as described in that link), or by just naming your branch e.g. 0.6.x (which packagist represents as 0.6.x-dev). However, if you were to push a branch named 0.6.8, packagist would convert that to 0.6.8.x-dev. Even literally specifying 0.6.8 will not match the branch. So there is overlap, but it's narrow and specific, not the default case.

(In fact, the way that composer achieves this is to classify branches as existing in semver's specified prerelease space, which composer classifies using its 'stability' system, then allow users to select stability levels. In other words, they've formally defined a semver relationship between branches and tags, which means the namespaces are no longer 'overlapping,' but 'intersecting'.)

Now, reading through these various approaches has made me think that I may need to make some changes to vsolver's version handling. I'd assumed a strict separation between the four types (branch, plain tag, semver tag, revision), but it may be better to not be quite so strict. I'm not sure how I feel about composer's approach - if allowing that particular branch-or-tag case is really worth the added cognitive and logical overhead - but if it is, I'll need to change it up.

That said, there is still a major problem here. Every single other system I've reviewed takes extensive steps to ensure that branch and tag namespaces don't get mixed up, and many take even more steps than that. Composer is the only real exception, and does so in a very specific limited circumstance which is still visibly obvious to anyone scanning a composer.json - and that includes the tooling.

glide's current approach does not rise to this standard. It does not go the full distance of disambiguating semver/plain tags, branches, and revs, nor the lesser disambiguation of branch and tag.

Now, having spent some more time reading glide's current code related to this, it makes more sense to me, and I understand better why the current approach has worked well so far. That's great. If I've been disparaging of that, I'm sorry. But we have a problem to solve that is qualitatively harder than every one of those other tools: there's no registry that can guarantee only semver to match against, and we have more vcs types to support than any other tool. glide has to rise to at least the level of rigor that the rest of the tools do on this front.

This could, probably, be done by keeping the single version field and encoding additional information into the version value. (I guess?) It could also be done with the typed fields I already described. Either way, the information has to be encoded somewhere in order to remove an ambiguity that six other major systems do not allow. If it's crucial that we allow that branch-or-semver use pattern that composer admits, OK. But that can, and so should, probably be its own discussion.

Now, given that these six other solidly successful tools have encoded this same information and everything's been OK, I don't think it should be a stretch for glide. However, here are some points directly on the UX front:

• One thing that good UX surely does is hide away unnecessary complexity, particularly for newer and less experienced users. However, it is also a tenet of good UX that such hiding should not be done at the expense of more expert users. The current approach does that by making it impossible for them to unambiguously specify their constraints.
• I say "expert" users, but it's not even really them who suffer here. In an ecosystem without a registry, the thought isn't, "I'm gonna depend on 2.x." It's "I'm gonna depend on the 2.x {branch,tag}". Asking people to provide this information isn't asking them to deal with a foreign concept - it's something they're already thinking about. There is a negligible amount of additional cognitive load here that doesn't already exist.
• If it is foreign, then it's a lesson they have to learn to use a tool like this safely. That's a reality; autodetection doesn't change it.
• This case is not analogous to the vcs field. In that case, there is only one possible right answer, so autodetection is possible. Where there is ambiguity, as in this case, unguided autodetection is not safe.
• Doing this hampers other aspects of UX. For one, I can't look at a glide.yaml and know how someone intends to rely on other projects, because that information just isn't there. Knowing if someone's chasing branches or not is a handy thing to know.
• The vast majority of the time, deps are going to be added via glide get, during which time it's perfectly fine to prompt the user to clarify their intent. There's ample opportunity there for helpful info text.
• Requiring the glide.yaml to be well-formed with these appropriately-named fields means that most anyone hand-editing it will always have examples to draw from.
• While relying on validation and error messages is sorta the UX equivalent of driving down the highway by bouncing off the guardrails, tight definitions for how these constraints have to work makes it very feasible to provide correspondingly clear, tight guidance.

Finally, let's please remember that the risks here is not to one person at a time. Ambiguous constraints mean people cannot defend against upstream craziness. That makes it an ecosystem-level threat. That is, it's not "Person A makes a mistake, and only Person A is affected." It's "Person A makes a choice, and persons B, C, D, and E's builds break, they have to take drastic steps to fix it, which in turn might even cause further breakages." (e.g., they each fork A's project and alias in the fork, making B, C, D, and E become mutually incompatible. This screws over F, who relied on the previously-compatible B and C.)

[halstead]

From an ops perspective the most important attributes of a package management system are safety and predictability. If a fresh install on a clean server could possibly end up with different source code than what my developer had intended then I need to essentially ignore the package manager and build another process to ensure reproduciblity.

It doesn't matter if it's because my developer used an ambiguous string in his or her yaml file. Or if the dev has a lock file that's been around for weeks and I'm starting fresh. Or if upstream changes cause the package manager to install something else for me then it did for the dev. Any of those cases break the tool from a systems perspective.

[technosophos]

It sounds like we're getting close to agreeing on this:

• The version: field can take a SEMVER, a TAG, or a VCS REVISION

The resolution would be something like this (in order):

1. If the version string is a VCS REVISION, use it as-is
2. If the string is a SEMVER version or range, use it as such
3. If the string is a TAG, use the VCS REVISION for that tag
4. Else emit an error (or warning)

I'm leaning away from allowing version: to contain a branch because (a) branches behave differently than the others, (b) user intent for branches is different (about tracking head on that branch, vs locking to a semantic range), and (c) the ambiguity between a branch and the other concepts forces us to solve a lot of edge cases introduced. So for my part, I'd like to see:

• A branch: introduced into Glide.yaml
• The algorithm tweaked so that "if isset(version) use version; else if isset(branch) use branch else use HEAD on master" (roughly speaking; details will be more nuanced.)
• A strategy for programmatically updating existing glide.yaml files for this new fix.

The remains a notable unsolved problem:

• If A requires B version: 1.0.0 and C requires B branch: feature/tinkertoys, what do we do to resolve?

/cc @mattfarina @sdboyer

[sdboyer]

sounds pretty good to me. couple notes:

If the string is a TAG, use the VCS REVISION for that tag

when you say "use," you mean what ends up in the lock file, right?

vsolver emits both a tag and the underlying revision in solutions; you can keep both, or discard the tag.

Else emit an error (or warning)

well...eh, timing.

maybe this just ends up being semantics, but because i favor an approach where the user's input gets turned into a well-formed constraint, with well-understood boundaries, independent of what happens to be in the repository, let me try rephrasing this in those terms and see if it still meets with your expectations:

1. If the version string conforms to a pattern that indicates it's a VCS REVISION, then for the purposes of determining compatibility with other projects' deps, it will match only if the other projects' dep is also that exact revision.
   • In git and hg, this pattern is 20 hex-encoded bytes (40 0-f chars).
   • bzr's a little more complicated, but matchable.
   • svn ???
2. ELSE, If the version string is valid semver, then interpret it as a semver constraint, which will allow it to match only semver-conformant tags that pass the semver match check.
3. ELSE, assume the version string is intended to match a literal tag name.

It would be possible (and I don't think harmful), to allow 2 and 3 to coexist - semver range, but also allow a literal tag as a fallback. That's only safe, though, if the literal match is used as a fallback - if it supercedes the semver constraint, then in effect, we're changing the semantics of semver ranges.

How does that sound?

The algorithm tweaked so that "if isset(version) use version; else if isset(branch) use branch

Sure, though I think it might make the files easier to understand if glide were to reject having both branch and version present at the same time; seems like allowing both to be present would be confusing. Doesn't really matter to me, though.

else use HEAD on master" (roughly speaking; details will be more nuanced.)

Yeah, there's a bunch more detail here. Default branch is one way to go, and the simplest one.

Just spitballing, but, there is another option: it could be treated as a total, any-type wildcard. So, it could match a branch, or a tag, or any rev. In vsolver-land, at least, the version that ultimately gets picked is determined by the order in which the available versions are tried. That would mean:

1. valid semver tags (in descending or ascending order; the tool controls that)
2. non-valid semver tags (alphasort)
3. branches (alphasort right now, though we could probably put default branch first)

Branches could also be put before non-semver tags...that might be a good change to make in general, anyway.

A strategy for programmatically updating existing glide.yaml files for this new fix.

I have a few thoughts on this, but yeah, a strategy must be had. We ALSO have to retain support for old glide.yaml files, and have a way for interpreting them on-the-fly into the new system, so that deps using the old form still more or less work.

The remains a notable unsolved problem:
If A requires B version: 1.0.0 and C requires B branch: feature/tinkertoys, what do we do to resolve?

That seems pretty straightforward to me: it's an unresolvable conflict. [re]solving fails.

Now, there's one avenue I haven't yet explored - you could try to check and see if 1.0.0 and feature/tinkertoys have the same underlying rev. If they do, then they match. Outside of that, though, this seems like a run-of-the-mill example of a case where it should fail, because the constraints are incompatible.

[sdboyer]

Well...I mean, there are other approaches. But I think they necessarily involve ignoring version constraints in some way. There's two I can readily think of:

First, you can try to just ignore the conflict: see if you can safely encapsulate B at 1.0.0 under a nested vendor dir within A, or do the same for B at feature/tinkertoys within C's nested vendor dir. I've talked about this as a possibility a fair bit, but doing it safely is far away. Plus, at least for vsolver, a pre-req will be swapping in immutable datastructures, basically so that we can rewind to arbitrary earlier failure points.

Second, there's always the static analysis approach: use static analysis to determine if C is type-compatible with B at 1.0.0, and/or if A is type-compatible with B at feature/tinkertoys. If either of those works, pick it as a solution.