Replies: 3 comments 14 replies
-
|
I’m unclear on what you’re proposing, but the ability to mutate the latest tag is critical, and must not be impeded in any way. |
Beta Was this translation helpful? Give feedback.
-
|
I don’t see why a problem with yarn should motivate a server side restriction that breaks clients that lack a resolutions feature when package authors prefer not to publish a revert. Why can’t yarn do “exactly what npm does” here? |
Beta Was this translation helpful? Give feedback.
-
I think you're missing the point. I don't see how this has anything to do with yarn or it's specific implementation. This is about semantic differences in the concepts at a higher-level, one which any and all package managers can take advantage of these changes (including npm), regardless of whatever current implementation npm has (just because npm does it one particular one now, doesn't mean doing it in that way is the best solution to the problem, so that's just not really an argument). What I inferred from reading this, is that the proposed solution would open up new capabilities for all package managers, which can only help the ecosystem as a whole, though I'm happy to be shown differently if I'm wrong. |
Beta Was this translation helpful? Give feedback.
-
|
Let me try again by describing some of my use cases more fully:
There's also a common scenario of mine where mutable @arcanis do you have a concrete proposal? Which scenarios of mine would no longer be possible under that proposal? |
Beta Was this translation helpful? Give feedback.
-
From the first line in my thread: users should be prevented from modifying [
None.
You don't set
This thread isn't about
You don't set
If you fully unpublish a version, then it's fine to update |
Beta Was this translation helpful? Give feedback.
-
|
Users should never use * anyways, since that will always result in unwanted upgrades, so I’m comfortable saying that’s on them. Package managers do not universally (including older versions that are still widely in use) have the ability to take deprecation as a signal. Additionally, they shouldn’t deprecated until the root cause is understood (in scenario 2) and at all (in scenario 1). Deprecation doesn’t mean “prefer avoiding this version” and should not be distorted to mean that - that’s what “latest” is for. You can’t unpublish any version that “latest” points to - as far as I’m aware, that has been a constraint since the creation of i publishing. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @arcanis I've shared this with this team and will follow up when I've had time to think on this a bit more. It seems like the biggest confusion is specifically when you have a semver in the range in the package.json that includes latest, but would install a newer version. In the case that folks publish a An initial thought would be semantically... do we want to be respecting the range in the package.json, or do we think "latest" should trump the "highest possible version" in the range if latest is in the range but not the highest possible version. |
Beta Was this translation helpful? Give feedback.
-
|
I think there's also a potential conceptual difference between the implicit "latest" when one does |
Beta Was this translation helpful? Give feedback.
-
|
So, to make sure I understand the request here, the idea is:
There are a number of challenges with this approach. First, it will break existing use cases (including one that npm itself uses) for providing releases on a semver-major line ahead of making that semver-major line the "default" installed copy. Second, it could make it much more challenging to release legacy versions of a package. I think this might have been addressed by making the If we do implement this in such a way as to avoid the legacy management problem, then I don't see how the semantics are any less misleading. If the core objection is that "latest" might not actually be the "latest" published, well... if my currently active release line is 3.x, and I then publish 2.2.1 to fix a bug in the legacy 2.2.0 version, then 3.2.0 is no longer the "latest" published. Either we accept this semantic wart, or I'm forced to then publish a no-change 3.2.1 just to re-claim the tag. The much easier solution to this problem would be for Yarn to adopt the same semantics for manifest resolution that npm uses, which is encapsulated in If you would still prefer not to rely on Since a solution to the problems presented is readily available for all to use, and adding this restriction to the registry would break several popular and valuable use cases, I think it is extremely unlikely we will move forward with the suggestion as presented, unless there is a significantly more compelling need to justify the disruption. That said, there is clearly work to be done evangelizing the use of |
Beta Was this translation helpful? Give feedback.
-
|
First, thanks for the comprehensive answer, I find it helpful. I however feel there's something I should clarify: in itself, I'm not worried about Yarn. We'll implement this logic and be done for the day. Still, I'm worried about what it actually means in terms of semantic for the ecosystem - just because it's not a problem for us doesn't mean it's not a problem in general. When you see a semver range (and I say this even if "semver ranges" is a bit loose, as ranges aren't technically part of the semver spec), it's fairly easy to know what to do of it. You take the version, you compare it with the range, and you know whether the version is semantically compatible with the range. It's simple, it's nice, it's synchronous, and as such many tools that depend on the However, what you say, and specifically "the version indicated by
Now, it's of course not impossible to do - that's how things usually work (do they? can you say for sure that all the aforementioned tools are even aware of the rule you shared? what would you say are the chances that Lerna has at least one bug about this when working with npm 6?). But it means that special logic has to be added in order to workaround semver, and make it work differently from expectations. I personally don't think this is the right approach.
For the sake of the debate, I find that a problematic but acceptable counter-argument, in that even if the |
Beta Was this translation helpful? Give feedback.
-
A bit afield from the OP here, so I'll reply to the other stuff in a separate comment, but you may be interested in the conversation over at semver/semver#584 The SemVer spec moves very slowly, as we all have a lot on our plates. For ranges in particular, part of the issue is that every package manager sort of implemented something similar out of necessity, but they're all subtly different. I threw out a proposal which just specifies what node-semver does to get the conversation started, but we're still kind of at the "figuring out where the cowpaths even are before we can go about paving them". My guess is that the spec will land in a place where the overlap is codified, and there's a bunch of "optional, but if implemented, must be work like this" stuff. |
Beta Was this translation helpful? Give feedback.
-
In practice, this is not the case. If your dependency is There is no rule in the SemVer specification or elsewhere that says a package manager must select the highest-precedence version number available that satisfies a dependency range. All package managers apply additional heuristics on top of naively selecting the presumably latest and greatest. (Deprecation, deduplication, known security vulnerabilities, the list goes on and on.) I'm not sure why the existence of such heuristics invalidates using the range as a valid statement of compatibility.
Not a bad idea for renovate/dependabot, if they intend to mirror the behavior of
I don't agree with this. npm is a "dep graph optimizer", and it does merge versions across latest if they're otherwise compatible. It just prioritizes the
Similar to renovate/dependabot, it might be a good idea for Snyk to prioritize the dist tagged version, but if
No, it means that special logic has to be added when selecting a manifest from a packument. Which is already the case, because it needs to be able to get the list of versions, etc. Selecting which version of a package to install will almost always take into account more than merely the available versions and dependency range, so I'm not sure I even really understand what the concern is here. It would be user-hostile to install something that is deprecated, or known vulnerable, or claims to conflict with the current platform, or can't be ideally deduplicated (especially on platforms other than node where full deduplication is a hard requirement!), if there is some way to avoid these problems. Prioritizing the version which the publisher has claimed should be the default, if it's a valid choice otherwise, strikes me as completely reasonable. It seems like you've gotten it in your head that Almost immediately, users were annoyed that they'd publish something, but it couldn't be installed until they ran a second command, so we just made Fairly shortly afterwards, users got annoyed that their dependency didn't get the "default" version when they had a dependency range that included a release that wasn't yet ready to be the default installed version. So someone would depend on So, I'm really not overstating it to say that:
Like, ok, maybe "main" or "default" would've been a better name choice, but the convention and the behavior is as established as anything else in JavaScript now, and it would be massively disruptive to change. I don't think you can credibly argue that it is "abandoning SemVer" to use heuristics other than "highest SemVer version within the range" when choosing an install target (since literally every package manager does so), and I haven't really seen a problem brought up here that can't be fixed by just helping users better understand the tools at their disposal. Since many of them do understand the behavior (at least well enough to build things that rely on it), changing it would not be good for them, and would cause an increase in confusion, even if more semantically "correct". |
Beta Was this translation helpful? Give feedback.
-
|
It seems clear you don't agree there's a problem so I'll mark this as the answer, but I disagree with your evaluation. Specifically, I think there's a conflict between what you use
Let's say I create a project
This last point is permitted because both
Consequently, the use case you and @ljharb mention (being able to use To fix that, and turn this into a proper guarantee, package manager authors would have to forego the point I quoted and make the optimization pass be aware that the
It's still unclear to me why those users don't use prereleases, whose purpose seem to be exactly this? To be clear, I'm not saying the use case doesn't exist, but I'm wondering if perhaps the way it got solved isn't to fit a square in a circular box. It works, but you see holes on the side... |
Beta Was this translation helpful? Give feedback.
-
|
To be clear, I’m always fine with ranges resolving past latest in some cases - i just don’t want the majority of cases to do so. I don’t actually want or need a guarantee. |
Beta Was this translation helpful? Give feedback.
-
tl;dr The
latesttag can currently point to non-latest releases, making it a semantical non-sense causing problems with tools attempting to give it one. Users should be prevented from modifying it, with it being fixed to whatever the latest stable release actually is at any given point in time. It wouldn't break use cases, because there are better tools to achieve what people typically want by doing this.An annoying problem we've been hitting on Yarn regarding the npm registry has been regarding the
latesttag on the registry. It's rather subtle, so first I'll explain a bit how things work, why it's problematic, some fixes, and what's the best one and why.Tags on the npm registry work very much like Git: a set of words, where one word points to a version. Just like Git tags, npm's dist-tags are mutable: it's fairly easy to change the target of a tag by using the
npm tagcommand. This capability is often used: various popular packages have anexttag pointing to the highest beta (and thus need to update the tag reference when a new beta is published), and even without that an overwhelming majority of packages make use of thelatesttag, which always point to the latest stable release from a package.Before going further, let me talk a bit about
yarn add(npm adddoesn't work exactly the same but suffers from the same problem, in different ways). When you run the command with a range (yarn add foo@bar), things are fairly simple: you tell us you want to usefooat versionbar, so that's what we put in thedependenciesfield. Easy. Now, what happens when you runyarn add foo, though? Since there is no explicit version requested, we fallback onlatest, which we resolve to whateverlatestis currently assigned on the registry, and then finally we store that in the manifest as a range (let's say^1.0.0). Fair enough, right?And now is the fun part: what happens if
latestdoesn't point to the latest stable release? That sounds a bit weird, but it's actually very possible. The npm registry doesn't currently enforce whatlatestpoints to, so you can usenpm tagto update it to an earlier release (I hear someone in the back say that's kinda like unpublishing a package, but safest, right? - let me continue and we'll come back to that later). As an example, Husky currently has the following releases (minus a few that aren't relevant to the problem at hand):We can notice a few things:
latesttag doesn't point to the highest stable (that would be5.0.4)4.3.1), which is probably a mistakenexttag doesn't point to a semver prerelease, which is probably a mistake tooThat's kind of a mess ... but so what, will you ask me? Let's go back to
yarn add. If you install Husky at the moment with Yarn (2), you'll have an funny behavior: Yarn will resolvelatestto whatever the registry says (4.3.0), turn it into a range (^4.3.0), and add that to thedependenciesfield. Then it'll run an install, and that's when things become interesting! Because when Yarn sees a range in its dependencies (like^4.3.0), it'll resolve that by asking the registry the list of all versions, keep only the compatible ones, and use whatever the highest is. Which, in this case, would be4.3.1. As a result, you get a slightly awkward behavior where the range in the manifest doesn't represent the version who got installed (but is still perfectly compatible with it).One question I can hear: is it a bug in Yarn? As I mentioned earlier, tags are words pointing to releases. In themselves, they have no bearing on the semver resolution - just because a release has a tag doesn't change anything regarding whether it is compatible or not with a range. Since
4.3.1is compatible with^4.3.0, then there's nothing preventing it from being selected... which can be surprising - we got at least two/three issues about this in the past year.So what can be done to prevent that?
Yarn could ignore all versions higher than
latestfor the purpose of semver range validation. I don't like that, because it's the kind of magical workaround that's susceptible to break in unexpected way (what if I use a range to a prerelease? What if I use*? etc)Yarn could make
yarn addsmarter by storing an exact range (4.3.0) instead of a caret range when it detects that alatestsemver range would match more than one unique version. This is undesirable because:It would break commands like
yarn upgrade-interactive, which would lose valuable information (that the user intends allows this dependency to be upgraded to future versions within the same major).It would break systems that enforce caret dependencies (or, more generally, would be confusing to many users that wouldn't understand why some dependencies are ranges and others exact).
Yarn could use a literal
latesttag in the dependencies field (instead of a semver range) when it detects that alatestcaret range would match more than one unique version. It would address the first problem, but leave the second: it would be very confusing to have one behavior or another without clear idea why.We could do what npm does, and use
^4.3.0and have some internal logic to install4.3.0during the followup install, rather than the highest compatible version. The would solve the confusion, but would leave theupgrade-interactiveproblem.We could completely ignore whatever the
latesttag points to, and letyarn addfind out what's the highest stable version available. In Husky's case however that would be5.0.4, not4.3.1.I find this latest option the most interesting one, because the only real blocker is that Husky made a mistake by publishing a prerelease (
5.0.4) as semver-stable, and thus it would be accidentally installed. But still, that's problematic! How to fix this if it happens? This bring me back to the question I put aside earlier:This scenario is the only case where it makes sense to pin
latestto not-latest. This way, people runningyarn addwon't accidentally use your accidental release. However it's also flawed: they still can upgrade to it, since the range will still match. What we need is a way to say "hey, I don't want this version to ever be considered for resolution, unless explicitly requested", so that package authors could truly "unpublish" a package, but without removing it from the registry (which could break projects who already started relying on it).Fortunately this already exists! Package versions can be deprecated, and package managers can use this heuristic to make them secondary during resolution, to prefer the non-deprecated versions when possible. Through this mean, we have a proper way to satisfy the only use case for pinning
latestto an earlier version - thus removing the only blocker that would prevent makinglatestimmutable.Beta Was this translation helpful? Give feedback.
All reactions