crates.io: Crate Deletions

[Turbo87]

Rendered

[epage]

I feel like this will replace yank in a lot of situations and feel like more friction or guard rails are needed.

[ehuss]

This does seem worrisome to me. I think depending on a deleted crate would be very disruptive for those users. I'm uncertain of the exact behavior, but I think users of those crates will either get a confusing download error (if using Cargo.lock), or a confusing resolution error (if not using Cargo.lock).

I think it would be helpful to make sure the UI has loud and explicit warnings about the consequences of this.

Would it be possible to also warn owners of reverse dependencies when this happens?

Would it be possible to include a marker in the index similar to yanked so that cargo can provide a better error message?

[Monadic-Cat]

Would it be possible to also warn owners of reverse dependencies when this happens?

Well, there aren't any owners of reverse dependencies, because "there are no reverse dependencies" is a requirement for deletion in this RFC.

I do think it's a good idea to include a marker in the index to provide a better error message for people who depend on the deleted crate in things which are not published to crates.io.

[ehuss]

Well, there aren't any owners of reverse dependencies, because "there are no reverse dependencies" is a requirement for deletion in this RFC.

I'm responding specifically to the version deletion, which does allow reverse dependencies. IIUC, the "no reverse dependencies" is for entire crate deletions only.

[Turbo87]

I think it would be helpful to make sure the UI has loud and explicit warnings about the consequences of this.

I agree. This is certainly not an action that should be taken lightly. If implemented as a button on the website it should have a confirmation step that highlights the consequences of this action. If implemented as a cargo plugin or in cargo itself a similar confirmation step will probably make sense.

Would it be possible to also warn owners of reverse dependencies when this happens?

• for crates that are deleted within 72 hours of being created: yes, that could be possible
• for crates that are deleted after 72 hours: in this case deletion is not possible if reverse deps exist
• for versions that are deleted: since most reverse deps are using implicit ^ version requirements the reverse dependency would fall back to the previous version, so I'm not sure if a warning is necessary in this case

Would it be possible to include a marker in the index similar to yanked so that cargo can provide a better error message?

generally, yes. when I thought about this question yesterday I thought a { "version": "1.2.3", "deleted": true } would be sufficient, but I have a feeling that older cargo versions won't like such an index record particularly much.

[Turbo87]

hmm, so if I understand correctly a v: 3 with deleted: true isn't really helping if old cargo would essentially ignore the v and deleted fields 🤔

I assume if we only included the v, version and deleted fields then old cargo would consider the whole index file invalid? or would it only consider these records invalid?

anyway, I'm open to adjusting the index to add the deleted information in some way if it helps cargo to display better errors and if we can figure out a way to make it work reasonably well with older cargo versions :)

[ehuss]

I assume if we only included the v, version and deleted fields then old cargo would consider the whole index file invalid? or would it only consider these records invalid?

No, cargo parses line-by-line, so it would just skip over deleted versions.

I'm not personally too concerned about older cargos. I'm more concerned about current cargo's ability to display a reasonable message.

For the record, I still think this 72-hour policy is dangerous and am concerned about the consequences.

[ehuss]

@Turbo87 I noticed that this conversation was marked as resolved, but I didn't see a response or corresponding update to the RFC to resolve it. Would it be possible to at least add the index considerations to the unresolved section?

[Turbo87]

Would it be possible to at least add the index considerations to the unresolved section?

sure, I'll add that :)

[epage]

1. This was marked resolved without any commits being pushed
2. That still leaves "this is too lax". Ideally this risk would be covered in the Drawbacks ("if we make the interface for deletion too easy and the metrics too light, people might over-rely on it rather than yanking") or at minimum left Unresolved. I would very much want other crates.io reviewers to see this and weigh this when approving the RFC.

[bend-n]

Your proposed motivations include:

• you want to rename a crate

Alright, so i have my crate, that is used by people who are not me, and i want to rename it, sounds great!

But then:

following conditions are met:

• the crate has no reverse dependencies

How does this help me rename my crate that has users?

[kornelski]

Maybe require all crate versions to be yanked for at least $long_time.

[joshtriplett]

Suggested change

	* The version has been published for less than 72 hours.
	* The version has been published for less than 72 hours.
	* There are no published crates that depend on the version (with a version requirement not satisfied by any other published version).

Otherwise, this may produce the result that people are hesitant to count on new versions even of well-established crates.

[Nemo157]

I don't know that only looking at published version requirements is enough for version deletion. A new crate with no reverse dependencies is unlikely to be picked up and used from git repositories within its first 72 hours. A new version of a widely used crate is likely to have dozens of repos updating to it within a day via dependabot or similar mechanisms.

[Turbo87]

I'm not sure about the dependencies requirement. I see two scenarios:

• crate A is used as a dependency of another library crate B published on crates.io: in this case the dependency declaration doesn't need to be bumped to always match the latest version of crate B since the implicit ^ version requirement makes cargo always choose the latest available version and it makes it possible to for downstream projects to choose which version to use (e.g. to not yet upgrade to https://github.com/servo/rust-url/releases/tag/v2.5.1 as a recent example 😅).

• crate A is used as a dev-dependency of another library, or as a proper dependency in some application: in this case you usually want to track the latest version and potentially even pin that version if you're using services like renovatebot. such services can already deal with deletions on registries like npm so I would assume they could also handle it for crates.io. worst case: you try to compile a project and cargo greets to with a "can not find matching version" error. I expect these cases to be pretty rare though, and I assume in most cases the crate author is likely to publish a fixed version afterwards anyway.

It is also worth keeping the technical complexity of such a check in mind. We would have to iterate through all dependency records in our database for this crate and run them against all of the versions of the crate. I'm not sure how viable that would be.

I'm open to a downloads requirement though, but as @Nemo157 pointed out, services like renovatebot will likely cause a large number of downloads immediately after release for the more popular packages, which will prevent them from ever deleting a version.

[ehuss]

the dependency declaration doesn't need to be bumped to always match the latest version of crate B since the implicit ^ version requirement makes cargo always choose the latest available version and it makes it possible to for downstream projects to choose which version to use

I wouldn't agree with this assumption. Some projects immediately bump the dependency declaration. The reason for doing that is that without testing minimal versions, there's no way to know if you are depending on something that was updated in a newer version. The only safe path is to also update Cargo.toml to the latest version.

[epage]

In a recent Cargo team meeting, we were discussing whether we should abandon -Zminimal-version and instead be more proactively moving version requirements up, which would hit this case more often.

See https://internals.rust-lang.org/t/zminimal-versions-cargo-update-and-cargo-upgrade/21335

[Rustin170506]

In my opinion, if we start with a more strict policy, it will be easier to relax it later. However, if we just publish this policy, it might be difficult to add additional requirements later on.

I'm curious to know if I used that version (= 1.2.3) and released my crate in a very short time. what should I do after this particular version of my dependency crate is deleted and what happens when cargo resolves my dependency? What happens to packages that depend on my crate? If this happens, do I need to remove my crate as well?

[Turbo87]

what should I do after this particular version of my dependency crate is deleted

if you're the author of a crate depending on a deleted version then you should probably widen the dependency to a previous version of that crate and release a new version with that widened dependency declaration.

do I need to remove my crate as well?

I don't think so. Removing/Deleting crates should be reserved for critical situations like the detection of malicious code or if you unintentionally published something that wasn't intended to be public. You can publish a new version of your crate and if you want you can yank the version of your crate that depends on the deleted dependency version.

what happens when cargo resolves my dependency?

as far as I remember cargo will show an error that it can't find a matching version for the dependency

[epage]

if you're the author of a crate depending on a deleted version then you should probably widen the dependency to a previous version of that crate and release a new version with that widened dependency declaration.

This comes across as a pretty glib stance towards breaking people and requiring ecosystem churn because of it. "Re-release" is no always a trivial answer.

I don't think so. Removing/Deleting crates should be reserved for critical situations like the detection of malicious code or if you unintentionally published something that wasn't intended to be public.

That implies that you are wanting a specific set of behaviors, or culture, around removing packages but this RFC does nothing to establish this culture. For ideas on how to instill such a culture, see #3660 (comment)

[Nemo157]

I'm not sure what this impossibility is referencing? If you publish a 2.0.0, realize you left an API token or something in it and delete it, then you can still publish a 2.0.1 as a new major version. Blocking a single version number indefinitely seems fine to me as there's ~infinite mostly equivalent numbers to use instead (the one case where there isn't is if you were publishing with gaps in your patch numbers and going back to fill in a missing patch number, but I've never seen that done with semver).

[joshtriplett]

I agree. And there's a lot of advantage in saying "this old version might stop existing, but it'll never show up again with a different hash". (Also keep in mind that people might have cached local builds with the old version.)

[programmerjake]

one place where i could see that running into problems is where you're trying to track exact versions of some 3rd-party library, where if they publish a particular version, you'd want to publish the exact same version number even if you mistakenly published something with that version number before

[Turbo87]

@Nemo157 I personally agree, but I've seen enough cases of "the marketing team" pushing for "the big 2.0.0 release" and if that version number has already been burned it becomes a problem. But that's why it's in the "Unresolved questions" section, I'm open to input on what the community prefers. I tend to follow what other successful projects/communities/registries are doing unless there are good reasons against it, and since npm blocks deleted version numbers I would personally vote for that option too.

[Turbo87]

How does this help me rename my crate that has users?

it doesn't. this will only be helpful for renames very early in a project's lifespan. for example: you discovered after two days that you wanted to use hyphens instead of underscores or vice versa. this won't help e.g. tokio rename itself to kyoto

Maybe require all crate versions to be yanked for at least $long_time.

This would likely have consequences for the 72 hour rule. npm allows deletion without prior deprecation (which I treat roughly similar to our yanking). if you made a publishing mistake, such a rule would probably prevent people from being able to quickly fix their mistake.

[clarfonthey]

I'm mostly in favour of this change, but with one caveat: I think that there should be an explicit mention that, while deleting a crate or version will immediately remove it from public downloads, crates.io should reserve the right to retain the deleted version for some amount of time (maybe a few days after deletion) just as a safeguard from abuse.

72 hours is a pretty long time, and just thinking of how this feature could be abused, you could upload malicious content for that interval, then "get rid of the evidence" pretty quickly. Effectively, I think that it should be possible for someone to notice that abuse occurred, report it, and then have someone review the deleted content before it's immediately sent to the shadow realm. Doesn't have to be long, but it feels like a safeguard that should exist if this feature is added, and it should be one that's clarified does exist.

---

Additionally, I think that the current requirements beyond the 72-hour window are a bit strict, and would exclude some crates that I personally think might be worth including under this policy. For example, take this incredibly old crate: https://crates.io/crates/bow

I am the owner of this crate, but it relied on the long-fixed incoherent_fundamental_impls soundness hole, and is now broken permanently on all versions of Rust since… a while. Six years ago, I published an explicitly empty version to the crate including a message about how it's broken, and back before I really understood everything, I didn't yank the old, broken versions. In fact, I should probably go and do that now.

However, through a series of also-broken dependencies by me that I also haven't checked in years, is part of the swc-hosts crate, which is downloaded a handful of times a day. In fact, it was downloaded 54 times on Jun 11, so, my guess is that someone has it running in their CI somewhere, and this itself is massively inflating the number of downloads.

Now, as I said, I'm going to probably take a bit within the next few days to go and make sure all of these crates have a blank latest version that does nothing, and their past versions are all yanked. But doing this now, after 6 years, means I'm basically never going to cover that 100-download-per-month requirement unless I have another several years where I have no downloads at all. (Which will happen, but, yeah.)

My proposal for this is to modify the rules slightly: as an alternative to the 100 downloads requirement, allow crates whose latest version satisfies the 100 downloads requirement but whose versions which do not satisfy that requirement are over a year old and have never been downloaded more than 100 times a day. This allows those pesky CI outliers to be weeded out while still effectively meaning that nobody uses the crate.

[maia-s]

100 downloads is too low. I'm an unknown crate author and I've released a few crates with no dependents and no advertising of the crates at all. They've all easily reached 100 downloads in the first one or two days. Bots keep downloading old versions of crates too, so each new version essentially acts as a multiplier for bot downloads. Also, releasing multiple new versions of a crate in quick succession (e.g. to fix semver breaks) seems to exponentially increase the number of bot downloads. One of my crates got a thousand bot downloads in a day and currently has 3700 downloads, more than twice the downloads of the crate it depends on. To my knowledge nobody uses it yet.

[Turbo87]

@clarfonthey as far as I know (part of) our backend storage already keep deleted files around for purposes such as this. deletions already exist, but at the moment they are restricted to admins.

@maia-s thanks for the feedback. as the RFC text mentions, these thresholds are not set in stone. if we notice that the thresholds are not sufficient we can always tweak the numbers.

It looks like so far no major blockers to such a change have been identified. I appreciate the feedback so far and the crates.io team will take it into consideration when deciding on this RFC :)

Over the past days, our support email address has received a significant number of deletion requests again from users that no longer think their crates are useful and would like to release those names to the public again. This currently leads to quite a bit of manual work for us, even if it is just responding with a rejection message.

To move this RFC one step further forward I propose to move it into FCP status, which will still be gated on reviews and support from the rest of the crates.io team for now:

@rfcbot fcp merge

[rfcbot]

Team member @Turbo87 has proposed to merge this. The next step is review by the rest of the tagged team members:

• @JohnTitor
• @LawnGnome
• @Rustin170506
• @Turbo87
• @carols10cents
• @eth3lbert
• @jtgeibel
• @mdtro

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[epage]

@kornelski

Maybe require all crate versions to be yanked for at least $long_time.

@Turbo87

This would likely have consequences for the 72 hour rule. npm allows deletion without prior deprecation (which I treat roughly similar to our yanking). if you made a publishing mistake, such a rule would probably prevent people from being able to quickly fix their mistake.

I was thinking more on my culture concern in #3660 (comment). The friction (and hidden nature of) deletion has helped to encourage a culture of yanking, rather than deletion. At least for myself, I've never considered the option of reaching out to the crates.io team to ask for a deletion.

I worry that offering deletion front-and-center on crates.io, or especially on the cargo command line, could end up shifting that culture in a negative direction. Looking back to when I was a new or intermediate community member, i could have seen myself clicking a Delete button, seeing that it was available.

I think requiring yanking (which is much more dramatic than the term deprecation, I can't speak to effects within npm) on the backend (without an elapsed time) would help ensure people see yanking as the primary / default action and would add a little bit of extra friction to discourage over use of deletion.

(didn't put this in that comment thread as I think this requirement is independent from adding something to the requirements for deleting versions)

[Turbo87]

@JohnTitor @LawnGnome @Rustin170506 @jtgeibel ICYMI #3660 (comment) 😉

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[Rustin170506]

Sorry for the delayed response. Just some questions I don't understand.

[Rustin170506]

Does this mean that even if there is a crate that depends on this crate, we still allow users to delete it within 72 hours?

[Rustin170506]

This seems different from the npm policy. Not sure if this is going to lead us to the pypi situation.(At least we have time limitation 😆)

[Turbo87]

Does this mean that even if there is a crate that depends on this crate, we still allow users to delete it within 72 hours?

yes

This seems different from the npm policy.

I admit that I don't remember why I didn't include the restriction in this case 😅

I don't think it is a case that will come up particularly often though. If this is a new crate (< 72 hours) then it usually won't have the popularity yet to get a lot of reverse dependencies in that time frame. And if the popularity happens to exist because of the popularity of the author then that author will most likely be aware of the consequences of deleting a popular new crate.

As I mentioned before in other comments, we can always adjust the rules if we notice that something isn't quite working as intended :)

[Rustin170506]

In my opinion, if we start with a more strict policy, it will be easier to relax it later. However, if we just publish this policy, it might be difficult to add additional requirements later on.

I'm curious to know if I used that version (= 1.2.3) and released my crate in a very short time. what should I do after this particular version of my dependency crate is deleted and what happens when cargo resolves my dependency? What happens to packages that depend on my crate? If this happens, do I need to remove my crate as well?

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[Mark-Simulacrum]

Are we expecting to at least clearly flag in the UI or index that the crate existed under some other author / version series? I could see this being very confusing when looking at e.g. crater results or other cases where we try to pull old crates.

[epage]

A marker in the index is one of several topics discussed at #3660 (comment)

[Mark-Simulacrum]

Right, I saw that, but I guess I interpreted that as mostly being when the version isn't reused - it's not obvious (lacking dates?) that we have a mechanism for identifying a crate after this RFC other than by hash of it's contents, since the same name and version can refer to ~infinitely many different hashes afaict.

[joshtriplett]

Linking to a planned policy change in a thread comment, so that it's easier to find: #3660 (comment)

[Turbo87]

planned policy change

• see crates.io: Remove version deletions from the "crate deletions" RFC #3731

[Turbo87]

FYI the revised RFC is now fully implemented. see rust-lang/crates.io#9352 for more details.