Cargo should reference git dependencies by commit hash instead of url

[brenzi]

cargo thinks the following dependencies are different although they are exactly the same:

[dependencies]
a = {git = "https://github.com/aaa/bbb"}
a = {git = "https://github.com/aaa/bbb.git"}
a = {git = "https://github.com/aaa///bbb"}
a = {git = "https://github.com/aaa/bbb", rev ="0123456789abcdef"}
a = {git = "https://github.com/aaa/bbb", branch = "ccc"}
a = {git = "https://github.com/aaa/bbb", tag = "ddd"}
a = {git = "https://github.com/fff/bbb", tag = "ddd"}

assuming that branch ccc HEAD as well as tag ddd points to rev = "0123456789abcdef", being the HEAD of master. Moreover, fff/bbb being a fresh fork of repository bbbby user fff

I would suggest that all these references are being treated as equal because they are.

The fact that cargo doesn't know that all these dependencies are equal causes a lot of trouble:

• two separate but identical versions of a crate in dependency tree rust#61725
• Patching dependencies does not work if it's for the same location but a different branch #5478
• having some sort of semver for git dependencies #6961

The great thing about git is that a commit hash is a unique fingerprint of a repository. It doesn't matter on which branch, tag or even in which organization the reference is located. The commit hash is the same, so is the code.

actually, just writing

[dependencies]
a = { git-rev = "0123456789abcdef" }

might not be practical, but it would be unique across the entirety of all git repositories in existence. The only problem being to retrieve the corresponding code if you don't know the url, so I'm not suggesting this syntax. I just want to make the point that - behind the scenes - cargo could very well use a git commit hash without any url, branch or tag as an identifier for a specific version of a crate.

[alexcrichton]

Thanks for the report! Unfortunately though this isn't a particularly actionable issue as-is and as-written is likely to basically be ignored for the rest of time. It's not really the most productive to off-hand propose large-scale changes to the foundations of Cargo (such as how git sources are handled). I agree there are bugs and improvements to be made with Cargo, but if you're curious to see actual change and see bugs fixed, proposing that Cargo completely rethinks how it handles git repositories is unlikely to do that. (this also ignores context for why Cargo does what it does today!)

[brenzi]

@alexcrichton: I'm aware that my suggestion goes beyond a quick fix. I may also have neglected valid reasons to do it like cargo does it today.

However: May I ask for your opinion on the approach suggested from a design point of view rather than the likelihood of it being done?

[dingelish]

Hi @alexcrichton , we are suffering from this for a period of time. Our rust-sgx-sdk has been transferred to the apache organization and renamed twice. In a foreseeable future, there'll be at least two more renaming. It's painful to modify hundreds of forked crates for each of the renaming action. We really need it.

A temporary workaround is to simulate the HTTP 302 redirection to help git find the correct repo. The result is that we can have a URL like https://sdk.sgx.rs which 302 to https://github.com/apache/incubator-mesatee-sgx. But this requires the bypass of certificate verification (sdk.sgx.rs != github.com), which cannot be disabled by the built-in libgit2. Fortunately, we find that the git cli can bypass it. So the solution is to enable this in .cargo/config:

[net]
git-fetch-with-cli = true

@brenzi 's comment is very reasonable

I'm not sure a custom domain will resolve the underlying problem. It's just a workaround. I could live with changing the global setting but I don't think this is elegant because it might change other behaviour as well as a side effect

It'll be the best if I can get some comments from you on the following questions:
(1) Do you think the hash-based deduplication is do-able in cargo? If so, could we have an estimated timeline of it?
(2) Do you think the 302 solution is appropriate in this circumstance? or is there any better way to solve it?

Thank you very much!

[alexcrichton]

I do not personally have time to prioritize and implement this at this time, but if you're willing to put in the work to design and implement this that'd be great!

[PiDelport]

Related: #7670

[richard-uk1]

It looks like you might be able to do a quick fix by resolving all the different forms of git revs in SourceIdInner::canonical_url (in src/cargo/core/source/source_id.rs). I haven't figured out if this is feasible yet.

EDIT So this solves my case (repo?rev=xxxxxx#xxxxxx == repo#xxxxxx), but not the general case. The way to solve the general case is to reference git dependencies by rev hash rather than url. It also requires some kind of concept of underspecified git dep (so that during dep resolution cargo can merge git deps if possible). Quite a lot of work: I'll think about how it might look.

[richard-uk1]

Would a possible solution be to wait until feature resolution is complete, then compare all git revisions to see if they share hash (the hash is 20 bytes long and 2 ** 160 is a very very large number). This would be n! comparisons. The questions here are: could there possibly be collisions?

The thing is, I don't think there is any other way to know if two repos share code in general. They might have completely different names, so if you can't use the Oids (hashes) then I'm stumped.

[richard-uk1]

If we didn't want to go down the route of unifying all gits based on revision hashes, we could do it for some obvious cases, for example we could identify commits based on a (location, rev) pair, where location is some sort of canonicalised repo url, and rev is the hash of the revision/commit. This might be the best safest option. If users are pulling in 2 different repos from different places, can they really expect us to dedup them?

[richard-uk1]

Note:

https://github.com/aaa/bbb.git

is already unified with

https://github.com/aaa/bbb

(https://github.com/rust-lang/cargo/blob/master/src/cargo/util/canonical_url.rs#L50)

[richard-uk1]

There are two solutions I see

1. Download repos as you are constructing the SourceId, and always resolve the given branch/tag/rev to a specific revision. Use this when comparing packages for equality during building the dependency graph.
2. Have a separate step after you've gathered all the SourceIds to add in the revs.

Both of these require downloading/accessing the git repo earlier I think. I don't think you should try to unify 2 different repositories, only revisions within a single repository. I also think the https://github.com/aaa///bbb case is pathological and we shouldn't try to support it.

What are others' thoughts? And if you like the idea can you point me in the direction of how to implement it, because I've been looking at the codebase and I can't figure out how the resolution bit works.

[graydon]

See also #10256

[epage]

#6921 is almost a duplicate of this. The slight difference is that this includes unifying two dependency specs to the same dependency, even if tag vs branch was used. #6921 is focusing on the user problem of wanting to keep git dependency specs in-sync with each other.