Can't sync anything that uses source replacement to get the sources to vendor

[gilescope]

#80 unfortunaltly prevents using source replacement to source your original crates.

One possibility is to roll back #80 , another solution is to sometimes ignore it:
#161

We could be even smarter and only ignore the source replacement if it's a local replacement.

[ChrisGreenaway]

We won't be rolling back #80 as it is a good change that's likely relied upon functionality.

I'd like to understand the use case better. Can you be more specific about what you mean by "prevents using source replacement to source your original crates"? Perhaps an example config file snippet might be useful.

[gilescope]

This sheds some light on why we have to use a source replacement to talk to creates.io: rust-lang/cargo#8174

_{Sent with GitHawk}

[ChrisGreenaway]

My initial reaction to this issue is that it is very niche requirement apparently caused by some very strange settings in your enterprise proxy. I'm surprised your proxy can even tell the difference between those two URLs. The protocol is HTTPS, so the path should not be visible to an intermediate proxy.

I'm reluctant to add more options. Indeed I would prefer to move in the direction of removing existing options if possible.

There's a further question of how this is supposed to work after the first run. Presumably you would change your config to use the local registry - that's the whole point of the local registry. If so, what happens when you want to update the local registry? You'd need to use this option to make the read from crates.io work, but with the option specified it will be trying to update the local registry from the local registry rather than from crates.io. The only way round this would be to keep editing your config file back and forth - a very unsatisfactory experience.

[gilescope]

We were planning on using this as the crate capturing tool to get all the dependencies we care about from a set of projects into one local repository thus isolating ourselves from the internet from that point forwards.

As we do win & linux we use conda as our packaging tool. I was planning to conda package the generated local repository and people can conda install it at the same time as the rustc conda package is installed.

So yes this is very much bespoke institutional rust usage rather than for single developers.

As I said I am happy to change the PR so that it only ignores the source replacement if the replacement is local to remove the need for an additional flag or we can just maintain a fork just for our usage if no one else needs that functionality. I’m easy either way.

_{Sent with GitHawk}

[ChrisGreenaway]

Isolating yourself from the Internet is, indeed, a substantial goal of this project.

But users need to be able to re-sync from the Internet when dependencies update. How does that fit with your proposal? What's the flow for a user after the point of creating that initial snapshot?

At the moment, this is looking very specific to your situation. And I suspect that your situation might better be resolved by different means: addressing your strange proxy / understanding how it can tell the difference between two HTTPS requests to the same domain.

[gilescope]

Changing the proxy isn't an option for the next few years; we do have a long term plan but it takes time to implement. (It's a long story but probably not one for the public domain).

Devs will be given a local repository and then request from the provisioning team for crates they would like to be added to it, so the syncing and the usage will be on different machines - I don't think there will be any contention on the setup. How well this will work in practice I guess we will see as I don't know of anyone who's done this kind of thing in rust before.

[gilescope]

#163 - is that more of a possibility? I feel like it's keeping to the original intention of the patch.

[ChrisGreenaway]

@gilescope - so I had a look at the PR. It looks like your plan is to only remove selected types of source rather than all sources. If so, I like the idea. And it means there are no command line options, so that's good.

I've added a couple of comments to the PR.

[ChrisGreenaway]

@gilescope - I wasn't sure if you still wanted the PR to go in. If so, is it in a state for me to look at it again?