Publish on crates.io?

[mre]

In lycheeverse/lychee#733 we've been discussing whether reqwest-impersonate could be a solution to getting blocked by various websites. After some testing we found that indeed we would be able to circumvent some of the issues thanks to this project.
The plan was to integrate reqwest-impersonate into lychee as an optional feedback and a fallback to reqwest for hard cases where the server does any browser detection.
However to achive this, we would need a publicly available crate. I saw that this project applied some patches to upstream crates and I wonder if you could publish these dependencies as well as reqwest-impersonate on crates.io.
It's totally understandable if you rather would not want to do this for maintainability reasons. In such case we'd have to find an alternative solution for lychee.

Thanks for this crate in any case.

[4JX]

I'm willing to upload the crates to crates.io, but there would still be an unresolved issues if you want to publish lychee itself later on.

reqwest-impersonate as it stands depends on the functionality of the patch section in the Cargo.toml to work somewhat frictionlessly with existing projects, but theres a really big catch to this system.

The CLI section of your project will probably work just fine, but if lychee is used as a library, the author working with it will also need to include the section in their own Cargo.toml. (At least from testing it doesn't seem like the section does anything if its defined in a dependency).

Therefore you would either need to

• Work with patches yourself and instruct any users of your library to add the corresponding section (which I'm not sure you'd get to upload to crates.io in the first place, since that method is not meant as a long term solution afaik)
• Fork all the dependencies of your project that depend on any of reqwest, hyper, h2 and mantain separate versions with the replaced dependency

I've tried to start upstreaming some of the changes to make this less of a chore, but progress on that end is rather slow.

[mre]

Thanks for the thorough answer. That's a very fair and balanced perspective.

My hope was, that we could publish the library if we put reqwest-impersonate behind a feature flag, which is disabled by default, but I don't think crates.io would allow us to do so.

If that's not a possibility, that leaves us with the options you mention, which, to be honest, both don't very enticing.

Wouldn't it be possible, to release new versions of the patched crates? This way, downstream crates like lychee wouldn't have to inherit the patches. You could make this more convenient by setting up some automation around the forks. Maybe if you don't make too many future changes, it might be okay to rebase on top of the upstream crates?

I'm pretty sure you already considered that option. Of course, it would be understandable if you don't want to put this burden upon yourself (with little upside). What's your opinion on this?

[4JX]

I can upload the patched versions of the crates that make reqwest-impersonate work (namely h2,hyper and the SSL related ones), but that alone wont be optimal. I originally decided to leave it as-is via the patch section for the following reason:

For every patched crate that you pull you introduce an incompatibility with every other crate that is expected to work with the unpatched version. Even if they both have some struct Foo defined by identical code and haven't been touched to make reqwest-impersonate work, rust will still recognize them as completely different structs and will not let you accept a Foo coming from reqwest-impersonate into the method of the other crate.

This in turn means that, unless the fields of said struct are exposed or you're able to grab and copy the information in another way, this other crate will also need to be patched to accept the Foo from reqwest-impersonate.

And now the cycle continues, since crate X has been patched to accept data coming from reqwest-impersonate, every struct/enum from X will now not be accepted by crate Y, which was expecting one from the unmodified version.

This is of course the worst case scenario, but the only proper long-term solution is to push for upstreaming all the changes (at least for hyper and h2 as a starting point).

[mre]

I see. That's very reasonable. I honestly don't know of a good strategy forward, which wouldn't entail a high maintenance burden. Even so, upstreaming the changes is helpful in and of itself. 👍