Crate should follow crates.io crawler policy

[jtgeibel]

According to the crates.io crawler policy, API requests should be limited to 1 per second. I'm seeing many occurrences of this tool sending large bursts of parallel requests. For example, the first occurrence I looked at shows 146 requests with this user agent from a single IP address within a 1 second interval!

When this occurs, a client can consume all database connections causing other requests to block and impacting overall site performance. While technically the tool should only send 1 request per second, at a minimum limiting itself to a single request in flight at a time (i.e. no parallel requests) would greatly alleviate the issue and would probably go unnoticed. Ideally, the tool would be able to read from the local cargo index cache to lookup the needed version information without querying the API.

Feel free to reach out to me here or to the team on the crates-io-team Discord channel if you have any questions.

[stiiifff]

Hi @jtgeibel, fair point, and apologies for late answer on this. I'll look into it w/ @gnunicorn, and will keep you posted.

[jtgeibel]

Hey @Xanewok, @stiiifff, and @gnunicorn. Thanks for updating this to use the index instead of the API! We've seen a few more instances of cargo-unleash users sending massive amounts of traffic and impacting other users. We're considering blocking old versions, but I see that 1.0.0-alpha.10 is still your latest release. Is it possible to do a new release soon? That way, if we have to block old requests, users will have a clear migration path to a new release that works.

[gnunicorn]

@jtgeibel sure thing. we were planning on releasing a 1.0 with semverver-support, but this might be a little longer. we will release an intermediary alpha.11, I keep you updated on that!

[gnunicorn]

@jtgeibel Glad to inform you alpha.11 is out. feel free to deprecate and soon block the older version, telling them to upgrade:
https://crates.io/crates/cargo-unleash/1.0.0-alpha.11