bug: Too Many Requests (HTTP error 429) Error when visiting github.com

[Yiyiyimu]

Thank you for your work! This is so helpful!

Here is the problem. When running liche on github actions, visiting github.com would return Too Many Requests (HTTP error 429), even when concurrency is set to 1.

Github Actions screenshot: https://github.com/chaos-mesh/chaos-mesh/pull/987/checks?check_run_id=1172843506#step:5:648

[MichaIng]

Same here. markdown-link-check has a retryOn429 option for this, but it is probably not the best to burst requests even more when receiving this error messages. Would be great if there was an option to have a minimum delay between requests, at best only requests to the same FQDN.

even when concurrency is set to 1.

Good to know, this would have been my attempt to workaround now 🤔.

Both, concurrency + delay in combination should cover all such issues.

EDIT: I can confirm that concurrency 1 does not solve the issue but only increases test time dramatically.

[MichaIng]

There are btw other issues that may occur due to too many or too fast repeating connections against to same host. When being behind Cloudflare DDoS protection there are a set of 52X error responses. Probably another feature to handle this and other potential similar issues is to allow ignoring a defined list of error responses, i.e. treating them as "success".

IMO it is not good to automatically repeat tests in those cases, as it might even cause the GitHub action worker IP to be banned either by Cloudflare or by a local DDoS protection/firewall or similar. Ignoring them breaks the guarantee that the commit/code is safe to merge but probably better than stressing the target even more on those kind of responses or lengthen the test enormously by adding a reasonable delay between every test, e.g. 5 seconds as a usual KeepAlive timeout, which can accumulate to a very long test time. That would be only doable when fine tuning this in combination with concurrent connections again, e.g. 32 concurrent connections against the same host (I just remember Apache2 with default 50 on Debian, quite low but okay for home servers) but then 5 or 10 seconds delay before doing the next 32 concurrent connections, something like that.