Proposal: Replace POST requests with GET to avoid interference from Anubis Bot detection

[LGA1150]

elixir.bootlin.com is protected by Anubis Bot detection, which can occasionally be triggered during a keyword search. Currently, the search request is sent using the HTTP POST method. If the bot protection is activated, the verification and redirect process may result in the loss of the original POST data, meaning the keyword is discarded and the search must be re-entered manually.

Since the search operation does not modify server state, it should be performed using the GET method instead of POST. This would allow the query string to be preserved through redirects and improve compatibility with bot protection mechanisms like Anubis.

[tleb]

Wouldn't the root issue be that Anubis might trigger in the middle of a session?

Session here in the sense of a group of requests that are close together (a few minutes), all sent by the same user-agent. It sounds like a bad idea to check any request that isn't the first one. It has happened to me a few times and it always looks bad.

Ping @fstachura that knows more about Anubis. Any insight? We can internally ask our sysadmin that deployed Anubis as well.

---

Maybe we should address both the POST->GET topic and the session logic of Anubis?

[fstachura]

I think switching search from POST to GET is a good idea, search done as a GET either way, it's just that the form does a POST and the server forms the POST contents into a redirect URL.

We could also increase cookie expiration time from default week to something longer. Maybe a month?
I also think it would make sense to exempt API, autocomplete and static endpoints. AFAIK bots do not hit API or autocomplete by themselves and Anubis triggering on those endpoints is a bug.

Wouldn't the root issue be that Anubis might trigger in the middle of a session?

I don't think it ever triggers in a middle of a session unless you are unlucky enough to have the cookie expire while you are using Anubis. I think it's more likely that on your first request Elixir was loaded from your cache after the cookie expired, and anything you do will hit the server and trigger Anubis.

[tleb]

I think it's more likely that on your first request Elixir was loaded from your cache after the cookie expired

It sounds like a reasonable explanation, but if I have my devtools open (with caching enabled), I never see the root HTML request NOT be served; it always does this request (and all others are cached).