Website slowdown by bots

[fbastian]

In GitLab by @fbastian on Feb 12, 2018, 17:19

Suggestions from Iakov about managing bots making too many requests:

1. crawl-delay in robots.txt for yahoo & bing (if they are the problem)
2. google webmaster tools allows you to set limit for google
3. http code 429 + retry-after for nasty bots (if they are useless, just disallow /*)
4. questionable: robots.txt + disallow for certain pages
5. advanced: perhaps we discussed that already, if I were you I would use a) cache all standard elements b) save all gene pages to files (e.g. excluding header/footer). it is also possible to cache sql requests if you have enough hdd/ssd.