robots.txt has not being migrated in library image and more

[kelson42]

In the original Nginx vhost:

  location /robots.txt {
    alias /var/www/library.kiwix.org/robots.txt;
  }

with robots.txt is

User-agent: *
Crawl-delay: 3

On the top it is also really important to threshold the number of request to library.kiwix.org. Our old Nginx configuration had:

limit_req_zone $binary_remote_addr zone=limit:10m rate=3r/s;

This should be migrated too

[florentk]

fixed in 533b43b

[kelson42]

@florentk I reopen this ticket as the fix somehow does not work. We have currently a load>2 on the server because of the library container. The kiwix-serve daemon delivers at high speed content, which is quite a sign the throttling does not work properly.

[florentk]

duplicate of #16