Bad Referrers Blocked: 6836
Bad User-Agents (Bots) Blocked: 578
Fake Googlebots Blocked: 217
- Created by: https://github.com/mitchellkrogza for use on Nginx Web Server https://www.nginx.com/
- Copyright Mitchell Krog mitchellkrog@gmail.com
Not Using Nginx? See the
It is important to please subscribe your email address to the mailing list or simply send a blank email here to subscribe.
Please make sure you are subscribed to notifications to be notified when the blocker is updated and also to be notified when any important or mission critical (potentially breaking) changes take place.
If you miss one step you will get an nginx EMERG ❗ error. This is normally a result of not downloading either blockbots.conf, ddos.conf, whitelist-ips.conf, whitelist-domains.conf, bad-referrer-words.conf, custom-bad-referrers.conf, blacklist-user-agents.conf or blacklist-ips.conf into your /etc/nginx/bots.d folder. If any of the include files are missing Nginx will EMERG and will not reload.
It is HIGHLY SUGGESTED to please use the auto installation scripts created by Stuart Cardall @itoffshore
COPY THE GLOBALBLACKLIST.CONF FILE FROM THE REPO
Copy the contents of /conf.d/globalblacklist.conf into your /etc/nginx/conf.d folder.
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/conf.d/globalblacklist.conf -O /etc/nginx/conf.d/globalblacklist.conf
If your Linux distribution does not have wget you can replace the wget commands above using curl as follows:
curl -sL https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/conf.d/globalblacklist.conf -o /etc/nginx/conf.d/globalblacklist.conf
COPY THE INCLUDE FILES FROM THE REPO
- From your command line in Linux type
sudo mkdir /etc/nginx/bots.d
- copy the all the following files into that folder
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/blockbots.conf -O /etc/nginx/bots.d/blockbots.conf
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/ddos.conf -O /etc/nginx/bots.d/ddos.conf
WHITELIST ALL YOUR OWN DOMAIN NAMES AND IP ADDRESSES
Whitelist all your own domain names and IP addresses. Please note important changes, this is now done using include files so that you do not have to keep reinserting your whitelisted domains and IP addresses every time you update.
- copy the whitelist-ips.conf file into that folder
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/whitelist-ips.conf -O /etc/nginx/bots.d/whitelist-ips.conf
- copy the whitelist-domains.conf file into the same folder
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/whitelist-domains.conf -O /etc/nginx/bots.d/whitelist-domains.conf
Use nano, vim or any other text editor to edit both whitelist-ips.conf and whitelist-domains.conf to include all your own domain names and IP addresses that you want to specifically whitelist from the blocker script. When pulling any future updates now your domains and IP whitelists will not be overwritten.
BLACKLIST USING YOUR OWN CUSTOM USER-AGENT BLACKLIST
Copy the custom User-Agents blacklist file into your /etc/nginx/bots.d folder
- copy the blacklist-user-agents.conf file from the repository
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/blacklist-user-agents.conf -O /etc/nginx/bots.d/blacklist-user-agents.conf
Use nano, vim or any other text editor to edit (if needed) blacklist-user-agents.conf to include your own custom list of bad agents that are not included in the blocker like "omgilibot" which some people choose to block. When pulling any future updates now your custom User-Agents blacklist will not be overwritten.
BLACKLIST USING YOUR OWN CUSTOM BAD REFERRERS
Copy the custom bad referrers blacklist file into your /etc/nginx/bots.d folder
- copy the custom-bad-referrers.conf file from the repository
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/custom-bad-referrers.conf -O /etc/nginx/bots.d/custom-bad-referrers.conf
Use nano, vim or any other text editor to edit (if needed) custom-bad-referrers.conf to include your own custom list of bad referrer domains that are not included in the blocker. When pulling any future updates now your custom referrers list will not be overwritten.
BLACKLIST IPS AND IP RANGES USING YOUR OWN CUSTOM LIST
Copy the custom IP blacklist file into your /etc/nginx/bots.d folder
- copy the blacklist-ips.conf file from the repository
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/blacklist-ips.conf -O /etc/nginx/bots.d/blacklist-ips.conf
Use nano, vim or any other text editor to edit (if needed) blacklist-ips.conf to include your own custom list of IP Addresses and IP Ranges that you wish to block.
When pulling any future updates now your custom IP blacklist will not be overwritten.
DOWNLOAD CUSTOM BAD REFERRER WORDS INCLUDE FILE FOR CUSTOMIZED SCANNING OF BAD WORDS
Copy the custom bad referrer words include file into your /etc/nginx/bots.d folder
- copy the bad-referrer-words.conf file from the repository
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/bad-referrer-words.conf -O /etc/nginx/bots.d/bad-referrer-words.conf
Use nano, vim or any other text editor to edit the bad-referrer-words.conf file as you like. When pulling any future updates now your custom bad referrer words list will not be overwritten.
INCLUDE IMPORTANT SETTINGS IN NGINX.CONF Also see SAMPLE-nginx.conf file in the root of this repository
- copy the botblocker-nginx-settings.conf file directly from the repo
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/conf.d/botblocker-nginx-settings.conf -O /etc/nginx/conf.d/botblocker-nginx-settings.conf
What is included in this settings file above for nginx? The important settings file above adds the rate limiting functions and hash_bucket settings for nginx for you. Below is what the file contains, you cn add these manually to your nginx.conf file if you so please but the include file above will do it for you ad nginx loads any .conf file in /etc/conf.d (See STEP 6)
server_names_hash_bucket_size 64;
server_names_hash_max_size 4096;
limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
limit_conn_zone $binary_remote_addr zone=addr:50m;
PLEASE NOTE: The above rate limiting rules are for the DDOS filter, it may seem like high values to you but for wordpress sites with plugins and lots of images, it's not. This will not limit any real visitor to your Wordpress sites but it will immediately rate limit any aggressive bot. Remember that other bots and user agents are rate limited using a different rate limiting rule at the bottom of the globalblacklist.conf file.
The server_names_hash settings allows Nginx Server to load this very large list of domain names and IP addresses into memory. You can tweak these settings to your own requirements.
MAKE SURE that your nginx.conf file contains the following include directive. If it's commented out make sure to uncomment it or none of this will work.
include /etc/nginx/conf.d/*;
ADD INCLUDE FILES INTO A VHOST
Open a site config file for Nginx (just one for now) and add the following lines.
These includes MUST be added within a server {} block of a vhost otherwise you will get EMERG errors from Nginx.
-
include /etc/nginx/bots.d/blockbots.conf; -
include /etc/nginx/bots.d/ddos.conf;
TESTING YOUR NGINX CONFIGURATION
sudo nginx -t
If you get no errors then you followed my instructions so now you can make the blocker go live with a simple.
sudo service nginx reload
The blocker is now active and working so now you can run some simple tests from another linux machine to make sure it's working.
TESTING
Run the following commands one by one from a terminal on another linux machine against your own domain name. substitute yourdomain.com in the examples below with your REAL domain name
curl -A "googlebot" http://yourdomain.com
Should respond with 200 OK
curl -A "80legs" http://yourdomain.com
curl -A "masscan" http://yourdomain.com
Should respond with: curl: (52) Empty reply from server
curl -I http://yourdomain.com -e http://100dollars-seo.com
curl -I http://yourdomain.com -e http://zx6.ru
Should respond with: curl: (52) Empty reply from server
The Nginx Ultimate Bot Blocker is now WORKING and PROTECTING your web sites !!!
UPDATING THE NGINX BAD BOT BLOCKER is now easy thanks to the automatic includes for whitelisting your own domain names.
Updating to the latest version is now as simple as:
cd /etc/nginx/conf.d
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/conf.d/globalblacklist.conf -O /etc/nginx/conf.d/globalblacklist.conf
sudo nginx -t
sudo service nginx reload
And you will be up to date with all your whitelisted domains included automatically for you now.
See the latest auto updater bash script at:
With great thanks to Stuart Cardall (https://github.com/itoffshore) for improving on it to be truly universal to other distro's.
Relax now and sleep better at night knowing your site is telling all those baddies they are FORBIDDEN !!!
All Additions, Removals and Corrections will all be checked for accuracy before being merged into main blocker.
- https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
- https://github.com/mitchellkrogza/Badd-Boyz-Hosts
- https://github.com/mitchellkrogza/Ultimate.Hosts.Blacklist
- https://github.com/mitchellkrogza/Stop.Google.Analytics.Ghost.Spam.HOWTO
- https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites
- https://github.com/mitchellkrogza/fail2ban-useful-scripts
- https://github.com/mitchellkrogza/linux-server-administration-scripts
- https://github.com/mitchellkrogza/Travis-CI-Nginx-for-Testing-Nginx-Configuration
- https://github.com/mitchellkrogza/Travis-CI-for-Apache-For-Testing-Apache-and-PHP-Configurations
- https://github.com/mitchellkrogza/Fail2Ban-Blacklist-JAIL-for-Repeat-Offenders-with-Perma-Extended-Banning
- https://github.com/funilrys/PyFunceble
- https://github.com/funilrys/dead-hosts
- https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites
- https://github.com/mitchellkrogza/Suspicious.Snooping.Sniffing.Hacking.IP.Addresses
- https://github.com/mitchellkrogza/Fail2Ban.WebExploits
This repository contains a list of all web sites I come across that are hacked with malware. Most site owners are unaware their sites have been hacked and are being used to plant malware.
Check it out at: https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites
Come drop by and visit me at mitchellkrog.com or Facebook or Follow Me on Twitter
Many Thanks to those contributing to this project.
Many parts of the generator scripts and code running behind this project have been adapted from snippets from hundreds of sources. In fact it is so hard to mention everyone but here are a few key people whose little snippets of code have helped me introduce new features all the time. Show them some love and check out some of their projects too.
- Stuart Cardall - https://github.com/itoffshore (Install, Update and Setup Scripts & Alpine Linux Package Maintainer)
- Nissar Chababy - https://github.com/funilrys/funceble (Excellent script for checking ACTIVE, INACTIVE and EXPIRED Domain Names)
- Marius Voila https://github.com/mariusv
- Cătălin Mariș https://github.com/alrra
- deformhead https://github.com/deformhead
- bluedragonz https://github.com/bluedragonz
- Alexander https://github.com/shoonois
- Steven Black https://github.com/StevenBlack
- Fail2Ban - https://github.com/fail2ban
- Stevie-Ray Hartog https://github.com/Stevie-Ray
- Sir Athos from StackOverFlow - http://stackoverflow.com/users/2245910/sir-athos (help with Travis Build Tagging and Committing)
- StackOverflow - http://stackoverflow.com/ (bash scripts from hundreds of questions and answers)
- SuperUser - http://superuser.com/ (snippets from various questions and answers)
If you believe your name should be here, drop me a line.
Thousand of hours of programming and testing have gone into this project, show some love
Copyright (c) 2017 Mitchell Krog - mitchellkrog@gmail.com
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.