Extended nginx config

[magenx]

#2562

[kandy]

Thanks for contributing, but your configuration is very specific for your installation. Moreover, i believe that for us don't need some production configuration in magento at all, only configuration for development.

[magenx]

@kandy sounds like you dont even understand what are you talking about... that sad to hear, especially when you have incomplete and bad Magento v1 config (yet in manuals, you dont even bother to revise it for few years), and now you have some castrated, childish and incomplete config for Magento v2.
so confused users without any knowledge proposing something stupid like this -> #2178

even more sad when we take random url from your list of early adopted Magento v2 webshops, and we able to download everything. you will ask why? simple because you as business application developer dont care about its security, even after that fact that Nginx is in your system requirements...

we never said this config is perfect or specific, but it is more likely as default config, that is better to use rather than nothing.

i would write more, but i guess it is all useless for you.

p.s. "you" i mean Magento and Magento employees who should really care about it.

so from your words kandy you dont care about security in production, and you dont even care how secure is your development server. you have to be kidding me...

answer this question - why do you put .htaccess files everywhere?? then this will be THE answer for all your doubts.

cheers

[kandy]

Thanks again for feedback.
Yes our sample config minimalistic as possible and not usable for production use because we do not try to cover production scenario at all.
Your configuration is more powerful and require more understanding and qualification to use, that is a cons for developing process.

[piotrekkaminski]

We have control over this header in configuration

[magenx]

maybe is better to send headers like these from the server not from the application. sending headers from application is good for debug or to report some errors with api ,.. etc

[piotrekkaminski]

@magenx thats for this PR. We will review. While we are not expecting anyone to use our sample nginx config for production, I agree this protects even dev environment much better.
@kandy there are a lot of publicly available dev sites and they are getting infected or used for attacks. Our sample configs, even if not designed for production, should be safe - and this nginx config is a very good starting point.
eg exposed .git is common issue on Nginx sites. This config fixes it.

[magenx]

guys we just need your input, because Magento 2 is very new, and even this stuff can be all wrong if you look from the other angle. but just to be sure we run on this config from web installation to some admin tasks, looks like there is no major errors.

[gwillem]

Why not 403 (or deny all) like with the other sensitive files?

[magenx]

actually as i remember we have been working with firewall , like these 403 can be configured with log scanners like fail2ban and csf firewall and block multiple abusers, where 444 the same, the only difference is that 444 will terminate request immediately and user just gets no response back. need to revise this logic. separate log for 403 errors.

[vkublytskyi]

Due to internal discussion should be included in Magento

[samm-git]

limit requests and return GONE 410 for search bots (if you need something more - use firewall/iptables)

location ~ ^/(wishlist|customer|catalogsearch|newsletter|contact|sendfriend|catalog/product_compare|review/product/post|(fire|one.+)?checkout)/ {
limit_req zone=goeasy burst=5;
limit_req_status 429;
if ($http_user_agent ~* "Baiduspider|Googlebot|bingbot|rogerbot|Yahoo|YandexBot") { return 410; }
try_files $uri $uri/ /index.php?$args;
}

Do we really want to include that in our distro? Sorry, its a custom decision which may (and will) affect search performance and i would never recommend to do this for the website without strict reason. Can we move all this crap out?

[samm-git]

Another note - hhvm is not officially supported, i am not recommending to include it configuration in the git. Its a perfect idea for 3rd party repository (or just a blog post) but i dont think we should include it into the repository until it is officially supported.

[samm-git]

this potentially can block some perl/python programs. Why do we need to response with 444 in this case? Also 444 have a special meaning " Used to indicate that the server has returned no information to the client and closed the connection." Why do you want to use this code? It does not look like correct behavior.

[magenx]

can block some perl/python programs.

which one?? when we tested this config few python and java clients where removed, others just a default noisy agents, it is good to keep them out. 444 will save you a lot of resources.

[magenx]

Do we really want to include that in our distro? Sorry, its a custom decision which may (and will) affect search performance and i would never recommend to do this for the website without strict reason. Can we move all this crap out?

lol, if you need search bots to to hit and index all these url wishlist|customer|catalogsearch|newsletter|contact|sendfriend|catalog/product_compare|review/product/post|(fire|one.+)?checkout
go ahead remove it on your server/shop configuration...

[samm-git]

" which one?? when we tested this config few python and java clients where removed, others just a default noisy agents, it is good to keep them out. 444 will save you a lot of resources."

E.g. HTTP::Lite. I dont think that it makes any sense at all. If you will get real ddos - they will use some "real" UA. If it is just some crawlers or integrations - its typically not using a lot of resources. About 444 code please read HTTP RFC before doing such weird things. It is a debugging hell.

About buckets for paths - one can just use robots.txt + block bots who not respecting it (not a lot). From my POV - this configs are kind of overcomplicated and are not necessary for the most of the users. Moreover - from my experience most of Magento high load customers do not using such things, because, again, it does not help you to survive real ddos but makes debugging and maintenance much more painful. E.g. if some integration will fall into "bad_client" pattern (very possible) last thing which web developer will check is nginx config. And excluding such requests from log make debugging even harder. So i personally dont see any reason to keep any of this "additional" logic enabled. We can supply them in .conf.extra, which is disabled by default and used only if enabled, just for the users who really needs that.

[magenx]

what ddos?? I wasn't even thinking about ddos protection in nginx, who will ever use nginx for anti-ddos?? there is minimal default protection from excessive requests, which you can easily split to different rules or increase rate as you need.

how it makes your debugging even harder?? what exactly falls into "bad_client" pattern ??

when the last time you succeeded with robots.txt, as i know robots.txt rules are not honored even by google bot...

[samm-git]

"when the last time you succeeded with robots.txt, as i know robots.txt rules are not honored even by google bot..."
its not true. I have positive results with all major search engines (google, bing, yandex, etc.). And i am blocking agents which do not respect robots only if they causing significant load. Sometime it happens, and mostly from chinese bots (nothing personal), so its easy to block them completely without affecting customer business.

[krasnuydyx]

Closing pull request because of too complex logic for the general-use configuration

[magenx]

Closing pull request because of too complex logic for the general-use configuration

indeed is too complex... sounds like we trying to create a vaccine against the terrible virus ....
no one even tried to offer any ideas on how to improve it, seems like only personal matter, i only see a lack of knowledge and experience here...

[krasnuydyx]

Thank you for your contribution, but we are trying to keep our web server configuration as simple as possible. Also all nginx logic should be equal to the apache one. I would recommend to create an article about your configuration with a github repository, so anyone could use it if needed.