[nginx] Allow custom configuration at multiple levels

[foxylion]

I think it would be great to allow customized configuration to be injected into the template without the need to modify it.

In the config map we could add the following options:
custom-http-configuration: A snippet which is added at the end of http { ... }
custom-server-configuration: A snippet which is added at the end of every server { ... } for ingress configurations
custom-location-configuration: A snippet which is added at the end of the location { ... } inside the server config

This would allow to solve most customization without modifying the template itself.

Additionally it could be allowed to use the configuration options for server and location also an ingress annitations. This could enable customized configuration for a specific ingress rule.

[aledbf]

@foxylion you already can do some of this at ingress level https://github.com/kubernetes/ingress/tree/master/examples/customization/configuration-snippets/nginx
and https://github.com/kubernetes/ingress/tree/master/examples/customization/custom-headers/nginx

[foxylion]

@aledbf This is so far great, yes. But some things I've to configure for every ingress (e.g. proxy_pass_request_headers on; it would be nice if this is possible.
Because ingress configurations are written by all developers, ingress deployment is only done by the people administrating the cluster. And the developers should not necessarily need to care about things like this.

[aledbf]

But some things I've to configure for every ingress (e.g. proxy_pass_request_headers on; it would be nice if this is possible.

The default for proxy_pass_request_headers is on