Nginx Ingress - 400 Bad Request: Request Header Or Cookie Too Large

[qrpike]

Whenever I have a ingress controller which proxies to an ingress controller in a different namespace, I sometimes get "400 Bad Request: Request Header Or Cookie Too Large"

Note that I am also using Basic Auth on the second ingress controller.

Thanks,

[aledbf]

@qrpike please check if adding large-client-header-buffers: "4 16k" in the configuration ConfigMap solves the issue.

[qrpike]

@aledbf I was looking at that. I have kubernetes self hosted on bare metal so I often need to talk across namespaces. As long as I don't enable auth it works fine, enabling auth causes the issues.

I worked around this by having the main ingress just point directly to that namespaces services. Gets rid of a network hop as well.

Thanks,

[aledbf]

@qrpike please reopen if you still have issues

[MilanDasek]

Hello,

We are experiencing same problem .

ingress version: gcr.io/google_containers/nginx-ingress-controller:0.8.3

I have added large-client-header-buffers: "4 16k" into configMap - but it was not propagated to to pod.

    args:
    - /nginx-ingress-controller
    - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
    - --nginx-configmap=$(POD_NAMESPACE)/nginx-ingress-controller

apiVersion: v1
data:
server-name-hash-bucket-size: "128"
large-client-header-buffers: "4 16k"
kind: ConfigMap
metadata:
name: nginx-ingress-controller

[qrpike]

@MilanDasek This is because ( i found out recently ) that the ingress controller actually listens in ALL namespaces. You get that header-buffer because it's doing a ton of redirects between your namespaces before finally arriving at the location ( based on luck of finally getting through ). Thats why you only get it sometimes.

So you should make 1 ingress controller deployment/daemonset in default namespace or where ever you like. And create the ingresses for your services in that services namespace. The ingress controller will see it and route the traffic accordingly. I was shocked too, but it works.

[MilanDasek]

@qrpike

I am sorry, I would like to be sure I do understand.

So Nginx Ingress Controller (basically Nginx server) should be deployed in "default" namespace and all ingresses for all services should be in "default" namespace as well, instead of respective namespaces where underlying services are sitting?

That means I have to deploy this ingress to "default" namespace (for example)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: proxy
annotations:
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:

• host: test.url
  http:
  paths:
  • path: /
    backend:
    serviceName: NAMESPACE/proxy
    servicePort: 80

Correct?

[qrpike]

@MilanDasek No, nginx ingress controller should be in default, and the ingress's should be in whichever namespace that service is in. The ingress MUST be in the same namespace as the service it is routing to.

[MilanDasek]

@qrpike so it is exactly how I have it set now.

nginx ingress controller in default NS
ingresses for services are in the same namespaces.

So what is the idea then?

[qrpike]

@MilanDasek Then it should work as expected. You dont need namespace in front of your service name. eg: serviceName: NAMESPACE/proxy

[MilanDasek]

Well but it isn't sometimes. I would like still to have a possibility to change

large-client-header-buffers: "4 16k"

[qrpike]

@MilanDasek Check to make sure you dont have ingress's or services in the wrong namespaces.

[aledbf]

@MilanDasek you can change that setting using the configuration configmap.
Just add large-client-header-buffers: "XX XX"

[MilanDasek]

I have this

apiVersion: v1
data:
large-client-header-buffers: 4 16k
server-name-hash-bucket-size: "128"
kind: ConfigMap
metadata:
creationTimestamp: 2017-03-07T18:31:22Z
name: nginx-ingress-controller

but when I search in nginx.conf in ingress
kubectl exec nginx-ingress-controller-XXX -- cat /etc/nginx/nginx.conf | grep client

I got only
client_max_body_size "1m";

[aledbf]

@MilanDasek client_max_body_size is configured with proxy-body-size. The body and headers are configured by different nginx directives

[aledbf]

@MilanDasek just in case this feature is not present in 08.3. You need to use one of the 0.9 betas.

[MilanDasek]

@aledbf that is the problem, I use 0.8.3 ,but when I use 0.9.0-beta2, TCP stream stops working.

my config:
apiVersion: v1
kind: ConfigMap
metadata:
name: name-27017
data:
27017: "namespace/service:27017"

ingress RC
- --nginx-configmap=$(POD_NAMESPACE)/nginx-ingress-controller
- --tcp-services-configmap=$(POD_NAMESPACE)/name-27017

I found out I have to change
- --configmap=$(POD_NAMESPACE)/nginx-ingress-controller
- --tcp-services-configmap=$(POD_NAMESPACE)/name-27017

but tcp does not work

I am following https://github.com/kubernetes/ingress/tree/master/controllers/nginx
but seems it is fixed in #344
which is not in beta2 https://github.com/kubernetes/ingress/blob/master/controllers/nginx/Changelog.md

[MilanDasek]

works in beta3
https://github.com/kubernetes/ingress/blob/master/controllers/nginx/Changelog.md

[aledbf]

@MilanDasek can we close this then?

[MilanDasek]

Yes, thank you.