[stable/nginx-ingress] http2 + large cookies

[huang-jy]

Describe the bug
When using large cookies and having http2 enabled, nginx-ingress returns an empty reply, or a 400 Bad Request if http2 is not enabled.

Version of Helm and Kubernetes:
Helm:

Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}

Kubernetes:

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.12", GitCommit:"a8b52209ee172232b6db7a6e0ce2adc77458829f", GitTreeState:"clean", BuildDate:"2019-10-15T12:12:15Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.12-gke.25", GitCommit:"654de8cac69f1fc5db6f2de0b88d6d027bc15828", GitTreeState:"clean", BuildDate:"2020-01-14T06:01:20Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
nginx-ingress version 1.6.17

What happened:
When using large cookies with http2 enabled, connection is dropped, regardless of whether or not http2 directives are being included.

What you expected to happen:
http2 directives to be followed, or clarified

How to reproduce it (as minimally and precisely as possible):

Generate cookies of various sizes and curl the dns endpoint for your ingress.
curl -I --fail -H "Cookie: test_size=\"$(openssl rand -hex 1100)\"" https://{dns-name-for-your-ingress} 2>&1

curl -I --fail -H "Cookie: test_size=\"$(openssl rand -hex 4100)\"" https://{dns-name-for-your-ingress} 2>&1

curl -I --fail -H "Cookie: test_size=\"$(openssl rand -hex 8100)\"" https://{dns-name-for-your-ingress} 2>&1

With the OOTB config, cookie size 1100 (generates 2200 byte cookie) works, but 4100 (8200 bytes) and 8100 (16200 bytes) will fail, with the response

curl: (52) Empty reply from server

If you add the use-http2: "false" directive to the configmap, then instead of "Empty reply from server", you will get

curl: (22) The requested URL returned error: 400 Bad Request

If we use the solution mentioned in kubernetes/ingress-nginx#319, and add

large-client-header-buffers: "4 4k"

Then all three cookie sizes will work.

However, if we switch http2 back on:

use-http2: "true"

We still get

curl: (52) Empty reply from server

Even adding:

http2_max_field_size: "16k"
http2_max_header_size: "24k"

Does not appear to help fix the problem

Anything else we need to know:

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[huang-jy]

still valid

…

On Sat, 21 Mar 2020 at 10:20, stale[bot] ***@***.***> wrote: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#20901 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVA5O6EIUW622X7WJ6Q4GLRISIGZANCNFSM4KYLPWEQ> .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[huang-jy]

still valid

…

On Thu, 23 Apr 2020 at 05:05, stale[bot] ***@***.***> wrote: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#20901 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVA5O5BMCZA37YR4SIS4MTRN65APANCNFSM4KYLPWEQ> .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[huang-jy]

Still valid

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.

[LeeStevens318]

Did you manage to find a solution i'm having the same issue.

[huang-jy]

Did you manage to find a solution i'm having the same issue.

No, we ended up having to try to find a way to reduce the cookie size. We still occasionally get 400s which isn't a good user experience.

[LeeStevens318]

still valid

[LeeStevens318]

@huang-jy, i think i managed to get it to work.
I created a file called nginx-config.yaml with the below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  large-client-header-buffers: "8 32k"
  http2-max-field-size: "8k"
  http2-max-header-size: "32k"

Ran kubectl apply -f nginx-config.yaml

and then tried curl -I --fail -H "Cookie: test_size=\"$(openssl rand -hex 4100)\"" https://{dns-name-for-your-ingress} 2>&1 it worked:

HTTP/2 302
date: Tue, 30 Mar 2021 10:48:25 GMT
location: {removed}
strict-transport-security: max-age=15724800; includeSubDomains

Before: curl: (52) Empty reply from server

[huang-jy]

You're patching the config post-deployment. You can add custom config directives by overriding the values.yaml during a helm install

https://github.com/kubernetes/ingress-nginx/blob/d4b3c126745879695524c6a3449e2b1625526ed8/charts/ingress-nginx/values.yaml#L27

Also which version ingress controller are you using? It's been over a year this I raised this, so maybe they've fixed it since

[LeeStevens318]

Ah yes, the deployment is different, i'm using the deployment "guide" here: https://kubernetes.github.io/ingress-nginx/deploy/#azure
Which is just: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/cloud/deploy.yaml

[huang-jy]

Ah, that now makes sense as to why you're patching your configmap.

[LeeStevens318]

Yeah sorry, i was more concentrating on getting it to work and not really noticing your deployment method.

[huang-jy]

It's fine :)