mirroring: can't increase accepted body size

[iameli]

EDIT: Worked around this with

      http-snippet: client_max_body_size 1G;

What happened:

I have a global body size parameter set via the configmap:

proxy-body-size: "1G"

I also have a mirroring rule enabled on my ingress:

But when I try and mirror those requests to the other ingress, it fails:

2023/02/28 17:03:19 [error] 5383#5383: *84242823 client intended to send too large body: 1526184 bytes, client: 10.42.0.1, server: redacted, request: "POST /live/redacted/2.ts HTTP/1.1", subrequest: "/_mirror-redacted", host: "redacted"

What you expected to happen:

Should mirror the request body.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.3.1
  Build:         92534fa2ae799b502882c8684db13a25cde68155
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:28:30Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1+k3s1", GitCommit:"f10af367c3c96863c081ada4018e94e085c9404d", GitTreeState:"clean", BuildDate:"2023-01-26T00:59:34Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Bare metal
• OS (e.g. from /etc/os-release): Ubuntu 20.04.5 LTS
• Kernel (e.g. uname -a): Linux dp5585 5.4.0-139-generic Fix issue for ratelimit #156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: k3s

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

@iameli I think the critical information already asked by the new issue template is very much required but not available in your issue description.

Kindly update your main description of the issue and add the info related to the issue extracted from the cluster as the live state. For example ;

• kubectl get svc,ng -A -o wide
• kubectl -n $appnamespace describe ing
• kubectl -n $controllernamespace logs $controllerpodname
• Your exact and complete curl command with -v and its response

[longwuyuan]

/remove-kind bug

[longwuyuan]

You may also want to test with the recently released version of the controller.

[iameli]

Respectfully, most of that information is irrelevant and would require quite a bit of redaction on my part to post here; our ingress logs contain the ids of user resources that aren't public, our service logs contain IP addresses that aren't public, and our ingresses contain URLs that aren't public.

The problem is clear: the proxy-body-size ConfigMap parameter correctly sets the client_max_body_size field in the nginx config for normal locations, but not for mirrored locations. Using http-snippet: client_max_body_size 1G; works around the issue by changing the default accepted body size for all locations.

I suspect the fix will involve updating the buildMirrorLocations function to add the client_max_body_size parameter. There are probably other annotations that should be added there as well to maintain parity with non-mirrored locations, but this one was sufficient for my purposes.

[longwuyuan]

@iameli that is reasonable.

Can you at least write down a step-by-step procedure, that someone can copy/paste from, on their minikube or kind cluster, to reproduce the problem.

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[sinacek]

Can you at least write down a step-by-step procedure, that someone can copy/paste from, on their minikube or kind cluster, to reproduce the problem.

I have same issue.

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Environment:

    Cloud provider or hardware configuration: Mac
    OS (e.g. from /etc/os-release): MacOS Sonoma 14.4
    Kernel (e.g. uname -a): Darwin MacBook-Air-2.local 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:51:37 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T8112 arm64

How to reproduce:

1. Install minukube and allow ingress minikube addons enable ingress
2. Run kubectl apply -f "minikube.txt"
3. Send curl request by curl --location '<your_ip_here>/foo' --data @ballast.txt
4. See logs for pods kubectl logs foo-app, kubectl logs foo-app, kubectl logs -n ingress-nginx services/ingress-nginx-controller

I got logs bellow and

#FOO
10.244.0.5:40150 | POST /foo

# NGINX
10.244.0.1 - - [18/Mar/2024:19:15:12 +0000] "POST /foo HTTP/1.1" 413 25 "-" "curl/8.4.0" 0 0.000 [-] [] - - - - 0c9ca5aba859d8ce2f095ee02aa79831
10.244.0.1 - - [18/Mar/2024:19:15:12 +0000] "POST /foo HTTP/1.1" 200 2620077 "-" "curl/8.4.0" 2614723 0.071 [-] [] 10.244.0.12:8080 2614954 0.007 200 0c9ca5aba859d8ce2f095ee02aa79831

minikube.txt
ballast.txt It has 2.6MB.

[longwuyuan]

Closing because the creator of the issue posted a message like this

EDIT: Worked around this with

      http-snippet: client_max_body_size 1G;

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

Closing because the creator of the issue posted a message like this

EDIT: Worked around this with

     http-snippet: client_max_body_size 1G;

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.