[stable/nginx-ingress] Get client real IP address with service type LoadBalancer

[Giancarlo1974]

Describe the bug
I have services with LoadBalancer in K3s and using Nginx ingress to expose these services to the internet. When I try to get the client IP address in the application I am getting cluster's node IP. How can I retrieve real client IP?

Version of Helm and Kubernetes:
kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3+k3s1", GitCommit:"96653e8d86b74a1f73eadf09d4edc44778c2f8dd", GitTreeState:"clean", BuildDate:"2020-06-01T16:47:11Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}

helm version
version.BuildInfo{Version:"v3.1.1", GitCommit:"afe70585407b420d0097d07b21c47dc511525ac8", GitTreeState:"clean", GoVersion:"go1.13.8"}

Which chart:
$ helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION nginx ingress-nginx 1 2020-07-20 08:02:09.225502812 +0200 CEST deployed nginx-ingress-1.41.0 v0.34.0

How to reproduce it (as minimally and precisely as possible):

I am using k3s on a single node
I installed the chart using these values:

cat values.yaml

controller:
service:
externalIPs:
- x.x.x.x
externalTrafficPolicy: Local

defaultBackend:
service:
externalIPs:
- x.x.x.x

rbac:
create: true

I installed the chart in this way:

`helm repo add stable https://kubernetes-charts.storage.googleapis.com/

kubectl create namespace ingress-nginx

kubectl create -f configmap.yaml
kubectl create -f custom-headers.yaml

helm install nginx stable/nginx-ingress --namespace ingress-nginx -f values.yaml

cat configmap.yaml`

apiVersion: v1
data:
proxy-set-headers: "ingress-nginx/custom-headers"
use-forwarded-headers: "true"
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx

cat custom-headers.yaml

apiVersion: v1
data:
X-Different-Name: "true"
X-Request-Start: t=${msec}
X-Using-Nginx-Controller: "true"
X-Real-IP: $remote_addr
kind: ConfigMap
metadata:
name: custom-headers
namespace: ingress-nginx

The nginx works, i can connect to public ip address x.x.x.x

When I try to get the client IP address in the application I am getting cluster's node IP.
In fact from the logs I see the IP of the container

`
$ kubectl -n ingress-nginx get po
NAME READY STATUS RESTARTS AGE
svclb-nginx-nginx-ingress-controller-k6snc 2/2 Running 0 84m
nginx-nginx-ingress-default-backend-679f548db6-zctpw 1/1 Running 0 84m
nginx-nginx-ingress-controller-6b855b474-r4df9 1/1 Running 0 84m

kubectl -n ingress-nginx logs nginx-nginx-ingress-controller-6b855b474-r4df9

10.42.0.1 - - [20/Jul/2020:07:28:03 +0000] "GET /wp-content/uploads/2019/12/photo.jpg HTTP/2.0" 200 104310 "https://www.mysite.com/folder/" "Mozilla/5.0 (Linux; Android 10; POT-LX1T Build/HUAWEIPOT-L21T; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/83.0.4103.106 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/279.0.0.43.120;]" 60 0.002 [php-nginx-443] [] 10.42.0.65:443 104310 0.004 200 7d3082d76e83a7cccf47a1568c8c81bf
`

How can I retrieve real client IP?

[lyuma]

Hey,
Guess what? I just ran into the same problem you ran into, and I struggled just as much. No way to find the ip. Always 10.42.0.1 - it seemed hopeless.

And then I discovered the hostNetwork: true option, and my world flipped upside down.

I wrote an in-depth explanation of what I did at the k3s bug tracker here (Caveat: only tested in a one-node cluster): k3s-io/k3s#1652 (comment)

Hope this solves your problem. Let me know if you run into trouble!

[Giancarlo1974]

I checked in the nginx chart if it is possible to set the property hostNetwork: true

controller.hostNetwork

If the nginx deployment / daemonset should run on the host's network namespace. Do not set this when controller.service.externalIPs is set and kube-proxy is used as there will be a port-conflict for port 80 | false

I currently use service.externalIPs

[lukos]

There is an article here that describes what it sounds like you want.

You can pass --set controller.service.externalTrafficPolicy=Local to the helm install and on some load balancers, this will cause the client ip to be passed in a proxy header.

This works on Azure with a Load Balancer.

[Giancarlo1974]

Thank you for the help, but it doesn't work on k3s

To prove that
controller.service.externalTrafficPolicy=Local
doesn't work on K3s I have configured stable/nginx-ingress with that line only

I installed wordpress
by enabling only

ingress = true
hostname: wordpress.local

helm install my-wp -f values.yaml bitnami/wordpress

I installed MetalLB

to make the configuration work:
service.type: LoadBalancer

ok bitnami wordpress run correctly
i see it browsing http://wordpress.local

Now to prove the bug on nginx
I modify the wordpress index.php
to show the client ip

kubectl exec -it my-wp-wordpress-58d88fd54f-l5h2w bash

cd /opt/bitnami/wordpress

echo '<?php
phpinfo();
' > index.php

Now browsing http://wordpress.local i see:
HTTP_X_REAL_IP 10.42.0.1

[Giancarlo1974]

I solved the problem by mixing the solution of all 3
the correct nginx configuration is:

controller.service.externalTrafficPolicy=Local
controller.hostNetwork=true

Finally it is necessary to install MetalLB, to make the public IP accessible on Bare Metal

[scottrigby]

stable/nginx-ingress has been deprecated in favor of the upstream ingress-nginx repo. See #22823. Issues if still applicable can be re-opened there. Thanks 😊