-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem with HTTPS, CloudFlare and X-Forwarded-Port header #6358
Comments
Hi ! I found a way to do a tcpdump in nginx-ingress-controllers. CloudFlare doesn't send a X-Forwarded-Port Header. I think this is related to this :
This variable is set to the server port. More critical is that I think this variable is not overwritable in general for example, nginx.conf when I added this variable to annotation :
|
I tried adding these on Ingress object but didn't change the header at all either. (Based on the "fix" in #3481 (comment))
and
I verified they showed up in |
Seems like this has somewhat been discussed in terms of the we can't override this header. #3481 (comment) @aledbf do you think it would make sense to add an config to let user override "X-Forwarded-Port" header, or do you know why |
Did you configure CloudFlare with the Full(strict) mode ? In this mode CloudFlare should talk to Ingress Nginx through port 443 which I think would cause Ingress Nginx to set the |
This. Please verify that CloudFlare it set to |
I think there is an issue with the way the Lua code sets ingress-nginx/rootfs/etc/nginx/lua/lua_ingress.lua Lines 147 to 149 in 6c729e9
Ideally if if ngx.var.http_x_forwarded_port then
ngx.var.pass_server_port = ngx.var.http_x_forwarded_port
elseif ngx.var.http_x_forwarded_proto == "http" then
ngx.var.pass_server_port = 80
elseif ngx.var.http_x_forwarded_proto == "https" then
ngx.var.pass_server_port = 443
end |
@ailurarctos this would solves the problem but it is not clear to me if this is an acceptable / standard behavior ? It seems to me that CloudFlare SSL should set |
CloudFlare does not set @tpoindessous
I assume you are not using a HTTPS Load balancer in GKE ? |
@toredash if you use a snippet to set the Ideally if the upstream loadbalancer is not sending an |
@Dohbedoh if there is a way to override or set those headers that would work for me. |
Thanks, I will try.
No, I'm not using a HTTPS Load Balancer in GKE, I'm using a
nginx-ingress-controller (associated automatically with a External TCP Load
Balancer).
For the record : GCP HTTP Load Balancer in GKE, in NEG configuration, has
the same issue : it's listening in HTTP/ port 80, and it sets a
X-Forwarded-Port:
80, even if it is placed behind a HTTPS Cloudflare configuration.
thanks !
Le mer. 4 nov. 2020 à 08:53, Tore <notifications@github.com> a écrit :
… CloudFlare does not set X-Forwarded-Port according to their docs:
https://support.cloudflare.com/hc/en-us/articles/200170986-How-does-Cloudflare-handle-HTTP-Request-headers-
@tpoindessous <https://github.com/tpoindessous>
I think you need to set a location based snippet, as such:
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header X-Forwarded-Port 443;
I assume you are *not using* a HTTPS Load balancer in GKE ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6358 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAXJYEGJPZAPS7XVZILOHU3SOEBLXANCNFSM4SZTQMVA>
.
|
This does not work as expected because the value is appended, resulting in header:
Edit: We are attempting to use
but it does not overwrite the Header. |
This doesn't work because the template contains
this works as expected for the same reason. Multiple proxy_set_header for the same header appends the values |
Changing the value of the header The content of such a thing is trivial: local ngx = ngx
local _M = {}
function _M.rewrite()
if ngx.var.http_cf_connecting_ip then
ngx.log(ngx.ERR, "Changing x-forwarded-port to 443")
ngx.var.pass_port = 443
end
end
return _M curl localhost -H 'CF-Connecting-IP: 1.1.1'
Hostname: http-svc-6b7fcd49cc-jb27g
Pod Information:
node name: kind-control-plane
pod name: http-svc-6b7fcd49cc-jb27g
pod namespace: default
pod IP: 10.244.0.11
Server values:
server_version=nginx: 1.12.2 - lua: 10010
Request Information:
client_address=10.244.0.7
method=GET
real path=/
query=
request_version=1.1
request_scheme=http
request_uri=http://localhost:8080/
Request Headers:
accept=*/*
cf-connecting-ip=1.1.1
host=localhost
user-agent=curl/7.68.0
x-forwarded-for=172.18.0.1
x-forwarded-host=localhost
x-forwarded-port=443
x-forwarded-proto=http
x-real-ip=172.18.0.1
x-request-id=dc00f8bf88bc383b786a227b7d2657e4
x-scheme=http
Request Body:
-no body in request- The condition to change the variable can check for any other header (like limiting the change to a particular host) |
Thanks @aledbf ! |
This problem is driving me nuts. Any solution yet? |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
/remove-lifecycle stale |
Hi @tpoindessous @kolorful @Uysim, can you guys can confirm that the issue still exists? Also with newer versions of ingress-nginx? |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
I can confirm this still happens on nginx 1.0.1 |
@clglavan I don't have info in my hand. My application has been moved. |
/remove-lifecycle stale |
@brsolomon-deloitte thanks this is working fine in the newer 4.0.19 chart. From what i can see by browsing the nginx.conf generated, This issue seems to arise because all the X-Forwarded headers are set after the server/configuration snippet. Would moving these X-Forwarded header definition above those definitions solve them? |
@brsolomon-deloitte @dhinesh-babu-bankbuddy I could not completely follow up on your conversation. Do you still consider this a bug? |
This is why I pushed this PR #7542
Workaround proposed can't be used when mixing HTTP / HTTPS. |
ok thanks for clarifying. BTW when looking at this issue there is no PR linked, which is why you should fill the template when opening a PR, so that issue and PR are linked. Anyway, I pingee @rikatz to follow up here :) |
I just wanted to mention that with the most recent ingress-nginx helm chart this solution needs to be modified to work correctly. |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
@DavidKittleSEL, doesn't work on helm chart 4.2.5 with Jenkins. I get internal calls redirected to http not respecting the proto call. |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The issue has been marked as an important bug and triaged. Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle frozen |
Is there any plan for this error?
|
Just to add a bit to this, using the bitnami chart, this is what we did:
|
having the same problem here (and with keycloak too)... |
@gab-despreslaberge thank you, this is the solution that worked. Can confirm with ingress-nginx 4.2.0. |
Not sure if this is affecting anyone else here, but OOTB on latest helm chart and ingress-nginx, this works fine for TLS routes (can't speak to non-tls/tls mixture) for me. The thing that was affecting me was that I was getting a 404 when my Cloudflare DNS reverse proxy setting was toggled on for my DNS A-record, but it worked when I toggled it off. This was because I had a Cloudflare Worker that had bound all HTTP routes to it on the same DNS zone for *.mydomain.com. This forced all traffic to route through my worker and not my TCP load balancer on GKE. The solution was to add an explicit HTTP route on my Workers Routes and set the worker to "Disabled" so it would just target my LB instead of any CF Worker. |
@tpoindessous is this still an issue? Did you test it on the latest like @Schachte talks about? |
Hi !
Sorry, I can't test anymore :(
Thanks !
Le ven. 26 avr. 2024 à 16:41, James Strong ***@***.***> a
écrit :
… @tpoindessous <https://github.com/tpoindessous> is this still an issue?
Did you test it on the latest like @Schachte <https://github.com/Schachte>
talks about?
—
Reply to this email directly, view it on GitHub
<#6358 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAXJYEGENBXBUEUOPKAT2OLY7JRABAVCNFSM4SZTQMVKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBXHE2TEMJYGU4A>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I tested on controller v1.10.1 and I saw X-Forwarded-Port retained at 443 as per my curl request. I will close this for now as no more tests can be done from users as posted earlier. If there is a problem found in the controller and can be reproduced on minikube or kind cluster, please re-opn the issue and provide the reproduce steps. /close |
@longwuyuan: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I am using the Lua plugin based on the above and it works fine. However, is there a more elegant way of doing this as we don't want to maintain a custom plugin for our needs? |
NGINX Ingress controller version: 0.40.2
Kubernetes version (use
kubectl version
):Environment: GKE
uname -a
): GKEWhat happened:
Hi, I have this setup
CloudFlare with HTTPS and Proxy mode => GKE nginx-ingress-controller in HTTP => myApp.
When I got to my website in HTTPS mode, myApp receives this request :
So, because of
X-Forwarded-Port: 80
, myApp thinks that original request was in HTTPS BUT on port 80 :( and then, links generated on the response are not OK.What you expected to happen:
I would like to receive a
X-Forwarded-Port: 443
header.How to reproduce it:
My conf :
Anything else we need to know:
I can't verify the original request (and so, headers added by CloudFlare) which arrives in nginx-ingress-controller, as I don't have tcpdump installed on this container. If you know how to dump the original request , I could dump it.
/kind bug
The text was updated successfully, but these errors were encountered: