Ingress rule with non-wildcard path does not work #8518

shashankram · 2022-04-27T22:24:51Z

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.2.0
  Build:         a2514768cd282c41f39ab06bda17efefc4bd233a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"8211ae4d6757c3fedc53cd740d163ef65287276a", GitTreeState:"clean", BuildDate:"2022-03-31T20:28:03Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:
AKS

Cloud provider or hardware configuration: AKS
How was the ingress-nginx-controller installed:
- If helm was used then please show output of helm ls -A | grep -i ingress

$ helm ls -A | grep -i ingress
ingress-nginx	ingress-nginx	1       	2022-04-27 12:32:39.5649468 -0700 PDT	deployed	ingress-nginx-4.1.0	1.2.0

If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>

$ helm -n ingress-nginx get values ingress-nginx
USER-SUPPLIED VALUES:
null

Current State of the controller:
- kubectl describe ingressclasses

$ kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.2.0
              helm.sh/chart=ingress-nginx-4.1.0
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

Nginx service:

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      meta.helm.sh/release-name: ingress-nginx
      meta.helm.sh/release-namespace: ingress-nginx
    creationTimestamp: "2022-04-27T19:32:42Z"
    finalizers:
    - service.kubernetes.io/load-balancer-cleanup
    labels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
      app.kubernetes.io/version: 1.2.0
      helm.sh/chart: ingress-nginx-4.1.0
    name: ingress-nginx-controller
    namespace: ingress-nginx
    resourceVersion: "36534"
    uid: af6861ae-b2ab-4dad-9d2a-491c0be8ea5a
  spec:
    allocateLoadBalancerNodePorts: true
    clusterIP: 10.0.198.145
    clusterIPs:
    - 10.0.198.145
    externalTrafficPolicy: Cluster
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - appProtocol: http
      name: http
      nodePort: 31583
      port: 80
      protocol: TCP
      targetPort: http
    - appProtocol: https
      name: https
      nodePort: 30268
      port: 443
      protocol: TCP
      targetPort: https
    selector:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
    sessionAffinity: None
    type: LoadBalancer

Additional info

$ kubectl get svc,ing -n httpbin
NAME              TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
service/httpbin   ClusterIP   10.0.61.27   <none>        14001/TCP   47h

NAME                                CLASS   HOSTS   ADDRESS         PORTS   AGE
ingress.networking.k8s.io/httpbin   nginx   *       20.85.153.119   80      42h

$ kubectl describe ingress -A
Name:             httpbin
Labels:           <none>
Namespace:        httpbin
Address:          20.85.153.119
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /get   httpbin:14001 (10.244.1.15:14001)
Annotations:  <none>
Events:
  Type    Reason  Age                 From                      Message
  ----    ------  ----                ----                      -------
  Normal  Sync    8m5s (x5 over 42h)  nginx-ingress-controller  Scheduled for sync


# Request from another container that is able to access the backend httpbin service at /get
$ kubectl exec -n curl -ti "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')" -c curl -- curl -I http://httpbin.httpbin:14001/get
HTTP/1.1 200 OK
Server: gunicorn/19.9.0
Date: Fri, 29 Apr 2022 16:48:15 GMT
Connection: keep-alive
Content-Type: application/json
Content-Length: 208
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

What happened:
I am seeing an extremely odd behavior where paths other than the root / don't work. My sample app httpbin serves requests on multiple paths such as /, /get, /status/200 etc. However, any path other than the root path / don't seem to work, and the requests simply hang with no logs in the nginx ingress controller pod.

Working: path=/

Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001

Request:

$ curl -sI  http://"$nginx_ingress_host":"$nginx_ingress_port"/get
HTTP/1.1 200 OK
Date: Wed, 27 Apr 2022 22:11:39 GMT
Content-Type: application/json
Content-Length: 289
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

Nginx log for request:

10.224.0.4 - - [27/Apr/2022:22:11:39 +0000] "HEAD /get HTTP/1.1" 200 0 "-" "curl/7.68.0" 81 0.005 [httpbin-httpbin-14001] [] 10.244.2.10:14001 0 0.004 200 5140242512fcba8d377f33d7157f6d09

Nginx conf:
nginx-widcard.txt

Failing: path=/get

Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /get
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001

Request hangs:

$ curl -sI  http://"$nginx_ingress_host":"$nginx_ingress_port"/get
^C

No logs in Nginx for this request.

Logs corresponding to Nginx config update when switching from path / to /get:
nginx-path-update.txt

Nginx conf:
nginx-specific.txt

What you expected to happen:

Request /get should work with path=/get

How to reproduce it:

Deploy httpbin application: https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.1/manifests/samples/httpbin/httpbin.yaml
Apply Ingress resources shared above and test it

The text was updated successfully, but these errors were encountered:

k8s-ci-robot · 2022-04-27T22:24:57Z

@shashankram: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

longwuyuan · 2022-04-28T00:24:52Z

/remove-kind bug
Please search other issues and read docs. Then post the info requested in the issue-template. Its not even clear if your service type is NodePort or LoadBalancer

shashankram · 2022-04-28T16:18:29Z

/kind bug

shashankram · 2022-04-28T16:21:25Z

/remove-kind bug Please search other issues and read docs. Then post the info requested in the issue-template. Its not even clear if your service type is NodePort or LoadBalancer

"Please search other issues and read docs" is not an acceptable answer. I have looked through various similar bugs and many have been closed due to inactivity. Please take the time to point out specifically which bugs are similar and which docs might help resolve the problem.

The service is a LoadBalancer service. Check the PR description to view the specific details. If there is additional information you need, kindly ask for specific information that can help resolve the issue.

longwuyuan · 2022-04-29T02:24:52Z

/triage needs-information
/remove-kind bug

Assign the label bug after there is data available in the issue that shows the bug. This problem looks like a configuration related issue so triage needs-information

It will help everyone, if you post the information that is relevant to the problem. For example ;

kubectl get svc,ing -A
kubectl describe ing -A
kubectl logs
Send the same http request from inside your httpbin pod to http://localhost/get with a curl -v and copy/paste here
Any other information that seems related
More importantly, test the same ingress.spec.rules.http.paths.path value with a simpler app like plain old nginx:alpine

shashankram · 2022-04-29T16:53:15Z

kubectl get svc,ing -A

kubectl describe ing -A

kubectl logs

Send the same http request from inside your httpbin pod to http://localhost/get with a curl -v and copy/paste here

Any other information that seems related

More importantly, test the same ingress.spec.rules.http.paths.path value with a simpler app like plain old nginx:alpine

All of this has been shared. Please read through the PR description to find the requested info. I have shared a lot of configs, nginx conf files, and relevant logs.

There should be enough info in the bug to determine if there is a misconfiguration.

The behavior I am seeing is that there are no logs corresponding to the request in Nginx ingress controller when using a non-root path /get, which works from another container as indicated in the description.

shashankram · 2022-04-29T17:10:26Z

/cc @ingress-nginx-maintainers

longwuyuan · 2022-04-29T17:20:29Z

what happens if you put a slash after "get" ?

ashwinpagarkhed · 2022-05-03T14:39:27Z

faced the same issue. We need a /* and then you can add any additional paths

shashankram · 2022-05-03T19:40:53Z

what happens if you put a slash after "get" ?

That doesn't work either.

shashankram · 2022-05-03T19:41:39Z

faced the same issue. We need a /* and then you can add any additional paths

@ashwinpagarkhed Are you implying that a root path is necessary for additional sub paths to work? Could you share what the configuration should look like? Thanks

nshankar13 · 2022-05-03T20:18:18Z

@ashwinpagarkhed were you also using AKS or was this on another distribution?

shashankram · 2022-05-05T18:19:03Z

I left the testbed up overnight and it started working without any configuration changes. It seems to me there is some issue with Nginx on AKS, but can't seem to find relevant logs when the requests fail as the request hangs with no logs in Nginx.

Addresses an issue where an ingress without a root path does not work for some non-kind distributions like AKS: kubernetes/ingress-nginx#8518. Adding a wildcard path in addition to the status/200 path resolves the issue. Signed-off-by: nshankar13 <nshankar@microsoft.com>

kanonkanin · 2022-07-28T08:31:31Z

Hi.

We are facing the same issue after installing an AKS cluster with newer version of Kubernetes:

AKS Kubernetes version: 1.22.6 (latest possible)
Ingress version installed: nginx-ingress-0.14.0 2.3.0

This still works in the old cluster, (1.21.9, some lines removed):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: xx-ingress
namespace: staging
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:

host: yy-beta.xx.no
http:
paths:
- backend:
  serviceName: xx-api-service
  servicePort: 80
  path: /server/?(.*)
- backend:
  serviceName: xx-web-service
  servicePort: 80
  path: /?(.*)

Not working in the new cluster (1.22.6):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: xx-ingress
namespace: test
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
ingressClassName: nginx
rules:

host: xx.byggforsk.no
http:
paths:
- path: /server/?(.*)
  pathType: Prefix
  backend:
  service:
  name: xx-api-service
  port:
  number: 80
- path: /?(.*)
  pathType: Prefix
  backend:
  service:
  name: xx-web-service
  port:
  number: 80

Experimented with just path: / without wildcards, and this matches. All other combinations not. (.* (.*) etc)

longwuyuan · 2022-07-28T13:29:58Z

@kanonkanin, this api version apiVersion: extensions/v1beta1 is deprecated in K8S v1.22

kanonkanin · 2022-07-28T14:53:34Z

We use networking.k8s.io/v1 in the new version.
It is in this version where wildcard is not working. It works in extensions/v1beta1.
The problem is thus in the new version.

longwuyuan · 2022-10-11T07:47:00Z

Kubectl describe outputs for ingress object, ingress controller pod, logs of controllerpod Thanks, ; Long

…

On Thu, 28 Apr, 2022, 9:51 PM Shashank Ram, ***@***.***> wrote: /remove-kind bug Please search other issues and read docs. Then post the info requested in the issue-template. Its not even clear if your service type is NodePort or LoadBalancer "Please search other issues and read docs" is not an acceptable answer. I have looked through various similar bugs and many have been closed due to inactivity. Please take the time to point out specifically which bugs are similar and which docs might help resolve the problem. The service is a LoadBalancer service. Check the PR description to view the specific details. If there is additional information you need, kindly ask for specific information that can help resolve the issue. — Reply to this email directly, view it on GitHub <#8518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWQBBQPAFRF665LWKYDVHK3JBANCNFSM5UQRCMAQ> . You are receiving this because you commented.Message ID: ***@***.***>

keithmattix · 2023-05-31T17:50:21Z

I bet Azure/AKS#3646 is the problem here; it has to do with the AKS loadbalancer

longwuyuan · 2024-05-01T02:15:33Z

/remove-kind bug
/kind support

There is no problem with the constroller as described in this issue's original report.

So I am closing this issue.

thanks

/close

k8s-ci-robot · 2024-05-01T02:15:36Z

@longwuyuan: Those labels are not set on the issue: kind/bug

In response to this:

/remove-kind bug
/kind support

There is no problem with the constroller as described in this issue's original report.

So I am closing this issue.

thanks

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2024-05-01T02:15:38Z

@longwuyuan: Closing this issue.

In response to this:

/remove-kind bug
/kind support

There is no problem with the constroller as described in this issue's original report.

So I am closing this issue.

thanks

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

shashankram added the kind/bug Categorizes issue or PR as related to a bug. label Apr 27, 2022

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority labels Apr 27, 2022

k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Apr 28, 2022

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Apr 28, 2022

k8s-ci-robot added triage/needs-information Indicates an issue needs more information in order to work on it. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Apr 29, 2022

nshankar13 mentioned this issue Apr 29, 2022

Temporarily skip ingressbackend e2e for AKS-addon and OSM-Arc Azure/osm-azure#154

Merged

nshankar13 mentioned this issue May 24, 2022

Add root path to ingress in ingress_backend_e2e openservicemesh/osm#4756

Merged

nshankar13 mentioned this issue May 25, 2022

Backport: Add root path ingress e2e test (#4756) openservicemesh/osm#4765

Merged

k8s-ci-robot added kind/support Categorizes issue or PR as a support question. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels May 1, 2024

k8s-ci-robot closed this as completed May 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ingress rule with non-wildcard path does not work #8518

Ingress rule with non-wildcard path does not work #8518

shashankram commented Apr 27, 2022 •

edited

Loading

k8s-ci-robot commented Apr 27, 2022

longwuyuan commented Apr 28, 2022

shashankram commented Apr 28, 2022

shashankram commented Apr 28, 2022

longwuyuan commented Apr 29, 2022

shashankram commented Apr 29, 2022

shashankram commented Apr 29, 2022

longwuyuan commented Apr 29, 2022

ashwinpagarkhed commented May 3, 2022

shashankram commented May 3, 2022 •

edited

Loading

shashankram commented May 3, 2022

nshankar13 commented May 3, 2022

shashankram commented May 5, 2022

kanonkanin commented Jul 28, 2022 •

edited

Loading

longwuyuan commented Jul 28, 2022

kanonkanin commented Jul 28, 2022

longwuyuan commented Oct 11, 2022 via email

keithmattix commented May 31, 2023

longwuyuan commented May 1, 2024

k8s-ci-robot commented May 1, 2024

k8s-ci-robot commented May 1, 2024

Ingress rule with non-wildcard path does not work #8518

Ingress rule with non-wildcard path does not work #8518

Comments

shashankram commented Apr 27, 2022 • edited Loading

Working: path=/

Failing: path=/get

k8s-ci-robot commented Apr 27, 2022

longwuyuan commented Apr 28, 2022

shashankram commented Apr 28, 2022

shashankram commented Apr 28, 2022

longwuyuan commented Apr 29, 2022

shashankram commented Apr 29, 2022

shashankram commented Apr 29, 2022

longwuyuan commented Apr 29, 2022

ashwinpagarkhed commented May 3, 2022

shashankram commented May 3, 2022 • edited Loading

shashankram commented May 3, 2022

nshankar13 commented May 3, 2022

shashankram commented May 5, 2022

kanonkanin commented Jul 28, 2022 • edited Loading

longwuyuan commented Jul 28, 2022

kanonkanin commented Jul 28, 2022

longwuyuan commented Oct 11, 2022 via email

keithmattix commented May 31, 2023

longwuyuan commented May 1, 2024

k8s-ci-robot commented May 1, 2024

k8s-ci-robot commented May 1, 2024

shashankram commented Apr 27, 2022 •

edited

Loading

shashankram commented May 3, 2022 •

edited

Loading

kanonkanin commented Jul 28, 2022 •

edited

Loading