Nginx ingress - all of css js and images 404

[zhuqinghua]

I can access it via http, but only the html is accessible, all of its css js and images 404.

ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: vsk8s.io
    http:
      paths:
      - path: /dashboard
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 9090
      - path: /grafana
        backend:
          serviceName: monitoring-grafana
          servicePort: 3000

/etc/nginx/nginx.conf

daemon off;

worker_processes 4;
pid /run/nginx.pid;

worker_rlimit_nofile 15360;
events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    real_ip_header      X-Forwarded-For;
    set_real_ip_from    0.0.0.0/0;
    real_ip_recursive   on;

    geoip_country       /etc/nginx/GeoIP.dat;
    geoip_city          /etc/nginx/GeoLiteCity.dat;
    geoip_proxy_recursive on;
    # lua section to return proper error codes when custom pages are used
    lua_package_path '.?.lua;./etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/lua-resty-http/lib/?.lua;';
    init_by_lua_block {
        require("error_page")
    }

    sendfile            on;
    aio                 threads;
    tcp_nopush          on;
    tcp_nodelay         on;
    log_subrequest      on;

    reset_timedout_connection on;

    keepalive_timeout 75s;

    client_header_buffer_size       1k;
    large_client_header_buffers     4 8k;

    types_hash_max_size             2048;
    server_names_hash_max_size      512;
    server_names_hash_bucket_size   64;
    map_hash_bucket_size            64;

    include /etc/nginx/mime.types;
    default_type text/html;
    gzip on;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
    gzip_proxied any;

    server_tokens on;

    log_format upstreaminfo '$remote_addr - '
        '[$proxy_add_x_forwarded_for] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" '
        '$request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';

    map $request_uri $loggable {
        default 1;
    }

    access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
    error_log  /var/log/nginx/error.log notice;

    resolver 10.68.7.142 223.5.5.5 valid=30s;

    # Retain the default nginx handling of requests without a "Connection" header
    map $http_upgrade $connection_upgrade {
        default          upgrade;
        ''               close;
    }

    # trust http_x_forwarded_proto headers correctly indicate ssl offloading
    map $http_x_forwarded_proto $pass_access_scheme {
        default          $http_x_forwarded_proto;
        ''               $scheme;
    }

    map $http_x_forwarded_port $pass_server_port {
       default           $http_x_forwarded_port;
       ''                $server_port;
    }

    # map port 442 to 443 for header X-Forwarded-Port
    map $pass_server_port $pass_port {
        442              443;
        default          $pass_server_port;
    }

    # Map a response error watching the header Content-Type
    map $http_accept $httpAccept {
        default          html;
        application/json json;
        application/xml  xml;
        text/plain       text;
    }

    map $httpAccept $httpReturnType {
        default          text/html;
        json             application/json;
        xml              application/xml;
        text             text/plain;
    }

    server_name_in_redirect off;
    port_in_redirect        off;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    # turn on session caching to drastically improve performance
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 10m;

    # allow configuring ssl session tickets
    ssl_session_tickets on;

    # slightly reduce the time-to-first-byte
    ssl_buffer_size 4k;

    # allow configuring custom ssl ciphers
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;

    # In case of errors try the next upstream server before returning an error
    proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

    upstream kube-system-kubernetes-dashboard-9090 {
        least_conn;
        server 10.32.0.1:9090 max_fails=0 fail_timeout=0;
    }
    upstream kube-system-monitoring-grafana-3000 {
        least_conn;
        server 10.40.0.71:3000 max_fails=0 fail_timeout=0;
    }
    upstream upstream-default-backend {
        least_conn;
        server 10.40.0.83:8080 max_fails=0 fail_timeout=0;
    }
    server {
        server_name _;
        listen [::]:80 ipv6only=off default_server reuseport backlog=511;
        listen 442  default_server reuseport backlog=511 ssl http2;
        # PEM sha: 146d6a1a176900b4af00faf1b177c82d6c7b39ae
        ssl_certificate                         /ingress-controller/ssl/system-snake-oil-certificate.pem;
        ssl_certificate_key                     /ingress-controller/ssl/system-snake-oil-certificate.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains; preload";
        location / {
            set $proxy_upstream_name "upstream-default-backend";

            port_in_redirect off;
            client_max_body_size                    "1m";

            proxy_set_header Host                   $host;

            # Pass Real IP
            proxy_set_header X-Real-IP              $remote_addr;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";

            proxy_http_version                      1.1;
            proxy_pass http://upstream-default-backend;
        }
        # health checks in cloud providers require the use of port 80
        location /healthz {
            access_log off;
            return 200;
        }

        # this is required to avoid error if nginx is being monitored
        # with an external software (like sysdig)
        location /nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }
    }

    server {
        server_name vsk8s.io;
        listen [::]:80;

        location ~* ^/grafana {
            set $proxy_upstream_name "kube-system-monitoring-grafana-3000";

            port_in_redirect off;
            client_max_body_size                    "1m";

            proxy_set_header Host                   $host;

            # Pass Real IP
            proxy_set_header X-Real-IP              $remote_addr;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";

            proxy_http_version                      1.1;
        rewrite /grafana/(.*) /$1 break;
        rewrite /grafana / break;
        proxy_pass http://kube-system-monitoring-grafana-3000;

        }
        location ~* ^/dashboard {
            set $proxy_upstream_name "kube-system-kubernetes-dashboard-9090";

            port_in_redirect off;
            client_max_body_size                    "1m";

            proxy_set_header Host                   $host;

            # Pass Real IP
            proxy_set_header X-Real-IP              $remote_addr;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";

            proxy_http_version                      1.1;
        rewrite /dashboard/(.*) /$1 break;
        rewrite /dashboard / break;
        proxy_pass http://kube-system-kubernetes-dashboard-9090;

        }
        location / {
            set $proxy_upstream_name "upstream-default-backend";

            port_in_redirect off;
            client_max_body_size                    "1m";

            proxy_set_header Host                   $host;

            # Pass Real IP
            proxy_set_header X-Real-IP              $remote_addr;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";

            proxy_http_version                      1.1;
            proxy_pass http://upstream-default-backend;
        }
    }

    # default server, used for NGINX healthcheck and access to nginx stats
    server {
        # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
        # Changing this value requires a change in:
        # https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/nginx/command.go#L104
        listen [::]:18080 ipv6only=off default_server reuseport backlog=511;

        location /healthz {
            access_log off;
            return 200;
        }
        location /nginx_status {
            access_log off;
            stub_status on;
        }

        # this location is used to extract nginx metrics
        # using prometheus.
        # TODO: enable extraction for vts module.
        location /internal_nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }

        location / {
            set $proxy_upstream_name "upstream-default-backend";
            proxy_pass             http://upstream-default-backend;
        }

    }

    # default server for services without endpoints
    server {
        listen 8181;
        set $proxy_upstream_name "-";

        location / {
            return 503;
        }
    }
}

stream {
    # map FQDN that requires SSL passthrough
    map $ssl_preread_server_name $stream_upstream {
        # send SSL traffic to this nginx in a different port
        default                         nginx-ssl-backend;
    }

    log_format log_stream '$remote_addr [$time_local] $protocol [$ssl_preread_server_name] [$stream_upstream] $status $bytes_sent $bytes_received $session_time';

    access_log /var/log/nginx/access.log log_stream;

    error_log  /var/log/nginx/error.log;

    # configure default backend for SSL
    upstream nginx-ssl-backend {
        server 127.0.0.1:442;
    }
    server {
        listen                  [::]:443 ipv6only=off;
        proxy_pass              $stream_upstream;
        ssl_preread             on;
    }

    # TCP services

    # UDP services
}

anyone can give me some suggestions? thank you

[foxylion]

Did you have a look on the requested URLs (network tab of your browser)? Do they match /dashboard/... or /grafana/...?
If not this may be a problem on the backend side. This is because "nginx rewrite" does not rewrite the html source, only location headers.

[zhuqinghua]

@foxylion
thank you for your help, you are right,the “rewirte” does not work on resources.
i have no idea to solve it now.
does Ingress has any ‘annotations‘’ options for this?

here is nginx-ingress-controller's log:

2017-02-27T04:24:51.987197813Z ::ffff:10.68.7.144 - [::ffff:10.68.7.144] - - [27/Feb/2017:04:24:51 +0000] "GET /grafana HTTP/1.1" 200 1937 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 460 0.004 [kube-system-monitoring-grafana-3000] 10.40.0.71:3000 7519 0.004 200
2017-02-27T04:24:52.012454927Z ::ffff:10.68.7.144 - [::ffff:10.68.7.144] - - [27/Feb/2017:04:24:52 +0000] "GET /public/css/fonts.min.css HTTP/1.1" 404 21 "http://vsk8s.io/grafana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 425 0.001 [upstream-default-backend] 10.40.0.83:8080 21 0.001 404
2017-02-27T04:24:52.015350951Z ::ffff:10.68.7.144 - [::ffff:10.68.7.144] - - [27/Feb/2017:04:24:52 +0000] "GET /public/css/grafana.dark.min.4a9eed6d.css HTTP/1.1" 404 21 "http://vsk8s.io/grafana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 441 0.001 [upstream-default-backend] 10.40.0.83:8080 21 0.001 404
2017-02-27T04:24:52.018171980Z ::ffff:10.68.7.144 - [::ffff:10.68.7.144] - - [27/Feb/2017:04:24:52 +0000] "GET /public/app/boot.1649cbaa.js HTTP/1.1" 404 21 "http://vsk8s.io/grafana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 413 0.000 [upstream-default-backend] 10.40.0.83:8080 21 0.000 404
2017-02-27T04:24:52.320368402Z ::ffff:10.68.7.144 - [::ffff:10.68.7.144] - - [27/Feb/2017:04:24:52 +0000] "GET /public/app/boot.1649cbaa.js HTTP/1.1" 404 21 "http://vsk8s.io/grafana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 413 0.001 [upstream-default-backend] 10.40.0.83:8080 21 0.001 404
2017-02-27T04:24:52.373192694Z ::ffff:10.68.7.144 - [::ffff:10.68.7.144] - - [27/Feb/2017:04:24:52 +0000] "GET /index.js.map HTTP/1.1" 404 21 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 333 0.000 [upstream-default-backend] 10.40.0.83:8080 21 0.000 404

The above paragraph
"GET /public/app/boot.1649cbaa.js HTTP/1.1" 404 21 "http://vsk8s.io/grafana"
should be written like this
"GET /grafana/public/app/boot.1649cbaa.js HTTP/1.1" 200 1937 "-"

[foxylion]

Grafana for example does have a configuration property to change the path: http://docs.grafana.org/installation/configuration/#static-root-path

[aledbf]

Closing. Please follow the advice from foxylion and reopen if you still have issues

[nikitazernov]

@zhuqinghua I'm wondering what did you set in static root path or server root in Grafana. I cannot make it work on Kubernetes using nginx-ingress too... Have only this:

[nemodev]

@nikitazernov I had a same issue too. In my case, I solved with below settings.

Create a values.yaml for helm install.

values.yaml

grafana.ini:
  server:
    root_url: http://demo.bruce.com/grafana/

Change root_url value properly. It should include subpath if any. I put 'grafana' in my example.

And install grafana with the values.yaml as below.
helm install --name grafana --namespace monitoring stable/grafana -f values.yaml

If you want to know all available settings, refer helm grafana chart / values.yaml and grafana.ini configuration

Create an ingress resource as below.

ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: ingress-grafana
  namespace: monitoring
spec:
  rules:
  - host: demo.bruce.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 80
        path: /grafana
status:
  loadBalancer: {}

I set rewrite-target as root / and gave sub path as /grafana.

After grafana pod is running you should be able to connect to it successfully.

[arjunkrishnasb]

Hi, I am having very similar issue. I am trying to visualize my mongodb data using mongo express.

• If i expose mongo-express using nodeport service it is working fine.
• But if i expose it using ingress, it is not loading the css, img, js files and giving 404.

Any help would be appreciated..

`apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: baster-api-ingress
namespace: {{ .Values.namespace }}
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:

• http:
  paths:
  • path: /baster/mongo-express
    backend:
    serviceName: {{ .Values.mongo_express.service_name }}
    servicePort: 8081
  • path: /(.*)
    backend:
    serviceName: {{ .Values.baster_api.service_name }}
    servicePort: 5000`

[aledbf]

But if i expose it using ingress, it is not loading the css, img, js files and giving 404.

@Arjunkrisha that is expected. The rewrite-target annotation changes the paths from the request to the mongo-express application, not the paths returned by the application.
You should change the ME_CONFIG_SITE_BASEURL environment variable to /baster/mongo-express and remove the rewrite-target annotation.

I suggest you remove the rewrite-target annotation after changing the env variable.

https://github.com/mongo-express/mongo-express#usage-docker

[arjunkrishnasb]

Hey @aledbf
Thanks a lot.. That worked!
I was trying to include two both my application endpoint and mongo express in the same ingress resource. But now it makes sense to split them, because one needs rewrite while the other does not.
Like you suggested, i updated the base url for mongo express.
I am pretty sure many would encounter similar issues. So like how mongo-express is configurable with an env, hopefully other tools should have something similar.

Now my config looks like:
mongo-express-ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: mongo-express-ingress
  namespace: {{ .Values.namespace }}
spec:
  rules:
  - http:
      paths:
      - path: {{ .Values.mongo_express.path }}
        backend:
          serviceName: {{ .Values.mongo_express.service_name }}
          servicePort: 8081

baster-api-ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: baster-api-ingress
  namespace: {{ .Values.namespace }}
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - http:
      paths:
      - path: /(.*)
        backend:
          serviceName: {{ .Values.baster_api.service_name }}
          servicePort: 5000

[sanzenwin]

https://aws.amazon.com/premiumsupport/knowledge-center/eks-kubernetes-dashboard-custom-path/#Create_an_Ingress_for_the_NGINX_Ingress_Controller

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^(/dashboard)$ $1/ redirect;
  namespace: kubernetes-dashboard
spec:
  rules:
  - http:
      paths:
      - path: /dashboard(/|$)(.*)
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 443

I think this is the best way, and it works for me

[arnab-code]

@sanzenwin Thanks a lot mate.... I have been searching this kind of a solution for the whole day... It works perfectly and as I want it.

[MuhammadNaeemAkhtar]

status:
  loadBalancer: {}

Hi @nemodev ,
I tried exactly the same solution as you said, but when I go to the http:///grafana it redirects to http:///grafana/login and show the message that the page is not reachable.

[RainyH2O]

@sanzenwin I met the problem and solved it. IMHO the key point is the difference between URL rewriting and URL redirection:

If a client has link-editing capabilities, it should update all references to the Request URL. -via: Wikipedia

Therefore the annotaion could solve these problems.

    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^(/dashboard)$ $1/ redirect; # Attention: the trailing slash of "$1/" prevent redirect looping.