Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specifying wrong ssl-dh-param secret name leads to bad memory access #3854

Closed
residentsummer opened this issue Mar 5, 2019 · 0 comments · Fixed by #3856
Closed

Specifying wrong ssl-dh-param secret name leads to bad memory access #3854

residentsummer opened this issue Mar 5, 2019 · 0 comments · Fixed by #3856

Comments

@residentsummer
Copy link

Is this a request for help?:
No

What keywords did you search in NGINX Ingress controller issues before filing this one?:
ssl-dh-param


Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

NGINX Ingress controller version:
0.22.0 (chart nginx-ingress-1.3.1)

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-08T16:31:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.5", GitCommit:"51dd616cdd25d6ee22c83a858773b607328a18ec", GitTreeState:"clean", BuildDate:"2019-01-16T18:14:49Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: baremetal
  • OS (e.g. from /etc/os-release): CoreOS
  • Kernel (e.g. uname -a): 4.14.67-coreos
  • Install tools: kubespray/helm
  • Others:

What happened:

  • nginx-ingress was working great with stock dhparam
  • I've set a value (that's the only change in configmap below) to use custom dhparam.pem from a secret. Specified it without a namespace to check if it'll pick up the secret from own NS (nginx-global in my case).
$ cat values.yml
controller:
  daemonset:
    useHostPort: true
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  kind: DaemonSet
  service:
    type: ClusterIP
  stats:
    enabled: true
  metrics:
    enabled: true
    # serviceMonitor:
    #   enabled: true
  config:
    # name of the secret with dhparam.pem
    ssl-dh-param: "ssl-dh-param"
    server-tokens: "false"
serviceAccount:
  create: true
$ helm upgrade -f values.yml --version 1.3.1 nginx-global stable/nginx-ingress
Release "nginx-global" has been upgraded. Happy Helming!
-- cut --
  • controller pods entered CrashLoopBackoff
NAME                                                          READY     STATUS             RESTARTS   AGE
nginx-global-nginx-ingress-controller-cvfj2                   0/1       CrashLoopBackOff   3          5d
nginx-global-nginx-ingress-controller-qt5nq                   0/1       CrashLoopBackOff   3          5d
$ kc logs -n nginx-global nginx-global-nginx-ingress-controller-cvfj2
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.22.0
  Build:      git-f7c42b78a
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

I0305 13:51:05.365211       9 flags.go:183] Watching for Ingress class: nginx
W0305 13:51:05.365697       9 flags.go:216] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.8
W0305 13:51:05.367890       9 client_config.go:548] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0305 13:51:05.368111       9 main.go:200] Creating API client for https://10.233.0.1:443
I0305 13:51:05.382874       9 main.go:244] Running in Kubernetes cluster version v1.12 (v1.12.5) - git (clean) commit 51dd616cdd25d6ee22c83a858773b607328a18ec - platform linux/amd64
I0305 13:51:05.412599       9 main.go:102] Validated nginx-global/nginx-global-nginx-ingress-default-backend as the default backend.
I0305 13:51:05.611083       9 nginx.go:267] Starting NGINX Ingress controller
I0305 13:51:05.620735       9 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"nginx-global", Name:"nginx-global-nginx-ingress-controller", UID:"a7c684a3-35c9-11e9-8a1a-14dae9ef70a4", APIVersion:"v1", ResourceVersion:"56807704", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap nginx-global/nginx-global-nginx-ingress-controller
I0305 13:51:05.624003       9 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"nginx-global", Name:"nginx-global-nginx-ingress-tcp", UID:"a7d2ee63-35c9-11e9-8a1a-14dae9ef70a4", APIVersion:"v1", ResourceVersion:"54890783", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap nginx-global/nginx-global-nginx-ingress-tcp
I0305 13:51:06.715440       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"mcj", UID:"9a4bb588-3b9c-11e9-b442-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"56038630", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress production/mcj
I0305 13:51:06.716013       9 backend_ssl.go:68] Adding Secret "production/clickavia-ru" to the local store
I0305 13:51:06.716132       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"mcx", UID:"9c19b019-3b9c-11e9-b442-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"56038631", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress production/mcx
I0305 13:51:06.731915       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"core-api", UID:"40c06b40-3f47-11e9-b442-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"56802681", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress production/core-api
I0305 13:51:06.732757       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"mc2j", UID:"076d85da-3141-11e9-b5e2-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"56037204", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress production/mc2j
I0305 13:51:06.733471       9 backend_ssl.go:68] Adding Secret "production/clickavia-su" to the local store
I0305 13:51:06.733615       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"front", Name:"new", UID:"9c62a4a9-3b68-11e9-b442-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"55996127", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress front/new
I0305 13:51:06.733983       9 backend_ssl.go:68] Adding Secret "front/clickavia-ru" to the local store
I0305 13:51:06.734146       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"afb", UID:"142fea8f-3c3a-11e9-b442-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"56168012", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress production/afb
I0305 13:51:06.734618       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"apb", UID:"15805a7d-3c3a-11e9-b442-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"56168011", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress production/apb
I0305 13:51:06.735101       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"mc2j-slow", UID:"82a7b7df-3eb0-11e9-b442-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"56680304", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress production/mc2j-slow
I0305 13:51:06.735589       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"internal-services-ingress", UID:"e98beb29-705a-11e8-a3d8-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"55984892", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/internal-services-ingress
I0305 13:51:06.736034       9 backend_ssl.go:68] Adding Secret "default/internal-services-certs" to the local store
I0305 13:51:06.736177       9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"front", Name:"co", UID:"9b11079f-3b67-11e9-b442-5404a6b213ef", APIVersion:"extensions/v1beta1", ResourceVersion:"55996005", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress front/co
I0305 13:51:06.812570       9 nginx.go:288] Starting NGINX process
I0305 13:51:06.812585       9 leaderelection.go:205] attempting to acquire leader lease  nginx-global/ingress-controller-leader-nginx...
I0305 13:51:06.813036       9 controller.go:172] Configuration changes detected, backend reload required.
W0305 13:51:06.821891       9 nginx.go:571] Error reading Secret "ssl-dh-param" from local store: no object matching key "ssl-dh-param" in local store
E0305 13:51:06.821971       9 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/go/src/k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:576
/go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:180
/go/src/k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:135
/go/src/k8s.io/ingress-nginx/internal/task/queue.go:129
/go/src/k8s.io/ingress-nginx/internal/task/queue.go:61
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/go/src/k8s.io/ingress-nginx/internal/task/queue.go:61
/usr/local/go/src/runtime/asm_amd64.s:1333
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x108 pc=0x11e8415]

goroutine 211 [running]:
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x108
panic(0x13869e0, 0x22fc300)
        /usr/local/go/src/runtime/panic.go:513 +0x1b9
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).OnUpdate(0xc0003a1520, 0xc0007f6de0, 0xb, 0xb, 0xc0007be7e0, 0x5, 0x5, 0xc0002fa700, 0x2, 0x2, ...)
        /go/src/k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:576 +0xe95
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress(0xc0003a1520, 0x1418080, 0xc00096c2e0, 0xc0307365ed, 0x56b4bc6f)
        /go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:180 +0xde5
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress-fm(0x1418080, 0xc00096c2e0, 0xa, 0xc0000c0dd0)
        /go/src/k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:135 +0x3e
k8s.io/ingress-nginx/internal/task.(*Queue).worker(0xc00042dfb0)
        /go/src/k8s.io/ingress-nginx/internal/task/queue.go:129 +0x2de
k8s.io/ingress-nginx/internal/task.(*Queue).worker-fm()
        /go/src/k8s.io/ingress-nginx/internal/task/queue.go:61 +0x2a
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00070cfa8)
        /go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009b3fa8, 0x3b9aca00, 0x0, 0x1, 0xc0000465a0)
        /go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbe
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00070cfa8, 0x3b9aca00, 0xc0000465a0)
        /go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
k8s.io/ingress-nginx/internal/task.(*Queue).Run(0xc00042dfb0, 0x3b9aca00, 0xc0000465a0)
        /go/src/k8s.io/ingress-nginx/internal/task/queue.go:61 +0x5d
created by k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).Start
        /go/src/k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:291 +0x1e4

What you expected to happen:
IDK, it probably should crash on such config issues, but definetely not with bad memory access.

How to reproduce it (as minimally and precisely as possible):
Please see What happened section.

Anything else we need to know:
Second configuration snippet in README is incorrect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant