Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ingress-Controller Docker version 0.2.1 is crashing #186

Closed
justinrcs opened this issue Oct 29, 2018 · 4 comments
Closed

Ingress-Controller Docker version 0.2.1 is crashing #186

justinrcs opened this issue Oct 29, 2018 · 4 comments
Assignees
@hbagdi
Labels
bug Something isn't working

Comments

@justinrcs
Copy link

justinrcs commented Oct 29, 2018 •
edited
Loading

Summary

Ingress controller version 0.2.1 is crashing on start-up.

Kong Ingress controller version
Attempted both images:
kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller-amd64:0.2.1
kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.2.1

Kong or Kong Enterprise version
kong: 0.14.1

Kubernetes version

paste kubectl version output

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-16T03:15:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.8", GitCommit:"7eab6a49736cc7b01869a15f9f05dc5b49efb9fc", GitTreeState:"clean", BuildDate:"2018-09-14T15:54:20Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment

  • Cloud provider or hardware configuration:
    AWS / Kops
  • OS (e.g. from /etc/os-release):
    Debian GNU/Linux 8 (jessie)
  • Kernel (e.g. uname -a):
    Linux 4.4.121-k8s Status of controller? #1 SMP Sun Mar 11 19:39:47 UTC 2018 x86_64 GNU/Linux
  • Install tools:
    Kops 1.10

What happened

Upgraded the ingress-controller to version 0.2.1, on re-deploy, the ingress-controller container crashes with the following stack trace:

Kong Ingress controller
Release: 0.2.1
Build: git-3ee649c7
Repository: git@github.com:Kong/kubernetes-ingress-controller.git

I1029 18:35:22.822938 5 flags.go:142] Watching for ingress class: kong
W1029 18:35:22.822975 5 flags.go:145] only Ingress with class "kong" will be processed by this ingress controller
W1029 18:35:22.822982 5 client_config.go:533] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1029 18:35:22.823292 5 main.go:191] Creating API client for ...
I1029 18:35:22.849217 5 main.go:235] Running in Kubernetes Cluster version v1.10 (v1.10.8) - git (clean) commit 7eab6a49736cc7b01869a15f9f05dc5b49efb9fc - platform linux/amd64
I1029 18:35:22.856248 5 main.go:87] validated kong/kong-proxy as the default backend
I1029 18:35:22.861635 5 main.go:138] kong version: 0.14.1
I1029 18:35:22.924072 5 run.go:139] starting Ingress controller
...
I1029 18:35:24.124999 5 leaderelection.go:175] attempting to acquire leader lease kong/ingress-controller-leader-internal-kong...
I1029 18:35:24.132022 5 leaderelection.go:184] successfully acquired lease kong/ingress-controller-leader-internal-kong
I1029 18:35:24.132082 5 status.go:217] new leader elected: kong-ingress-controller-85464d7555-pbm8t
I1029 18:35:27.458787 5 controller.go:128] syncing Ingress configuration...
...
I1029 18:35:27.586007 5 kong.go:117] syncing global plugins
E1029 18:35:27.607682 5 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:522
/usr/local/Cellar/go/1.11/libexec/src/runtime/panic.go:513
/usr/local/Cellar/go/1.11/libexec/src/runtime/panic.go:82
/usr/local/Cellar/go/1.11/libexec/src/runtime/signal_unix.go:390
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/ingress/controller/kong.go:789
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/ingress/controller/kong.go:99
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/ingress/controller/controller.go:129
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/ingress/controller/run.go:80
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/task/queue.go:112
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/task/queue.go:59
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/task/queue.go:59
/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10eab0c]

goroutine 149 [running]:
github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x108
panic(0x12a8540, 0x22269d0)
/usr/local/Cellar/go/1.11/libexec/src/runtime/panic.go:513 +0x1b9
github.com/kong/kubernetes-ingress-controller/internal/ingress/controller.(*NGINXController).syncRoutes(0xc0000ec630, 0xc0008ce2d0, 0x0, 0x0, 0x0)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/ingress/controller/kong.go:789 +0x327c
github.com/kong/kubernetes-ingress-controller/internal/ingress/controller.(*NGINXController).OnUpdate(0xc0000ec630, 0xc0008ce2d0, 0x0, 0x0)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/ingress/controller/kong.go:99 +0x1fb
github.com/kong/kubernetes-ingress-controller/internal/ingress/controller.(*NGINXController).syncIngress(0xc0000ec630, 0x1314200, 0xc0002f8240, 0xc007e01e5c, 0x4ee5126e)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/ingress/controller/controller.go:129 +0x2db
github.com/kong/kubernetes-ingress-controller/internal/ingress/controller.(*NGINXController).syncIngress-fm(0x1314200, 0xc0002f8240, 0xa, 0xc0001c4dd8)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/ingress/controller/run.go:80 +0x3e
github.com/kong/kubernetes-ingress-controller/internal/task.(*Queue).worker(0xc000372c00)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/task/queue.go:112 +0x2de
github.com/kong/kubernetes-ingress-controller/internal/task.(*Queue).worker-fm()
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/task/queue.go:59 +0x2a
github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00017dfa8)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00059dfa8, 0x3b9aca00, 0x0, 0x7f7d4821f901, 0xc000084900)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbe
github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00017dfa8, 0x3b9aca00, 0xc000084900)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
github.com/kong/kubernetes-ingress-controller/internal/task.(*Queue).Run(0xc000372c00, 0x3b9aca00, 0xc000084900)
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/task/queue.go:59 +0x5d
created by github.com/kong/kubernetes-ingress-controller/internal/ingress/controller.(*NGINXController).Start
/Users/harry/go/src/github.com/kong/kubernetes-ingress-controller/internal/ingress/controller/run.go:147 +0xe6

Expected behavior

Ingress controller startup without crash

Steps To Reproduce

  1. Setup kong ingress controller 0.2.0 and all prereqs on any cloud provider
  2. Add a service, route, credential and consumer
  3. Verify it all works
  4. Update the ingress controller to the 0.2.1. container version
  5. Allow the container to reboot
  6. Observe the crash

Update: version 0.1.2 is running as expected, the docs all point to 0.2.x; what is the recommended image to use?
@hbagdi hbagdi self-assigned this Oct 29, 2018
@hbagdi hbagdi added the bug Something isn't working label Oct 29, 2018
hbagdi added a commit that referenced this issue Oct 30, 2018
@hbagdi
fix: avoid nil pointer reference when overriding route with KongIngress
6c049ae 
See: #186
@hbagdi hbagdi mentioned this issue Oct 30, 2018
Merged
@hbagdi
Copy link
Member

hbagdi commented Oct 30, 2018

@justinrcs Thanks for reporting this. I myself ran into this over the weekend. I've opened up #188 to fix it.

hbagdi added a commit that referenced this issue Nov 1, 2018
@hbagdi
fix: avoid nil pointer reference when overriding route with KongIngress
369c10a 
ca2c087 changed the underlying library being
used for interacting with Kong.
It also introduced this bug where a nil reference
was possible.
The code has been refactored
and includes a test.
This is a stop-gap solution
till we have the bigger refactor
in place and make the code
clean and testable.

See: #186
From #188
@miles-
Copy link

miles- commented Nov 5, 2018

I just tried upgrading to 0.2.1 as well and ran into this issue. I've rolled back to 0.1.2 for now.

hbagdi added a commit that referenced this issue Nov 8, 2018
@hbagdi
fix: avoid nil pointer reference when overriding route with KongIngress
41991d9 
ca2c087 changed the underlying library being
used for interacting with Kong.
It also introduced this bug where a nil reference
was possible.
The code has been refactored
and includes a test.
This is a stop-gap solution
till we have the bigger refactor
in place and make the code
clean and testable.

See: #186
From #188
@hbagdi
Copy link
Member

hbagdi commented Nov 11, 2018

@justinrcs @miles This should be fixed in 0.2.2

@justinrcs
Copy link
Author

Confirmed - 0.2.2 working, thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hbagdi hbagdi

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@miles- @hbagdi @justinrcs