Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

keda-add-ons-http-external-scaler fatal error: concurrent map writes #414

Closed
morganchristiansson opened this issue Mar 15, 2022 · 6 comments · Fixed by #415
Closed

keda-add-ons-http-external-scaler fatal error: concurrent map writes #414

morganchristiansson opened this issue Mar 15, 2022 · 6 comments · Fixed by #415
Labels
bug Something isn't working

Comments

@morganchristiansson
Copy link

morganchristiansson commented Mar 15, 2022 •
edited
Loading

Report

My keda-add-ons-http-external-scaler pod is crashlooping with fatal error: concurrent map writes error. See below for full output:

I'll keep digging but thought it's worth reporting. May try reinstall again. Not sure what map it's referring to?

Sorry for long output I can move it to github gist instead? Or delete all except first goroutine trace?

$ kubectl -n keda logs -f keda-add-ons-http-external-scaler-69dbd8d84b-qkrg2
{"level":"info","ts":1647382109.3127644,"caller":"scaler/main.go:144","msg":"starting grpc server","address":"0.0.0.0:9090"}
{"level":"info","ts":1647382109.3127651,"logger":"startHealthcheckServer","caller":"scaler/main.go:216","msg":"starting health check server","addr":"0.0.0.0:9091"}
fatal error: concurrent map writes

goroutine 147 [running]:
runtime.throw({0x180ee20, 0x6000104})
        /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc000325978 sp=0xc000325948 pc=0x433831
runtime.mapassign_faststr(0xc00007e520, 0x145473f, {0xc0005a0250, 0xf})
        /usr/local/go/src/runtime/map_faststr.go:211 +0x39c fp=0xc0003259e0 sp=0xc000325978 pc=0x412a9c
main.mergeCountsWithRoutingTable(0xc000456280, {0x1a068a8, 0xc00007e520})
        /go/src/github.com/kedacore/http-add-on/scaler/merge.go:18 +0xa7 fp=0xc000325a40 sp=0xc0003259e0 pc=0x1454007
main.(*impl).IsActive(0xc0003f9220, {0xc000325b78, 0x203000}, 0xc000184410)
        /go/src/github.com/kedacore/http-add-on/scaler/handlers.go:68 +0x145 fp=0xc000325b20 sp=0xc000325a40 pc=0x14513e5
main.(*impl).StreamIsActive(0xc0003f9220, 0x16f9b20, {0x1a3c230, 0xc00045b190})
        /go/src/github.com/kedacore/http-add-on/scaler/handlers.go:98 +0x14a fp=0xc000325bf0 sp=0xc000325b20 pc=0x145184a
github.com/kedacore/http-add-on/proto._ExternalScaler_StreamIsActive_Handler({0x16a8ac0, 0xc0003f9220}, {0x1a39c98, 0xc0000ea180})
        /go/src/github.com/kedacore/http-add-on/proto/scaler_grpc.pb.go:156 +0xd0 fp=0xc000325c30 sp=0xc000325bf0 pc=0x1420f30
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00045c8c0, {0x1a47a10, 0xc00060b860}, 0xc00048ca20, 0xc0003d49f0, 0x2653de0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:1540 +0xeba fp=0xc000325e48 sp=0xc000325c30 pc=0x141097a
google.golang.org/grpc.(*Server).handleStream(0xc00045c8c0, {0x1a47a10, 0xc00060b860}, 0xc00048ca20, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:1620 +0x9e5 fp=0xc000325f68 sp=0xc000325e48 pc=0x1411e65
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:921 +0x98 fp=0xc000325fe0 sp=0xc000325f68 pc=0x140bbf8
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000325fe8 sp=0xc000325fe0 pc=0x463081
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:919 +0x294

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0000001a0)
        /usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000182240)
        /usr/local/go/src/sync/waitgroup.go:130 +0x71
golang.org/x/sync/errgroup.(*Group).Wait(0xc00016bce0)
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:40 +0x27
main.main()
        /go/src/github.com/kedacore/http-add-on/scaler/main.go:130 +0x8c7

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
        /go/pkg/mod/k8s.io/klog/v2@v2.30.0/klog.go:1181 +0x6a
created by k8s.io/klog/v2.init.0
        /go/pkg/mod/k8s.io/klog/v2@v2.30.0/klog.go:420 +0xfb

goroutine 7 [sleep]:
time.Sleep(0x6fc23ac00)
        /usr/local/go/src/runtime/time.go:193 +0x12e
sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/log/log.go:63 +0x38
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/log/log.go:62 +0x25

goroutine 38 [IO wait]:
internal/poll.runtime_pollWait(0x7f24eb13e538, 0x72)
        /usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc000456680, 0xc000050800, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000456680)
        /usr/local/go/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc000456680)
        /usr/local/go/src/net/fd_unix.go:173 +0x35
net.(*TCPListener).accept(0xc000458c78)
        /usr/local/go/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc000458c78)
        /usr/local/go/src/net/tcpsock.go:262 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc00045c8c0, {0x1a18760, 0xc000458c78})
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:779 +0x362
main.startGrpcServer({0x1a1c038, 0xc00004a440}, {{0x1a39b78, 0xc0004541e0}, 0x1000000010000}, 0x0, 0xc000456280, 0xc00007e520, 0x64, 0xc8)
        /go/src/github.com/kedacore/http-add-on/scaler/main.go:166 +0x36e
main.main.func3()
        /go/src/github.com/kedacore/http-add-on/scaler/main.go:97 +0x73
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x92

goroutine 34 [select]:
net/http.(*persistConn).readLoop(0xc000482900)
        /usr/local/go/src/net/http/transport.go:2207 +0xd8a
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1747 +0x1e05

goroutine 35 [select]:
net/http.(*persistConn).writeLoop(0xc000482900)
        /usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1748 +0x1e65

goroutine 49 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*filteredWatch).loop(0xc000376020)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/watch/filter.go:65 +0xa5
created by k8s.io/apimachinery/pkg/watch.Filter
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/watch/filter.go:42 +0xef

goroutine 37 [chan receive]:
main.(*queuePinger).start(0xc000456280, {0x1a1c038, 0xc00004a440}, 0xc0003f9130)
        /go/src/github.com/kedacore/http-add-on/scaler/queue_pinger.go:77 +0x145
main.main.func2()
        /go/src/github.com/kedacore/http-add-on/scaler/main.go:89 +0x75
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x92

goroutine 36 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc00004a400)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/watch/mux.go:247 +0x49
created by k8s.io/apimachinery/pkg/watch.NewBroadcaster
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/watch/mux.go:73 +0x11b

goroutine 29 [chan receive]:
main.startGrpcServer.func1()
        /go/src/github.com/kedacore/http-add-on/scaler/main.go:163 +0x3b
created by main.startGrpcServer
        /go/src/github.com/kedacore/http-add-on/scaler/main.go:162 +0x35a

goroutine 27 [IO wait]:
internal/poll.runtime_pollWait(0x7f24eb13e620, 0x72)
        /usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc000456500, 0xc00002c000, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000456500, {0xc00002c000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000456500, {0xc00002c000, 0x436387, 0xc000598c30})
        /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00043e300, {0xc00002c000, 0x12, 0xc0004a4b60})
        /usr/local/go/src/net/net.go:183 +0x45
net/http.(*persistConn).Read(0xc0004838c0, {0xc00002c000, 0xc00059c000, 0xc000598d30})
        /usr/local/go/src/net/http/transport.go:1926 +0x4e
bufio.(*Reader).fill(0xc000451860)
        /usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).Peek(0xc000451860, 0x1)
        /usr/local/go/src/bufio/bufio.go:139 +0x5d
net/http.(*persistConn).readLoop(0xc0004838c0)
        /usr/local/go/src/net/http/transport.go:2087 +0x1ac
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1747 +0x1e05

goroutine 28 [select]:
net/http.(*persistConn).writeLoop(0xc0004838c0)
        /usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1748 +0x1e65

goroutine 39 [semacquire]:
sync.runtime_Semacquire(0xc00018c680)
        /usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc00035e0d8)
        /usr/local/go/src/sync/waitgroup.go:130 +0x71
golang.org/x/sync/errgroup.(*Group).Wait(0xc0005a61e0)
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:40 +0x27
github.com/kedacore/http-add-on/pkg/routing.StartConfigMapRoutingTableUpdater({0x1a1c038, 0xc00004a440}, {{0x1a39b78, 0xc0004541e0}, 0x0}, 0xc00016b230, {0xc000042058, 0x4}, 0xc00007e520, 0xc000182198)
        /go/src/github.com/kedacore/http-add-on/pkg/routing/config_map_updater.go:79 +0x3c6
main.main.func4()
        /go/src/github.com/kedacore/http-add-on/scaler/main.go:110 +0x78
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x92

goroutine 40 [IO wait]:
internal/poll.runtime_pollWait(0x7f24eb13e450, 0x72)
        /usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc0003ca100, 0x4, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0003ca100)
        /usr/local/go/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc0003ca100)
        /usr/local/go/src/net/fd_unix.go:173 +0x35
net.(*TCPListener).accept(0xc00000c1e0)
        /usr/local/go/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc00000c1e0)
        /usr/local/go/src/net/tcpsock.go:262 +0x3d
net/http.(*Server).Serve(0xc0004c4000, {0x1a18760, 0xc00000c1e0})
        /usr/local/go/src/net/http/server.go:3001 +0x394
net/http.(*Server).ListenAndServe(0xc0004c4000)
        /usr/local/go/src/net/http/server.go:2930 +0x7d
github.com/kedacore/http-add-on/pkg/http.ServeContext({0x1a1c038, 0xc00004a440}, {0xc00004c880, 0xc}, {0x19ef780, 0xc00004a4c0})
        /go/src/github.com/kedacore/http-add-on/pkg/http/server.go:18 +0x112
main.startAdminServer({0x1a1c038, 0xc00004a440}, {{0x1a39b78, 0xc0004541e0}, 0x0}, 0xc0004500c0, 0x0, 0xc000456280)
        /go/src/github.com/kedacore/http-add-on/scaler/main.go:217 +0x445
main.main.func5()
        /go/src/github.com/kedacore/http-add-on/scaler/main.go:122 +0x65
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x92

goroutine 50 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0002540c8, 0x35)
        /usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0xc0006eb060)
        /usr/local/go/src/sync/cond.go:56 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0002540a0, 0xc0003180c0)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/delta_fifo.go:527 +0x233
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00059e360)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/controller.go:183 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f24eb0b95d0)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x9bf6e8, {0x19eefc0, 0xc0005a6270}, 0x1, 0xc00059c240)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00059e3c8, 0x3b9aca00, 0x0, 0xc0, 0x7f24eb0bc050)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc00059e360, 0xc00059c240)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/controller.go:154 +0x2fb
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0001506e0, 0x0)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/shared_informer.go:414 +0x498
github.com/kedacore/http-add-on/pkg/k8s.(*InformerConfigMapUpdater).Start(0x0, {0x1a1c038, 0xc0003d6140})
        /go/src/github.com/kedacore/http-add-on/pkg/k8s/config_map_cache_informer.go:36 +0x59
github.com/kedacore/http-add-on/pkg/routing.StartConfigMapRoutingTableUpdater.func1()
        /go/src/github.com/kedacore/http-add-on/pkg/routing/config_map_updater.go:42 +0x4e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x92

goroutine 51 [select]:
github.com/kedacore/http-add-on/pkg/routing.StartConfigMapRoutingTableUpdater.func2()
        /go/src/github.com/kedacore/http-add-on/pkg/routing/config_map_updater.go:48 +0x15a
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x92

goroutine 13 [chan receive]:
github.com/kedacore/http-add-on/pkg/http.ServeContext.func1()
        /go/src/github.com/kedacore/http-add-on/pkg/http/server.go:15 +0x3c
created by github.com/kedacore/http-add-on/pkg/http.ServeContext
        /go/src/github.com/kedacore/http-add-on/pkg/http/server.go:14 +0x108

goroutine 53 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0000b8bd0, 0x0)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/shared_informer.go:638 +0x45
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:56 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:71 +0x88

goroutine 54 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/controller.go:130 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/controller.go:129 +0x105

goroutine 65 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/shared_informer.go:782 +0x49
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f24eb0bb9f0)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005ab738, {0x19eefc0, 0xc00061a000}, 0x1, 0xc000618000)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000136200)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/shared_informer.go:781 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:71 +0x88

goroutine 66 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000136200)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/shared_informer.go:752 +0x156
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:71 +0x88

goroutine 55 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0005b6000, {0x0, 0x0, 0x26745a0}, {0x1a09aa8, 0xc00031e400}, 0xc0000bfd18, 0xc000700120, 0xc00059c240)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/reflector.go:469 +0x1b6
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0005b6000, 0xc00059c240)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/reflector.go:429 +0x696
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f24eb0b95d0)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005b2040, {0x19eefa0, 0xc00033c050}, 0x1, 0xc00059c240)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0005b6000, 0xc00059c240)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/reflector.go:220 +0x1f8
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:56 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:71 +0x88

goroutine 30 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/reflector.go:374 +0x12d
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /go/pkg/mod/k8s.io/client-go@v0.23.2/tools/cache/reflector.go:368 +0x378

goroutine 31 [IO wait]:
internal/poll.runtime_pollWait(0x7f24eb13e708, 0x72)
        /usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc000456400, 0xc00033e000, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000456400, {0xc00033e000, 0x3952, 0x3952})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000456400, {0xc00033e000, 0xc0006f2c68, 0xc000028300})
        /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00043e080, {0xc00033e000, 0x17a7920, 0xc000327488})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc0006f8168, {0xc00033e000, 0x0, 0x4099cd})
        /usr/local/go/src/crypto/tls/conn.go:777 +0x3d
bytes.(*Buffer).ReadFrom(0xc000244978, {0x19ec920, 0xc0006f8168})
        /usr/local/go/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc000244700, {0x19ef6e0, 0xc00043e080}, 0xc0006a9080)
        /usr/local/go/src/crypto/tls/conn.go:799 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc000244700, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:606 +0x112
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:574
crypto/tls.(*Conn).Read(0xc000244700, {0xc000122000, 0x1000, 0x16})
        /usr/local/go/src/crypto/tls/conn.go:1277 +0x16f
net/http.(*persistConn).Read(0xc000482900, {0xc000122000, 0x17d8240, 0xc0005cf7c0})
        /usr/local/go/src/net/http/transport.go:1926 +0x4e
bufio.(*Reader).fill(0xc000181740)
        /usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc000181740, 0x5a)
        /usr/local/go/src/bufio/bufio.go:360 +0x2f
net/http/internal.readChunkLine(0x2)
        /usr/local/go/src/net/http/internal/chunked.go:122 +0x25
net/http/internal.(*chunkedReader).beginChunk(0xc0005a7ad0)
        /usr/local/go/src/net/http/internal/chunked.go:48 +0x28
net/http/internal.(*chunkedReader).Read(0xc0005a7ad0, {0xc00023a601, 0x19eee80, 0xc000241960})
        /usr/local/go/src/net/http/internal/chunked.go:93 +0x12e
net/http.(*body).readLocked(0xc00031e380, {0xc00023a601, 0x48, 0x380})
        /usr/local/go/src/net/http/transfer.go:843 +0x3c
net/http.(*body).Read(0xc0006f69c0, {0xc00023a601, 0xc00034d500, 0x2})
        /usr/local/go/src/net/http/transfer.go:835 +0x125
net/http.(*bodyEOFSignal).Read(0xc00031e3c0, {0xc00023a601, 0x5ff, 0x5ff})
        /usr/local/go/src/net/http/transport.go:2768 +0x142
encoding/json.(*Decoder).refill(0xc0001e0f00)
        /usr/local/go/src/encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc0001e0f00)
        /usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc0001e0f00, {0x1601ca0, 0xc0006f8150})
        /usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0005a7b30, {0xc000121c00, 0x400, 0x400})
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000184230, 0xc0006ae700, {0x1a09878, 0xc0006f1080})
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000220ea0)
        /go/pkg/mod/k8s.io/client-go@v0.23.2/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00031e400)
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/watch/streamwatcher.go:105 +0x11c
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /go/pkg/mod/k8s.io/apimachinery@v0.23.2/pkg/watch/streamwatcher.go:76 +0x135

goroutine 61 [IO wait]:
internal/poll.runtime_pollWait(0x7f24eb13e368, 0x72)
        /usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc000136180, 0xc0001f8000, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000136180, {0xc0001f8000, 0x121a, 0x121a})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000136180, {0xc0001f8000, 0xc0001f8005, 0x8d5})
        /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00043e048, {0xc0001f8000, 0x6, 0xc0005967f8})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc000403bf0, {0xc0001f8000, 0x0, 0x4099cd})
        /usr/local/go/src/crypto/tls/conn.go:777 +0x3d
bytes.(*Buffer).ReadFrom(0xc0000a8978, {0x19ec920, 0xc000403bf0})
        /usr/local/go/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc0000a8700, {0x19ef6e0, 0xc00043e048}, 0x121a)
        /usr/local/go/src/crypto/tls/conn.go:799 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc0000a8700, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:606 +0x112
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:574
crypto/tls.(*Conn).Read(0xc0000a8700, {0xc0000e7000, 0x1000, 0x0})
        /usr/local/go/src/crypto/tls/conn.go:1277 +0x16f
net/http.(*persistConn).Read(0xc0005ca240, {0xc0000e7000, 0xc0000463c0, 0xc000596d30})
        /usr/local/go/src/net/http/transport.go:1926 +0x4e
bufio.(*Reader).fill(0xc00031c660)
        /usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).Peek(0xc00031c660, 0x1)
        /usr/local/go/src/bufio/bufio.go:139 +0x5d
net/http.(*persistConn).readLoop(0xc0005ca240)
        /usr/local/go/src/net/http/transport.go:2087 +0x1ac
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1747 +0x1e05

goroutine 62 [select]:
net/http.(*persistConn).writeLoop(0xc0005ca240)
        /usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1748 +0x1e65

goroutine 138 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00009a2d0, 0x1)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000450900)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:326 +0xc6
created by google.golang.org/grpc/internal/transport.NewServerTransport
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:323 +0x180f

goroutine 139 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00060b860)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:1125 +0x259
created by google.golang.org/grpc/internal/transport.NewServerTransport
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:335 +0x1857

goroutine 140 [IO wait]:
internal/poll.runtime_pollWait(0x7f24eb13e280, 0x72)
        /usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc0003ca000, 0xc0006b0000, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0003ca000, {0xc0006b0000, 0x8000, 0x8000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc0003ca000, {0xc0006b0000, 0x60100000000, 0x8})
        /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00043e0e0, {0xc0006b0000, 0x40cc94, 0xc00060ba00})
        /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000700780, {0xc0004c4200, 0x9, 0x101000000000018})
        /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x19ec7c0, 0xc000700780}, {0xc0004c4200, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc0004c4200, 0x9, 0xc0001a3578}, {0x19ec7c0, 0xc000700780})
        /go/pkg/mod/golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0004c41c0)
        /go/pkg/mod/golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00060b860, 0x0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:605 +0xb2
google.golang.org/grpc.(*Server).serveStreams(0xc00045c8c0, {0x1a47a10, 0xc00060b860})
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:905 +0x142
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:847 +0x46
created by google.golang.org/grpc.(*Server).handleRawConn
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:846 +0x185

goroutine 148 [runnable]:
google.golang.org/grpc.(*serverStream).SendMsg.func1()
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/stream.go:1472 +0x2f7
google.golang.org/grpc.(*serverStream).SendMsg(0xc0000ea240, {0x16a4200, 0xc0007eea80})
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/stream.go:1528 +0x482
github.com/kedacore/http-add-on/proto.(*externalScalerStreamIsActiveServer).Send(0xc0003f9220, 0xc000323b78)
        /go/src/github.com/kedacore/http-add-on/proto/scaler_grpc.pb.go:169 +0x2b
main.(*impl).StreamIsActive(0xc0003f9220, 0x16f9b20, {0x1a3c230, 0xc00045b470})
        /go/src/github.com/kedacore/http-add-on/scaler/handlers.go:106 +0x1ca
github.com/kedacore/http-add-on/proto._ExternalScaler_StreamIsActive_Handler({0x16a8ac0, 0xc0003f9220}, {0x1a39c98, 0xc0000ea240})
        /go/src/github.com/kedacore/http-add-on/proto/scaler_grpc.pb.go:156 +0xd0
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00045c8c0, {0x1a47a10, 0xc000583860}, 0xc00048cd80, 0xc0003d49f0, 0x2653de0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:1540 +0xeba
google.golang.org/grpc.(*Server).handleStream(0xc00045c8c0, {0x1a47a10, 0xc000583860}, 0xc00048cd80, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:1620 +0x9e5
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:921 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:919 +0x294

goroutine 144 [runnable]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00009a500, 0x1)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001808a0)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:326 +0xc6
created by google.golang.org/grpc/internal/transport.NewServerTransport
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:323 +0x180f

goroutine 177 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000583860)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:1125 +0x259
created by google.golang.org/grpc/internal/transport.NewServerTransport
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:335 +0x1857

goroutine 178 [IO wait]:
internal/poll.runtime_pollWait(0x7f24eb13e198, 0x72)
        /usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc000136100, 0xc0006c8000, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000136100, {0xc0006c8000, 0x8000, 0x8000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000136100, {0xc0006c8000, 0x60100000000, 0x8})
        /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000702090, {0xc0006c8000, 0x40cc94, 0xc000583a00})
        /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000700cc0, {0xc0004c42e0, 0x9, 0x101000000000018})
        /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x19ec7c0, 0xc000700cc0}, {0xc0004c42e0, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc0004c42e0, 0x9, 0xc000049758}, {0x19ec7c0, 0xc000700cc0})
        /go/pkg/mod/golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0004c42a0)
        /go/pkg/mod/golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000583860, 0x0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/internal/transport/http2_server.go:605 +0xb2
google.golang.org/grpc.(*Server).serveStreams(0xc00045c8c0, {0x1a47a10, 0xc000583860})
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:905 +0x142
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:847 +0x46
created by google.golang.org/grpc.(*Server).handleRawConn
        /go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:846 +0x185

Expected Behavior

Not crash

Actual Behavior

keda-add-ons-http-external-scaler is Crashlooping

Steps to Reproduce the Problem

Installed latest keda and http-add-on charts. No relevant custom values. Added some HTTPScaledObject resources. I've just switched to arm64 docker images for keda - but http-external-scaler was already crashlooping.

Logs from KEDA HTTP operator

No errors from keda http operator - only info and debug.

$ kubectl -n keda logs keda-add-ons-http-controller-manager-5657db987f-jwm2j -c keda-add-ons-http-operator

$ kubectl -n keda logs -f keda-operator-795b4d499c-mwvxb
<snip>
1.6473825983627646e+09  ERROR   external_scaler error calling IsActive on external scaler       {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.43.9.159:9090: connect: connection refused\""}
<snip>
1.647382598372701e+09   ERROR   scalehandler    Error getting scale decision    {"scaledobject.Name": "keda-add-ons-http-interceptor", "scaledObject.Namespace": "keda", "scaleTarget.Name": "keda-add-ons-http-interceptor", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.43.9.159:9090: connect: connection refused\""}
<snip>
1.6473826192677171e+09  ERROR   external_scaler error calling IsActive on external scaler       {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.43.9.159:9090: i/o timeout\""}

What version of the KEDA HTTP Addon are you running?

0.3.0

Kubernetes Version

1.21

Platform

Other

Anything else?

No response
@morganchristiansson morganchristiansson added the bug Something isn't working label Mar 15, 2022
@arschles
Copy link
Collaborator

thanks for this @morganchristiansson, I'll look into this as well. the trace helps greatly

@stefanbulzan
Copy link

I encountered the same issue. I'm looking forward to see this issue fixed as it makes the addon unreliable. Thank you!

@arschles arschles mentioned this issue Mar 18, 2022
Merged
2 tasks
@arschles
Copy link
Collaborator

@morganchristiansson @stefanbulzan #415 should fix this.

@stefanbulzan
Copy link

@arschles I tested it with a locally build image from the fix branch. Looks good, doesn't fail anymore and scaling works. Brilliant! Thank you. I'm looking forward to have this in 0.4.0 hopefully.

@arschles
Copy link
Collaborator

arschles commented Mar 22, 2022 •
edited
Loading

Yup, this will be in 0.4.0 for sure 👍

Thanks for testing it 😄

@arschles
Copy link
Collaborator

@stefanbulzan @morganchristiansson now that #415 is merged, you should see the fix in the canary images as well as the images tagged with sha-2d35d0d after they are done building (within a few minutes from now)

@arschles arschles mentioned this issue Mar 24, 2022
Closed
@JorTurFer JorTurFer mentioned this issue Jan 24, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixing concurrent map access panic in scaler arschles/http-add-on
3 participants
@morganchristiansson @arschles @stefanbulzan