Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upgrading from 0.10 to 0.13 - "concurrent map read and map write" crashes kustomize-controller #341

Closed
caddac opened this issue May 5, 2021 · 1 comment · Fixed by #343
Closed

upgrading from 0.10 to 0.13 - "concurrent map read and map write" crashes kustomize-controller #341

caddac opened this issue May 5, 2021 · 1 comment · Fixed by #343

Comments

@caddac
Copy link

caddac commented May 5, 2021

Describe the bug

During the upgrade from 0.10 to 0.13 I experienced the kustomize-controller crash with a stack trace and the error fatal error: concurrent map read and map write. The controller restarts and successfully reconciles as expected, but only after re-bootstrapping a second time.

This bug does not prevent an upgrade and I am not blocked, but I wanted to report it as there might be an issue in the controller that could pop up later.

To Reproduce

Steps to reproduce the behaviour:

  1. I followed this guide: Upgrade image automation to v1alpha2 APIs flux2#1333
    The issue was intermittent and only occurred on 2 of my 5 clusters.

Expected behavior

The kustomize-controller shouldn't crash due to this error

Additional context

  • Kubernetes version: GKE 1.17.17-gke.3700
  • Git provider: go-git

Below please provide the output of the following commands:

flux --version
  flux version 0.13.2
flux check
► checking prerequisites
✔ kubectl 1.20.2 >=1.18.0-0
✔ Kubernetes 1.17.17-gke.3700 >=1.16.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.10.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.9.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.9.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.12.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.13.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.12.1
✔ all checks passed

Here are the kustomize-controller logs captured from the crash pod:

{"level":"info","ts":"2021-05-05T15:44:16.617Z","logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2021-05-05T15:44:16.618Z","logger":"setup","msg":"starting manager"}
I0505 15:44:16.618591       7 leaderelection.go:243] attempting to acquire leader lease flux-system/kustomize-controller-leader-election...
{"level":"info","ts":"2021-05-05T15:44:16.619Z","msg":"starting metrics server","path":"/metrics"}
I0505 15:44:16.645125       7 leaderelection.go:253] successfully acquired lease flux-system/kustomize-controller-leader-election
{"level":"info","ts":"2021-05-05T15:44:16.719Z","logger":"controller.kustomization","msg":"Starting EventSource","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-05-05T15:44:16.719Z","logger":"controller.kustomization","msg":"Starting EventSource","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-05-05T15:44:16.820Z","logger":"controller.kustomization","msg":"Starting EventSource","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-05-05T15:44:16.920Z","logger":"controller.kustomization","msg":"Starting Controller","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization"}
{"level":"info","ts":"2021-05-05T15:44:16.920Z","logger":"controller.kustomization","msg":"Starting workers","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","worker count":4}
fatal error: concurrent map read and map write
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/log.go:63 +0x45
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/log.go:62 +0x35
goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0002b2dc0)
    /go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
    /go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:32 +0x57
goroutine 275 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc00033cd80)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:742 +0x157
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0003bfc60, 0xc000bd1960)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:71 +0x65
goroutine 59 [IO wait]:
internal/poll.runtime_pollWait(0x7fc452ed4718, 0x72, 0x23cff20)
    /usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000182198, 0x72, 0x23cff00, 0x30bf7a0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000182180, 0xc000850000, 0x26fa, 0x26fa, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000182180, 0xc000850000, 0x26fa, 0x26fa, 0x203000, 0x6fb87b, 0xc000764f60)
    /usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0004aa248, 0xc000850000, 0x26fa, 0x26fa, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:182 +0x8e
crypto/tls.(*atLeastReader).Read(0xc001134840, 0xc000850000, 0x26fa, 0x26fa, 0x274, 0x266f, 0xc00001d708)
    /usr/local/go/src/crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc000765080, 0x23c91e0, 0xc001134840, 0x40b7a5, 0x1e57600, 0x20416c0)
    /usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc000764e00, 0x23ccc80, 0xc0004aa248, 0x5, 0xc0004aa248, 0x263)
    /usr/local/go/src/crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000764e00, 0x0, 0x0, 0xc0009e84e0)
    /usr/local/go/src/crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(...)
    /usr/local/go/src/crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc000764e00, 0xc000381000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1252 +0x15f
bufio.(*Reader).Read(0xc0007fed80, 0xc0007723b8, 0x9, 0x9, 0xc0051765a0, 0xc00001dd18, 0x21b5c00)
    /usr/local/go/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x23c8fa0, 0xc0007fed80, 0xc0007723b8, 0x9, 0x9, 0x9, 0xc0011543f0, 0xc000102050, 0x0)
    /usr/local/go/src/io/io.go:314 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:333
golang.org/x/net/http2.readFrameHeader(0xc0007723b8, 0x9, 0x9, 0x23c8fa0, 0xc0007fed80, 0x0, 0x0, 0xc000a1b5c0, 0xc00001ddd0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000772380, 0xc000a1b5c0, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/frame.go:492 +0xa5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00001dfa8, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/transport.go:1819 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00028a900)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/transport.go:1741 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/transport.go:705 +0x6c5
goroutine 292 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc00070a8a0, 0xc00065c2d0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/controller.go:129 +0xa5
goroutine 271 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start(0xc000acc460, 0x24232c0, 0xc000ae30c0, 0x0, 0x0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:228 +0x153
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1(0xc000ae0000, 0x7fc452cfbdf8, 0xc000acc460)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/internal.go:681 +0x7e
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/internal.go:679 +0x73
goroutine 301 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000b1b910, 0x0)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000b1b900)
    /usr/local/go/src/sync/cond.go:56 +0x9d
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000b3aba0, 0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/util/workqueue/queue.go:145 +0x89
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000acc460, 0x24232c0, 0xc000ae30c0, 0x0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:236 +0x68
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x24232c0, 0xc000ae30c0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000542f50)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000542f50, 0x23cc720, 0xc0009ec5a0, 0xc000ae3001, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000542f50, 0x3b9aca00, 0x0, 0xc00000a901, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x24232c0, 0xc000ae30c0, 0xc0003c3a50, 0x3b9aca00, 0x0, 0x100000000000001)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x24232c0, 0xc000ae30c0, 0xc0003c3a50, 0x3b9aca00)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:213 +0x45e
goroutine 286 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000b73400)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:742 +0x157
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00075a140, 0xc0003c3380)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:73 +0x51
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0004425b0)
    /usr/local/go/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc0004425a8, 0xc000b41e00, 0x200, 0x200, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc000442580, 0xc000b41e00, 0x200, 0x200, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/transport.go:2108 +0xaf
encoding/json.(*Decoder).refill(0xc0002b42c0, 0x18, 0x7fc479d02108)
    /usr/local/go/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc0002b42c0, 0x0, 0x0, 0x1e04d60)
    /usr/local/go/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc0002b42c0, 0x1e3c540, 0xc00022ec60, 0xc00006f000, 0x0)
    /usr/local/go/src/encoding/json/stream.go:63 +0x79
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000a1bd70, 0xc000480800, 0x400, 0x400, 0xc0003c36c0, 0x7fc4530a1430, 0x38)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000a2b720, 0x0, 0x23df740, 0xc000356c40, 0x0, 0xa, 0x2, 0x100000000000000, 0x0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00022ec20, 0xc00039a790, 0xc0001caeb8, 0xc00039a780, 0x12e35cb, 0xc0004cff40, 0xc000413160)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000356b40)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/watch/streamwatcher.go:104 +0x14a
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/watch/streamwatcher.go:71 +0xbe
goroutine 254 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000340720, 0x6)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000340710)
    /usr/local/go/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc000340708, 0xc0007cb001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc0003406e0, 0xc0007cb001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/transport.go:2108 +0xaf
encoding/json.(*Decoder).refill(0xc0000cb8c0, 0xa, 0x9)
    /usr/local/go/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc0000cb8c0, 0x0, 0x0, 0x1e04d60)
    /usr/local/go/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc0000cb8c0, 0x1e3c540, 0xc00087c940, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:63 +0x79
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc00048d8f0, 0xc000906000, 0x1000, 0x1300, 0xc0007f5d98, 0x7fc452e08868, 0x38)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc00049c4b0, 0x0, 0x23df740, 0xc000933140, 0x0, 0x0, 0x464b00, 0xc00070ac00, 0xc0007f5e80)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000527ea0, 0xc0007f5f18, 0x8, 0x23d9200, 0xc000ae1340, 0x0, 0x0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0009aa940)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/watch/streamwatcher.go:104 +0x14a
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/watch/streamwatcher.go:71 +0xbe
goroutine 302 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000b1b910, 0x1)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000b1b900)
    /usr/local/go/src/sync/cond.go:56 +0x9d
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000b3aba0, 0x0, 0x0, 0xc000702300)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/util/workqueue/queue.go:145 +0x89
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000acc460, 0x24232c0, 0xc000ae30c0, 0x0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:236 +0x68
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x24232c0, 0xc000ae30c0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00024ef50)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00024ef50, 0x23cc720, 0xc0009ec5d0, 0xc000ae3001, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00024ef50, 0x3b9aca00, 0x0, 0x3b9aca01, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x24232c0, 0xc000ae30c0, 0xc0003c3a60, 0x3b9aca00, 0x0, 0x10000c000981901)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x24232c0, 0xc000ae30c0, 0xc0003c3a60, 0x3b9aca00)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:213 +0x45e
goroutine 358 [select]:
net/http.(*persistConn).writeLoop(0xc00091e5a0)
    /usr/local/go/src/net/http/transport.go:2346 +0x11c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1716 +0xcdc
goroutine 274 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:772 +0x5d
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00024ef60)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00083ef60, 0x23cc720, 0xc0008f9320, 0x1de8c01, 0xc0006c4900)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00024ef60, 0x3b9aca00, 0x0, 0x7fc452d36201, 0xc0006c4900)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc00033cd80)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0003bfc60, 0xc000bd1950)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:71 +0x65
goroutine 303 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0009c2150, 0xc00099b6e0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:71 +0x65
goroutine 235 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).serveHealthProbes(0xc000ae0000)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/internal.go:431 +0xea
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/internal.go:478 +0x44a
goroutine 240 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc0009c81c0, 0xc00070a8a0, 0xc0006c5bc0, 0xc000b3a240)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:373 +0x16f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:367 +0x2c5
goroutine 260 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x23dfb80, 0xc00036d380, 0xc000bd1130)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/record/event.go:301 +0xaa
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/record/event.go:299 +0x6e
goroutine 291 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc00075a7e0, 0xc0006c50e0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:628 +0x53
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0004d0af0, 0xc00022e6a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:71 +0x65
goroutine 198 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*InformersMap).Start(0xc00094bba0, 0x24232c0, 0xc000ae30c0, 0x0, 0x0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/cache/internal/deleg_map.go:68 +0xf9
sigs.k8s.io/controller-runtime/pkg/cluster.(*cluster).Start(0xc000946e00, 0x24232c0, 0xc000ae30c0, 0x0, 0x0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/cluster/internal.go:127 +0x8a
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1(0xc000ae0000, 0x7fc452cfc640, 0xc000946e00)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/internal.go:681 +0x7e
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/internal.go:679 +0x73
goroutine 233 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager/signals.SetupSignalHandler.func1(0xc000a959e0, 0xc000a675f0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/signals/signal.go:38 +0x34
created by sigs.k8s.io/controller-runtime/pkg/manager/signals.SetupSignalHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/signals/signal.go:37 +0xe8
goroutine 293 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000772460, 0xc01ccd343123e525, 0x276ecbc9, 0x327c4c0, 0x23dfb40, 0xc000356b40, 0xc00001bbc8, 0xc000980ea0, 0xc00070a8a0, 0x0, ...)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:463 +0x1a5
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000772460, 0xc00070a8a0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:427 +0x6ac
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000542ef0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00001bef0, 0x23cc700, 0xc000a2b590, 0x1, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0xad
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000772460, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:220 +0x1c5
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0004d0b10, 0xc00022e6e0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:71 +0x65
goroutine 298 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:772 +0x5d
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000543760)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000543760, 0x23cc720, 0xc0009ec600, 0x1de8c01, 0xc0006c5ec0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000543760, 0x3b9aca00, 0x0, 0x1, 0xc0006c5ec0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000b73500)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00075a840, 0xc0003c39c0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:71 +0x65
goroutine 256 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc000772460, 0xc00070a8a0, 0xc00070aea0, 0xc000980ea0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:373 +0x16f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:367 +0x2c5
goroutine 360 [select]:
net/http.(*persistConn).writeLoop(0xc00070f200)
    /usr/local/go/src/net/http/transport.go:2346 +0x11c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1716 +0xcdc
goroutine 272 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000b3aba0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
    /go/pkg/mod/k8s.io/client-go@v0.20.4/util/workqueue/queue.go:58 +0x135
goroutine 284 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc0007720e0, 0xc00070a8a0, 0xc0006c4d80, 0xc000b3b800)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:373 +0x16f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:367 +0x2c5
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc000abfd40, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/controller.go:154 +0x2e5
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000acc960, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:410 +0x42a
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).addInformerToMap
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/cache/internal/informers_map.go:229 +0x5a5
goroutine 208 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0009c81c0, 0xc01ccd34259f2fb8, 0x1bea165c, 0x327c4c0, 0x23dfb40, 0xc0009ab180, 0xc000fddbc8, 0xc000b3a240, 0xc00070a8a0, 0x0, ...)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:463 +0x1a5
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0009c81c0, 0xc00070a8a0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:427 +0x6ac
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004b76f0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000fddef0, 0x23cc700, 0xc00049cfa0, 0x1, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0xad
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0009c81c0, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:220 +0x1c5
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0009c2160, 0xc00099b720)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:71 +0x65
goroutine 234 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).serveMetrics(0xc000ae0000)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/internal.go:393 +0x1fa
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/manager/internal.go:473 +0x474
goroutine 287 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000accc08, 0x0)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000accbf8)
    /usr/local/go/src/sync/cond.go:56 +0x9d
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000accbe0, 0xc0003c3580, 0x0, 0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/delta_fifo.go:493 +0x98
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00065c2d0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/controller.go:183 +0x42
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0009cee68)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0009cee68, 0x23cc720, 0xc000a1b980, 0xc0004d0b01, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009cee68, 0x3b9aca00, 0x0, 0xc00000a901, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc00065c2d0, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/controller.go:154 +0x2e5
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000accb40, 0xc00070a8a0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:410 +0x42a
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).addInformerToMap
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/cache/internal/informers_map.go:229 +0x5a5
goroutine 246 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc000ae54e0, 0x2)
    /usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000ae54d0)
    /usr/local/go/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc000ae54c8, 0xc00017a001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc000ae54a0, 0xc00017a001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20210326060303-6b1517762897/http2/transport.go:2108 +0xaf
encoding/json.(*Decoder).refill(0xc0009fe420, 0xa, 0x9)
    /usr/local/go/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc0009fe420, 0x0, 0x0, 0x1e04d60)
    /usr/local/go/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc0009fe420, 0x1e3c540, 0xc00094adc0, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:63 +0x79
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0009ed8c0, 0xc00032b300, 0x1000, 0x1300, 0xc000395d98, 0x7fc452e09b88, 0x38)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc00049d310, 0x0, 0x23df740, 0xc0006f8100, 0x0, 0x0, 0x464b00, 0xc00070ad20, 0xc000395e80)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0009e9a60, 0xc000395f18, 0x8, 0x23d9040, 0xc000343600, 0x0, 0x0)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0009ab180)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/watch/streamwatcher.go:104 +0x14a
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/watch/streamwatcher.go:71 +0xbe
goroutine 299 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000b73500)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:742 +0x157
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00075a840, 0xc0003c39d0)
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:71 +0x65
goroutine 259 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x23dfb80, 0xc00036d350, 0xc00036d320)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/record/event.go:301 +0xaa
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/record/event.go:299 +0x6e
goroutine 207 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc00070a8a0, 0xc000983d40)
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
    /go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/controller.go:129 +0xa5
stream closed```
@stefanprodan stefanprodan transferred this issue from fluxcd/flux2 May 6, 2021
@stefanprodan
Copy link
Member

stefanprodan commented May 6, 2021 •
edited
Loading

Duplicate of: #310
Upstream bug: kubernetes-sigs/kustomize#3659

@stefanprodan stefanprodan mentioned this issue May 11, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Upgrade controller to Kustomize v4 fluxcd/kustomize-controller
2 participants
@stefanprodan @caddac