AWX-Operator's rediness and liveness probs are failing

[manneharikrishna]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Liveness and rediness probs filing from awx-operator and keep restarting the awx-manager container from k8s pod

AWX version

23.9.0

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

2.12.8

Operating system

centos 8

Web browser

Chrome

Steps to reproduce

Deploy awx-operator version 2.12.2

Expected results

Pod shoud successfully deploy without any errors

Actual results

[ec2-user@ip-192-168-31-118 awx-21]$ k logs awx-operator-controller-manager-768c7bcbdd-knnmk -n awx-21 -c awx-manager -f
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"cmd","msg":"Version","Go Version":"go1.20.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.34.0","commit":"d26c43bf94960d292152862a6685696be33190fb"}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"cmd","msg":"Watching namespaces","namespaces":["awx-21"]}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWX_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXBACKUP_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXRESTORE_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXMESHINGRESS_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWX"}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXBackup"}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXRestore"}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1alpha1","Options.Kind":"AWXMeshIngress"}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"apiserver","msg":"Starting to serve metrics listener","Address":"localhost:5050"}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-04-08T09:32:52Z","msg":"starting server","kind":"health probe","addr":"[::]:6789"}
{"level":"info","ts":"2024-04-08T09:32:52Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false}
I0408 09:32:52.522017 7 leaderelection.go:250] attempting to acquire leader lease awx-21/awx-operator...
I0408 09:33:17.733681 7 leaderelection.go:260] successfully acquired lease awx-21/awx-operator
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting EventSource","controller":"awx-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting EventSource","controller":"awxbackup-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting Controller","controller":"awxbackup-controller"}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting Controller","controller":"awx-controller"}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting EventSource","controller":"awxmeshingress-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting Controller","controller":"awxmeshingress-controller"}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting EventSource","controller":"awxrestore-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting Controller","controller":"awxrestore-controller"}
{"level":"error","ts":"2024-04-08T09:33:17Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:49\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:50\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:33:17Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting workers","controller":"awxbackup-controller","worker count":8}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting workers","controller":"awx-controller","worker count":8}
{"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting workers","controller":"awxrestore-controller","worker count":8}
{"level":"error","ts":"2024-04-08T09:33:27Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:33:37Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:33:47Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:33:57Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:34:07Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:34:17Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:34:27Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:34:37Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:34:47Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:34:57Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:35:07Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind "AWXMeshIngress" in group "awx.ansible.com"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"}
{"level":"error","ts":"2024-04-08T09:35:17Z","msg":"Could not wait for Cache to sync","controller":"awxmeshingress-controller","error":"failed to wait for awxmeshingress-controller caches to sync: timed out waiting for cache to be synced for Kind *unstructured.Unstructured","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:203\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:208\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/manager/runnable_group.go:223"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"awxrestore-controller"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"awx-controller"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"awxbackup-controller"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"All workers finished","controller":"awxrestore-controller"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"All workers finished","controller":"awx-controller"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"All workers finished","controller":"awxbackup-controller"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for caches"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for HTTP servers"}
{"level":"info","ts":"2024-04-08T09:35:17Z","logger":"controller-runtime.metrics","msg":"Shutting down metrics server with timeout of 1 minute"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"shutting down server","kind":"health probe","addr":"[::]:6789"}
{"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":"2024-04-08T09:35:17Z","logger":"cmd","msg":"Proxy or operator exited with error.","error":"failed to wait for awxmeshingress-controller caches to sync: timed out waiting for cache to be synced for Kind *unstructured.Unstructured","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.run\n\tansible-operator-plugins/internal/cmd/ansible-operator/run/cmd.go:261\ngithub.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.NewCmd.func1\n\tansible-operator-plugins/internal/cmd/ansible-operator/run/cmd.go:81\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\nmain.main\n\tansible-operator-plugins/cmd/ansible-operator/main.go:40\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.12/x64/src/runtime/proc.go:250"}
{"level":"error","ts":"2024-04-08T09:35:17Z","msg":"error received after stop sequence was engaged","error":"leader election lost","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/manager/internal.go:490"}

Additional information

Events:
Type Reason Age From Message

---

Normal Scheduled 21m default-scheduler Successfully assigned awx-21/awx-operator-controller-manager-768c7bcbdd-knnmk to ip-192-168-74-156.ec2.internal
Normal Pulling 21m kubelet Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0"
Normal Pulled 20m kubelet Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0" in 12.675458384s (12.675472904s including waiting)
Normal Created 20m kubelet Created container kube-rbac-proxy
Normal Started 20m kubelet Started container kube-rbac-proxy
Normal Pulling 20m kubelet Pulling image "468681639671.dkr.ecr.us-east-1.amazonaws.com/concierto-uat-awx-operator:23.9.0.2"
Normal Pulled 6m19s kubelet Successfully pulled image "468681639671.dkr.ecr.us-east-1.amazonaws.com/concierto-uat-awx-operator:23.9.0.2" in 14m35.572378533s (14m35.572392633s including waiting)
Normal Created 5m29s kubelet Created container awx-manager
Normal Started 5m27s kubelet Started container awx-manager
Warning Unhealthy 2m11s (x3 over 2m51s) kubelet Liveness probe failed: Get "http://192.168.83.70:6789/healthz": dial tcp 192.168.83.70:6789: connect: connection refused
Normal Killing 2m11s kubelet Container awx-manager failed liveness probe, will be restarted
Warning Unhealthy 41s (x17 over 3m1s) kubelet Readiness probe failed: Get "http://192.168.83.70:6789/readyz": dial tcp 192.168.83.70:6789: connect: connection refused

[kurokobo]

@manneharikrishna
Hi, if you are using Helm to install Operator, refer to the docs and apply changes for CRDs.
https://github.com/ansible/awx-operator/tree/devel/.helm/starter#caveats-on-upgrading-existing-installation

EDIT: URL is corrected

[TheRealHaoLiu]

@kurokobo why is this related to #15053 (comment)?

[kurokobo]

@TheRealHaoLiu
Oops my bad, just pasting incorrect URL. Thanks for letting me know 😞

[fosterseth]

@manneharikrishna did you try out kurokobo's suggestion, did it help you get past this error?

[weiyentan]

is there a gitops thing that we can do here to fix this? like a hook or something upon the upgrade?

@fosterseth for posterity i have tried it and it works. my controller hasnt crashed yet