Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ingress-controller failed to list routes #811

Closed
xiongyunhua-star opened this issue Dec 24, 2021 · 8 comments
Closed

ingress-controller failed to list routes #811

xiongyunhua-star opened this issue Dec 24, 2021 · 8 comments
Labels
question Further information is requested triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@xiongyunhua-star
Copy link

Issue description

logs:
[root@k8smaster apisix]# kubectl logs -f apisix-ingress-controller-f68756d54-59vsk -n ingress-apisix
2021-12-24T13:37:57+08:00 info ingress/ingress.go:112 apisix ingress controller started
2021-12-24T13:37:57+08:00 info ingress/ingress.go:114 version:
Version: 1.3.0
Git SHA: no-git-module
Go Version: go1.13.8
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

2021-12-24T13:37:57+08:00 info ingress/ingress.go:120 use configuration
{
"cert_file": "/etc/webhook/certs/cert.pem",
"key_file": "/etc/webhook/certs/key.pem",
"log_level": "info",
"log_output": "stderr",
"http_listen": ":8080",
"https_listen": ":8443",
"enable_profiling": true,
"kubernetes": {
"kubeconfig": "",
"resync_interval": "6h0m0s",
"app_namespaces": [
""
],
"election_id": "ingress-apisix-leader",
"ingress_class": "apisix",
"ingress_version": "networking/v1",
"watch_endpoint_slices": false,
"apisix_route_version": "apisix.apache.org/v2beta1"
},
"apisix": {
"default_cluster_name": "default",
"default_cluster_base_url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin",
"default_cluster_admin_key": "edd1c9f034335f136f87ad84b625c8f1",
"base_url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin",
"admin_key": "edd1c9f034335f136f87ad84b625c8f1"
}
}
2021-12-24T13:37:57+08:00 warn api/server.go:71 failed to load x509 key pair, will not start admission server {"Error": "open /etc/webhook/certs/cert.pem: no such file or directory", "CertFilePath": "/etc/webhook/certs/cert.pem", "KeyFilePath": "/etc/webhook/certs/key.pem"}
I1224 13:37:57.837406 1 leaderelection.go:243] attempting to acquire leader lease ingress-apisix/ingress-apisix-leader...
2021-12-24T13:37:57+08:00 warn ingress/controller.go:328 found a new leader apisix-ingress-controller-67579b5f67-4bpqs
2021-12-24T13:37:57+08:00 info ingress/controller.go:330 controller now is running as a candidate {"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-f68756d54-59vsk"}
[GIN] 2021/12/24 - 13:37:58 | 200 | 78.803µs | 10.244.4.1 | GET "/healthz"
[GIN] 2021/12/24 - 13:38:04 | 200 | 39.683µs | 10.244.4.1 | GET "/healthz"
[GIN] 2021/12/24 - 13:38:04 | 200 | 56.416µs | 10.244.4.1 | GET "/healthz"
[GIN] 2021/12/24 - 13:38:14 | 200 | 33.952µs | 10.244.4.1 | GET "/healthz"
[GIN] 2021/12/24 - 13:38:14 | 200 | 27.516µs | 10.244.4.1 | GET "/healthz"
2021-12-24T13:38:15+08:00 info ingress/controller.go:290 LeaderElection {"message": "apisix-ingress-controller-f68756d54-59vsk became leader", "event_type": "Normal"}
I1224 13:38:15.252511 1 leaderelection.go:253] successfully acquired lease ingress-apisix/ingress-apisix-leader
2021-12-24T13:38:15+08:00 warn ingress/controller.go:328 found a new leader apisix-ingress-controller-f68756d54-59vsk
2021-12-24T13:38:15+08:00 info ingress/controller.go:370 controller tries to leading ... {"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-f68756d54-59vsk"}
2021-12-24T13:38:15+08:00 warn apisix/cluster.go:304 waiting cluster default to ready, it may takes a while
2021-12-24T13:38:15+08:00 info apisix/cluster.go:155 syncing cache {"cluster": "default"}
2021-12-24T13:38:15+08:00 info apisix/cluster.go:344 syncing schema {"cluster": "default"}
2021-12-24T13:38:15+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/379501036760466577", "error": "unexpected non-empty object"}
2021-12-24T13:38:15+08:00 error apisix/cluster.go:202 failed to list upstreams in APISIX: unexpected non-empty object
2021-12-24T13:38:17+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/379501036760466577", "error": "unexpected non-empty object"}
2021-12-24T13:38:17+08:00 error apisix/cluster.go:202 failed to list upstreams in APISIX: unexpected non-empty object
2021-12-24T13:38:19+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/379501036760466577", "error": "unexpected non-empty object"}
2021-12-24T13:38:19+08:00 error apisix/cluster.go:202 failed to list upstreams in APISIX: unexpected non-empty object
2021-12-24T13:38:21+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/379501036760466577", "error": "unexpected non-empty object"}
2021-12-24T13:38:21+08:00 error apisix/cluster.go:202 failed to list upstreams in APISIX: unexpected non-empty object
2021-12-24T13:38:23+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/379501036760466577", "error": "unexpected non-empty object"}
2021-12-24T13:38:23+08:00 error apisix/cluster.go:202 failed to list upstreams in APISIX: unexpected non-empty object
2021-12-24T13:38:23+08:00 error apisix/cluster.go:164 failed to sync cache {"cost_time": "8.033891703s", "cluster": "default"}
2021-12-24T13:38:23+08:00 error ingress/controller.go:396 failed to wait the default cluster to be ready: unexpected non-empty object
2021-12-24T13:38:23+08:00 info ingress/controller.go:337 controller now is running as a candidate {"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-f68756d54-59vsk"}
I1224 13:38:23.286893 1 leaderelection.go:243] attempting to acquire leader lease ingress-apisix/ingress-apisix-leader...
2021-12-24T13:38:23+08:00 info apisix/cluster.go:155 syncing cache {"cluster": "default"}
2021-12-24T13:38:23+08:00 error apisix/route.go:117 failed to list routes: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:23+08:00 error apisix/cluster.go:197 failed to list route in APISIX: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:23+08:00 info apisix/cluster.go:344 syncing schema {"cluster": "default"}
2021-12-24T13:38:23+08:00 error apisix/plugin.go:46 failed to list plugins' names: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugins/list: context canceled
2021-12-24T13:38:23+08:00 error apisix/cluster.go:364 failed to list plugin names in APISIX: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugins/list: context canceled
2021-12-24T13:38:23+08:00 warn apisix/cluster.go:327 failed to sync schema: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugins/list: context canceled
2021-12-24T13:38:23+08:00 info ingress/controller.go:290 LeaderElection {"message": "apisix-ingress-controller-f68756d54-59vsk became leader", "event_type": "Normal"}
I1224 13:38:23.290143 1 leaderelection.go:253] successfully acquired lease ingress-apisix/ingress-apisix-leader
2021-12-24T13:38:23+08:00 info ingress/controller.go:370 controller tries to leading ... {"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-f68756d54-59vsk"}
2021-12-24T13:38:23+08:00 warn apisix/cluster.go:304 waiting cluster default to ready, it may takes a while
[GIN] 2021/12/24 - 13:38:24 | 200 | 33.287µs | 10.244.4.1 | GET "/healthz"
[GIN] 2021/12/24 - 13:38:24 | 200 | 30.477µs | 10.244.4.1 | GET "/healthz"
2021-12-24T13:38:25+08:00 error apisix/route.go:117 failed to list routes: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:25+08:00 error apisix/cluster.go:197 failed to list route in APISIX: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:27+08:00 error apisix/route.go:117 failed to list routes: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:27+08:00 error apisix/cluster.go:197 failed to list route in APISIX: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:29+08:00 error apisix/route.go:117 failed to list routes: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:29+08:00 error apisix/cluster.go:197 failed to list route in APISIX: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:31+08:00 error apisix/route.go:117 failed to list routes: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:31+08:00 error apisix/cluster.go:197 failed to list route in APISIX: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-24T13:38:31+08:00 error apisix/cluster.go:164 failed to sync cache {"cost_time": "8.000947086s", "cluster": "default"}
2021-12-24T13:38:31+08:00 error ingress/controller.go:396 failed to wait the default cluster to be ready: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled

versions:
[root@k8smaster apisix]# helm ls -n ingress-apisix
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
apisix ingress-apisix 1 2021-12-24 13:46:20.308346669 +0800 CST deployed apisix-0.7.2 2.10.0
apisix-dashboard ingress-apisix 1 2021-12-24 13:51:05.741042576 +0800 CST deployed apisix-dashboard-0.3.0 2.9.0
apisix-ingress-controller ingress-apisix 1 2021-12-24 13:52:48.510484801 +0800 CST deployed apisix-ingress-controller-0.8.0 1.3.0

apisix configmap:

  allow_admin:                  # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
    - 0.0.0.0/0
etcd:
  host:                                 # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
    - "http://192.168.128.201:2379"             # multiple etcd address

pod status:

[root@k8smaster apisix]# kubectl get pod -n ingress-apisix
NAME READY STATUS RESTARTS AGE
apisix-5fcc8d9475-2kqt2 1/1 Running 0 49m
apisix-dashboard-57b5474cfd-thtcj 1/1 Running 0 42m
apisix-ingress-controller-596dd4ccd5-82vrd 1/1 Running 0 43m

Environment

  • your apisix-ingress-controller version (output of apisix-ingress-controller version --long):
  • your Kubernetes cluster version (output of kubectl version):
  • if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):
@tao12345666333
Copy link
Member

How did you install it?

@tao12345666333 tao12345666333 added question Further information is requested triage/needs-information Indicates an issue needs more information in order to work on it. labels Dec 24, 2021
@tokers
Copy link
Contributor

tokers commented Dec 24, 2021

@xiongyunhua-star Does the Ingress Controller run before Apache APISIX? I suspect that you're suffering from this bug #774.

@xiongyunhua-star
Copy link
Author

How did you install it?

参考的这个链接 https://github.com/apache/apisix-helm-chart/blob/master/docs/en/latest/apisix-ingress-controller.md

@xiongyunhua-star
Copy link
Author

@xiongyunhua-star Does the Ingress Controller run before Apache APISIX? I suspect that you're suffering from this bug #774.

以前有运行Apache apisix, 然后更新了helm重新部署，这个bug计划什么时候能修复

@allenhaozi
Copy link

Have the same problem

2021-12-26T00:33:38+08:00	warn	apisix/cluster.go:327	failed to sync schema: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugins/list: context canceled
2021-12-26T00:33:38+08:00	info	ingress/controller.go:290	LeaderElection	{"message": "apisix-ingress-controller-f68756d54-2qbtw became leader", "event_type": "Normal"}
I1226 00:33:38.277388       1 leaderelection.go:253] successfully acquired lease ingress-apisix/ingress-apisix-leader
2021-12-26T00:33:38+08:00	info	ingress/controller.go:370	controller tries to leading ...	{"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-f68756d54-2qbtw"}
2021-12-26T00:33:38+08:00	warn	apisix/cluster.go:304	waiting cluster default to ready, it may takes a while
[GIN] 2021/12/26 - 00:33:38 | 200 |      101.22µs |    172.27.70.36 | GET      "/healthz"
2021-12-26T00:33:40+08:00	debug	apisix/route.go:111	try to list routes in APISIX	{"cluster": "default", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes"}
2021-12-26T00:33:40+08:00	error	apisix/route.go:117	failed to list routes: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled
2021-12-26T00:33:40+08:00	error	apisix/cluster.go:197	failed to list route in APISIX: Get http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes: context canceled

@tao12345666333
Copy link
Member

#774 has been merged. It will release in v1.4 (next week

@allenhaozi
Copy link

Thank u
I used the master branch to build a new image,
This upgrade fixed the problem

@tao12345666333
Copy link
Member

Can I close this one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tao12345666333 @allenhaozi @tokers @xiongyunhua-star