Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consider tweaking CRD controller backoff #587

Closed
lkysow opened this issue Aug 2, 2021 · 0 comments · Fixed by #788
Closed

Consider tweaking CRD controller backoff #587

lkysow opened this issue Aug 2, 2021 · 0 comments · Fixed by #788
Labels
hacktoberfest type/enhancement New feature or request

Comments

@lkysow
Copy link
Member

lkysow commented Aug 2, 2021

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Is your feature request related to a problem? Please describe.

When applying a config entry that gets an error due to an incorrect protocol, the next action is to create a servicedefaults entry that sets the correct protocol. I've observed that I have to then wait a long time (2m 30s) for the previously erroring config entry to reconcile and get created.

In the below example I created an ingress gateway using the http protocol, this failed because the backing service was using tcp, then I created the servicedefaults setting protocol to http, then I had to wait 2.5 minutes for the ingress entry to reconcile.

Ideally I'd wait <30s max. I think the backoff schedule doesn't make sense for our use case. It seems to try 10 times within 1s and then steeply backs off whereas we need something like:

  • retry 3 times quickly
  • backoff to 5s, 10s, 20s, 30s, 30s, 30s, 30s, 1m, 1m steady
logs 
2021-08-02T04:52:13.641Z	INFO	webhooks.ingressgateway	validate create	{"name": "ingress-gateway"}
2021-08-02T04:52:13.747Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:13.807Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:13.810Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:13.821Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:13.823Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:13.905Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:13.909Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:13.927Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:13.940Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:13.952Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:14.005Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:14.019Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:14.023Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:14.037Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:14.183Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:14.196Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:14.840Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:14.853Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:16.136Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:16.180Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:16.183Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:16.222Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:18.743Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:18.788Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:18.791Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:18.883Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:29.031Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:29.044Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:52:29.047Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:52:29.059Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:53:09.938Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:53:09.951Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:53:09.954Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:53:09.964Z	ERROR	controller.ingressgateway	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "IngressGateway", "name": "ingress-gateway", "namespace": "default", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: service \"frontend\" has protocol \"tcp\", which does not match defined listener protocol \"http\")"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2021-08-02T04:53:20.573Z	INFO	webhooks.servicedefaults	validate create	{"name": "frontend"}
2021-08-02T04:53:20.597Z	INFO	controller.servicedefaults	config entry not found in consul	{"request": "default/frontend"}
2021-08-02T04:53:20.646Z	INFO	controller.servicedefaults	config entry created	{"request": "default/frontend", "request-time": "8.147ms"}


2021-08-02T04:55:53.624Z	INFO	controller.ingressgateway	config entry not found in consul	{"request": "default/ingress-gateway"}
2021-08-02T04:55:53.630Z	INFO	controller.ingressgateway	config entry created	{"request": "default/ingress-gateway", "request-time": "5.8055ms"}
@lkysow lkysow added the type/enhancement New feature or request label Aug 2, 2021
@lkysow lkysow mentioned this issue Oct 18, 2021
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
hacktoberfest type/enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Change CRD retry-backoff hashicorp/consul-k8s
2 participants
@david-yu @lkysow