Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Never-provisioned LoadBalancer blocks controller operation #2001

Closed
1 task done
rainest opened this issue Nov 9, 2021 · 1 comment · Fixed by #2005
Closed
1 task done

Never-provisioned LoadBalancer blocks controller operation #2001

rainest opened this issue Nov 9, 2021 · 1 comment · Fixed by #2005
Labels
bug Something isn't working

Comments

@rainest
Copy link
Contributor

rainest commented Nov 9, 2021

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Installing a controller+Kong Deployment with the default LoadBalancer type proxy Service in a cluster that does not actually provide LoadBalancers prevents it from ingesting configuration. Ingresses do not result in routes, etc.

Expected Behavior

The controller proceeds as usual and starts handling configuration even though it cannot determine status of the LoadBalancer.

Steps To Reproduce

1. Start a KIND cluster with no arguments (so no LB provisioning)
2. `helm install trk /tmp/symkong --set ingressController.admissionWebhook.enabled=true --set ingressController.env.anonymous_reports=false`
3. Create a basic Ingress, e.g. https://docs.konghq.com/kubernetes-ingress-controller/2.0.x/guides/getting-started/
4. Check the admin API route list. There will be no routes.
5. Edit the proxy Service and change its type to NodePort.
6. Check routes again.

Kong Ingress Controller version

2.0.5

Kubernetes version

No response

Anything else?

Initially I thought this was directly related to

log.V(util.InfoLevel).Info("LoadBalancer type Service for the Kong proxy is not yet provisioned, waiting...", "service", publishService)
, which spams constantly with this configuration. However, this is within the status update loop and should be separate from the rest of the controller. It should not obviously block config updates. However, logs clearly show that the regular config sync message does not appear until after editing the service type:

logs.txt
cmds.txt

We probably need to do a bit more digging to see where this blocks (unless deleting that block from status does magically resolve the issue).

@rainest rainest added bug Something isn't working needs investigation labels Nov 9, 2021
@rainest
Copy link
Contributor Author

rainest commented Nov 9, 2021

Although the status code above was blocking here, it should not have worked previously. The failure to process configuration occurs because sendconfig blocks on sending to the status update channel, and the status subsystem does not start receiving on that channel until after its setup finishes. Prior to #1931, we simply failed immediately if LB IP info was not available, and we would not start receiving. After, we wait indefinitely for an LB IP to become available, and if that never happens we never start receiving.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
1 participant