Kong Upstream Time Drastically Improves When Not Using Ingress Controller

[probably-not]

Summary

I have a GKE cluster using Kong as the Ingress Controller. Until now, I've been using the Kubernetes Ingress Controller for routes/services, everything according to the documentation.

Yesterday, we decided to switch 50% of our traffic from using the ingress controller managed services/routes/upstreams/targets to defining the service directly in the Admin API, and using an upstream that targets the Kubernetes host name of the service, instead of letting the ingress controller define the targets to the pods directly. Miraculously, this resulted in a 10x performance improvement in the p99 of the upstream time, going from 100ms to 10ms. No changes to the service itself, just changing Kong from using the ingress controller managed services/routes/upstreams/targets to setting the upstream to be a single target as opposed to the multiple targets set by the controller.

Steps To Reproduce

1. Use the Kong Ingress Controller per the documentation
2. Create upstream with single target and move traffic to the new upstream
3. See 10x performance improvement

Additional Details & Logs

• Kong version: 2.2 (KIC 1.1)

[fffonion]

@probably-not Would you be able to compare the difference of one service/route/upstream/target set from
Kong Ingress Controller managed and the one that is created through Admin API? You can do so by using
db_export function.
Also are there different plugins being applied?

[probably-not]

I only have one plugin (prometheus as a global plugin). That's actually where I'm getting the data about the p99, from the latency buckets reported by the prometheus plugin.

I can try to compare the differences although adding the ingresses back via the ingress controller is potentially damaging, since it will use the slower ingresses... so I'll have to wait on that until tomorrow when we have a lower traffic time, I will update when I have it.

The main difference that I know of is the amount of upstream targets. When using the ingress controller, it defines each pod individually as a target, so there are between 150 (in low traffic) and 600 (in high traffic) targets. When I switched to using a single target of the Kubernetes host name, there was only the one target. Could that lead to an issue in the speed?

[probably-not]

I unfortunately can't provide the information from the db since rolling back to the ingresses via the ingress controller is a problem for us (it affects our latencies and uptime).
If someone wants to try and reproduce this, like I mentioned, we followed the Kong Ingress Controller documentation almost exactly when adding in the ingresses, so the ingress controller should just be writing to the db normally.

As a side note: If you can break down the latency breakdown metrics to more than just "proxy"/"upstream"/"request" so that we know what steps are actually part of each section, then this will help tremendously. That way we can determine if it's due to retries, DNS resolving, target resolving, etc.