Replies: 18 comments 6 replies
-
|
Any suggestion? |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sany98215! |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sany98215, 2 seconds indeed seems very long. Could you enable debug logging for Kong and paste it here so we can tell better what is using the processing times? |
Beta Was this translation helpful? Give feedback.
-
and got those output: |
Beta Was this translation helpful? Give feedback.
-
|
@dndx @locao Latency come from |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the logs. It seems that the resolver that Kong is using for resolving the domain may be slow. Could you try to change the upstream address to IP and see if the latency disappears? If so, then you will probably have to investigate on the slow resolver performance in order to solve this. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for reply. When I change the upstream address to IP, the latency disappears.
DNS is a CoreDNS service by K8s,which the Max latency is about 385ms,but the latency by Kong is 2s.
What shoud i do next?
At 2021-07-01 01:10:00, "Datong Sun" ***@***.***> wrote:
@sany98215
Thanks for the logs. It seems that the resolver that Kong is using for resolving the domain may be slow. Could you try to change the upstream address to IP and see if the latency disappears?
If so, then you will probably have to investigate on the slow resolver performance in order to solve this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Beta Was this translation helpful? Give feedback.
-
|
@sany98215 please, check the DNS server latency, it is the most probable source of the latency you are experiencing: You will get something like this: The |
Beta Was this translation helpful? Give feedback.
-
|
@locao |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sany98215 , Is there any resolvers inside Kong uses all resolvers it can find from |
Beta Was this translation helpful? Give feedback.
-
|
Hi ,@datong Sun!
I just have one dns which is suplied by k8s. The difference is :Each Service is register as "{app}.{namespace}" , but the real domain name is {app}.{namespace}.svc.cluster.local.
And the max latancy of "{app}.{namespace}.svc.cluster.local " is about 300ms from CoreDNS.
At 2021-07-22 01:12:30, "Datong Sun" ***@***.***> wrote:
Hello @sany98215 ,
Is there any resolvers inside /etc/resolv.conf that may be running slow.
Kong uses all resolvers it can find from /etc/resolv.conf, so if some of them are slow or unreachable and some are fast, the resolving time would still jump up high sometimes.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Beta Was this translation helpful? Give feedback.
-
|
The latency reported by DNS is for a single query, not for the sequential queries of all types. k8s typically uses What happens if you make multiple requests, such that Kong gets the opportunity to use its DNS cache (note that the cache is per worker process, so you might have to hit it several times before the latency goes down). How is your That said, 2sec is still very long. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, @Tieske！
You are right! The latency came from many queries to try. The configure of serives is "http://{app}.{namespace}:{port}/{service_url}", but the real virtual domain name is `app.namespace.svc.cluster.local`.
The first query to resolve 'app.namespace' is the main latency.
But can you give some suggest for configuration in k8s? The service domain name come from three part:serviceName,namespace and defaut suffix of CoreDNS.
At 2021-07-22 13:44:07, "Thijs Schreijer" ***@***.***> wrote:
The latency reported by DNS is for a single query, not for the sequential queries of all types. k8s typically uses ndots=5 in the dns config and has a long list of domains to try. So that can be many queries to try.
What happens if you make multiple requests, such that Kong gets the opportunity to use its DNS cache (note that the cache is per worker process, so you might have to hit it several times before the latency goes down).
How is your service object (in Kong) configured?
That said, 2sec is still very long.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Beta Was this translation helpful? Give feedback.
-
|
@sany98215 what happens if you try multiple requests? The dns cache should help, does it? Also; typically only the first dns lookup is synchronous/in-band and causes a high latency on the first request. Refreshing is done in the background (async/out-of-band). What you can try is to use an upstream entity in Kong (load balancer) since that will resolve the name up front iirc |
Beta Was this translation helpful? Give feedback.
-
|
@Tieske Dns cache works, the percentage of low performance is very low. But the latency happend top requests every day. |
Beta Was this translation helpful? Give feedback.
-
|
in general if there is no traffic for that service, for longer than the DNS record TTL, then it might be expelled from the cache, and the first request after will do a synchronous lookup again, causing the longer delay. As long as there is plenty of traffic, the updates will be in the background. |
Beta Was this translation helpful? Give feedback.
-
|
I'm converting this to a discussion as this is not an issue with Kong itself. |
Beta Was this translation helpful? Give feedback.
-
|
I'm seeing very slow response times from kong when under sustained work load and it doesn't seem to be dns resolution time
Run 1 is sending requests directly to a using the kubernetes fqdn for the service. Run 2 is sending requests to kong which is then sending requests to the same service. The workload for the runs is exactly the same. The yellow line is the total response time for p95. When hitting the service directly, its averages 1.5 seconds. When hitting Kong it averages 5 seconds. The median response time is oddly enough lower through kong than the service directly, but The p95 latency is exceedingly high. |
Beta Was this translation helpful? Give feedback.
-
|
Just to make sure we are still talking about the same thing: this difference doesn't exist when you use IP addresses instead of FQDNs for the service, right? |
Beta Was this translation helpful? Give feedback.
-
|
The FQDN to kong demonstrates the diffiference. |
Beta Was this translation helpful? Give feedback.
-
|
The majority of this issue was a problem with the default load balancer w/ kong. Changing it to least-connections solved much of the problem. But there is still quite a bit of additional latency going through kong ~100ms vs 10ms. There is only 1 plugin enabled - prometheus metrics |
Beta Was this translation helpful? Give feedback.
-
|
to further clarify - effectively curl -XPOST @data.json -H 'Host: debug.test.com' -H 'Content-Type: application/json' https://kong.test.com vs curl -XPOST @data.json -H 'Content-Type: application/json' http://debug-service.default.svc.cluster.local |
Beta Was this translation helpful? Give feedback.
-
|
To eliminate the differences in dns resolution over the public internet, I pointed the same test and cluster local dns
The ingress for kong in kubernetes has only 4 rules and 1 plugin
|
Beta Was this translation helpful? Give feedback.
-
I register an app service with
/app1, when i access on the FF browser, I found long lantency than no gateway.Then I trace the request ,found this:
some request have long lantency than 2s, try open log but found nothing.
I want to know why?
Beta Was this translation helpful? Give feedback.
All reactions