-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Very slow and increasing "Average upstream response time" #6818
Comments
I have this exact same issue, especially with cloudflare. I had to remove 1.1.1.1 completely, but i'm still having issues using only google and opendns. running on rpi4 |
i use unbound and dnscrypt proxy upstream to quad9 resolver.
I don't know ControlD and its service - but maybe you could test it without it. It's all doubled twice. - Its not bad, but for testing we should use only one DNS Service to exclude the problem.... My results for the DNS Service you use:
|
@whyisthisbroken I will check those settings, thanks. Do you use quad9 DoH and should
Yes, I do. Even though it might be redundant, I tested these upstream DNS with and without filter lists and the processing time was the same, so I decided to use those with filter lists anyway. After setting Now in General statistics "Average processing time" is 8ms, although "Average upstream response time" is still quite big
I assume this means the most of DNS queries are served from cache, hence why this 8ms, while, when necessary, querying upstream gives these big numbers above? Also, one semi-related question. One client has always-on VPN and all the traffic goes through it, meanwhile AGH logs every minute or so a query from that client to www.google.com Type: A, Plain DNS. How is that possible? |
Can confirm the issue. Have to use my local root resolver. 1.1.1.1 (and ipv6) are terrible slow from adguard. |
I think I'm having the same issue. It's improved with some of the suggestions here, but something still seems off. |
Same issue here. Restarting the docker container helps for couple of hours... |
It's definitely abnormal and constantly increasing. The latest results for "Average upstream response time":
|
Where are you located ? Keep in mind that even though ControlD or Quad9 are both using unicast, the peerings used by your ISP might affect quite a lot ... |
I'm in Europe and there are multiple quad9 server locations near me, which dnsperftest script confirms as it shows 3ms on average. Unfortunately, with AGH that's not the case considering from my last reply "Average upstream response time" now increased to:
|
You maybe already did, but could you run an extended test on https://www.dnsleaktest.com/ ? to check which locations are actually answering your requests ? |
Sure.
Based on the domain of last three, seems like they are in the same city. One more thing I noticed, in the past couple of days average upstream response time decreased, but only slightly, as numbers are still huge:
Meanwhile, "Average processing time" increased from 8-9ms to now 153ms. |
I have the issue with my ISP DNS as well… Using my unbound (and let using it my isp dns) didn’t have these issue. So it’s a adguard thing since some versions. |
I also got the same problem |
@Cebeerre would you mind removing "waiting for data" label and add "bug" label, please? All these replies beside mine imply it's definitely a bug. |
Hi @netizeni In order tag an issue as a bug, reproducibility is key. The previous replies, apart from saying "me too", are not actually adding any additional information that might help. This is my own AGH instance using the NS you provided after running for some hours: Have you tried to clear the statistics and see if maybe it was just a very bad connectivity period (AGH timesouts by default at 10ms) ? |
As mentioned, I can add the exact same DNS servers to my opnsense instead of agh and have no issue. It doesn’t matter if i use my isp dns or Cloudflare. |
Have you tried to see if there are specific queries that are making the average increase ?
|
@Cebeerre I did a couple of times ever since opening this issue, and it starts with those numbers you shared, but eventually increases to these I pasted. |
I've seen that you've set a cache size of 134 Mb which honestly looks quite overkill ... Could you please check how much RAM you're actually consuming right now by the AGH process ? Do you have other stuff running on top of this rpi3 ? |
tostring))' | sort -t: -nrk2 | head -20 but currently the numbers are ok with my isp dns… in will try 1.1.1.1 |
Wow ! All of these ones are over the 10 seconds ! quite odd given that you're using a public resolver and these entries were probably in their cache already ... I use unbound as a recursive DNS upstream, which tipically "takes more time" and this is what I get for login.aliexpress.com: @dMopp , what kind of hardware are you using ? Am I right assuming it's directly connected by ethernet and you're not using wifi ? |
Hi. Yes this is extreme. As mentioned, using the same resolver in opnsense (unbound) I don’t see this spikes. I can even add my opnsense as dns for adguard home and it’s fine. The Hardware: Adguard Home and opnsense are both running in proxmox. Ah and yes, all wired. ICMP requests are not spiking. |
lxc or vm ? Have you tried to install the AdGuardHome plugin in proxmox itself to see if it makes any difference ? |
Opnsense running as a VM with PCIe pass through . AGH as LXC in a Debian container. (And yes, in the past this was fine). And which plugin you mean? |
|
Ah, you mean in OPNsense. No, because I don’t want the filtering in the firewall. This would cause some new issues. |
I'm curious about when you said that if you set the unbound instance in OPNSense as an AGH upstream it works fine. It shouldn't make any kind of difference than using a public resolver, what made me think if you've any kind of traffic shaping rules applied in OPNSense and the LXC container fell apart in an upload or download pipe without enough bandwidth ? |
Indeed i have traffic shaping, but just 2 Pipes and every traffic is passing them. UDP / 53 Traffic is even high prio in my network (independent from the Source/Destination) |
Great, could you please share your entire adguardhome.yaml ? |
Same here & seeing spike up to the 1000ms or more. |
Can you reopen this issue, as obviously it's not solved? Just like people above, I'm experiencing the same problem. Response times are constantly increasing. I understand this is not a trivial issue, but keeping it closed and ignoring it, won't magically make it go away. This is literally the most commented issue on the whole repository. |
In the end we'll have to create new issue describing same problem... it's been 2 weeks now since community asked to reopen issue. |
Still experiencing the same increased resolve times here too. Even with logging turned off and stats period return down 6 hours. Device then requires restart to reduce times however within 12-24 hours issue will return. Situation occurs on primary device (raspberry pi 4B 1GB) with 40-50k DNS queries and a second device (raspberry pi 3B+) with <5k DNS queries. Previous occurred on a Latte Panda v1 with an Intel Atom z8350 CPU. Apply changes to the configuration file seem to accelerate the longer DNS resolution times. |
In the edge release, we have fixed the bug that was causing slow upstream responses. Hopefully, it will be the last one. You can also download the binary for your platform from the following link |
@schzhn, for me it's working great with both the edge and beta, just not with v0.107.54. |
I'm running v0.108.0-a.989+d578c713 for 3 days now on two machines, and it's working great, so probably any changes made regarding this issue can be merged into stable branch. |
We have released a beta version that includes the fix. |
In beta 61 it is not fixed i tried for two days and the response times are increasing, in 59 i had no problem |
Where can i find that ver? In releases i see only b and stable releases |
@gitlaman That version is probably an edge release. |
You're 100% sure as I'm running that .61 beta for 3 days now on two instances are there's no problem at all. |
As @nevaran i did the same with quic and Rate Limit: 0 i have no problems with 61 beta with over 90k queries. |
Average upstream response time has been steadily increasing, resulting in noticeable slowdowns in DNS query resolution. This occurs despite no significant changes to my network setup. I've tried basic troubleshooting steps like updating to the latest version (v0.107.54) and restarting the service, but the issue persists. Below is a screenshot illustrating the increased response times and the current version of the software: |
Anyone know when the fix will be out in stable release? My upstream response times are now exceeding 900ms :O |
Running v0.107.55 and it seems to be working fine, no issues with either high or increasing response times. |
Fresh after install and config:
few hours later without any change:
Running v0.107.55 in docker (Home Assistant) |
Try the beta Version: vo. 108.0-b.59. For me it is working ok, i tried all others except edge versions. |
This issue is back after couple of days on latest version. No offense but I'm starting to doubt the competency of adguard devs.. this tool has one job and its failing at that and its taking so long to fix. Moving back to pihole. |
Just use te beta that i post in the last reply. It is easy to overwrite the old or new adguard executable, if you need help i can guide you. |
If you are using the latest version
The next time you encounter a slowdown, check the number of goroutines again. If you notice that the current number of goroutines is significantly higher than it was initially, please post the content of the following page: http://127.0.0.1:6060/debug/pprof/goroutine?debug=1. |
Prerequisites
I have checked the Wiki and Discussions and found no answer
I have searched other issues and found no duplicates
I want to report a bug and not ask a question or ask for help
I have set up AdGuard Home correctly and configured clients to use it. (Use the Discussions for help with installing and configuring clients.)
Platform (OS and CPU architecture)
Custom (please mention in the description)
Installation
Other (please mention in the description)
Setup
On one machine
AdGuard Home version
v0.107.45
Action
I used to use DoH of various DNS services and recently noticed it takes quite a while to load websites, so I decided to switch to old regular DNS, hoping to speed it up, but it didn't happen. Once added, upstream DNS starts increasing the average response time for more than 10x.
On the same machine where AGH is installed, running dnsperftest script multiple times a day, returns more or less consistent results:
While the current "Average upstream response time" in AGH looks like this (and progressively increases more and more):
When I'm using VPN and its DNS, website are loading noticeably faster. Is there something to change in AdGuard Home DNS settings shown below which should hopefully speed up the response time?
Expected result
Lower "Average upstream response time" over time and faster responses.
Actual result
"Average upstream response time" getting increased over time. Websites take quite a while to load.
Additional information and/or screenshots
AdGuard Home is installed on RPi 3B+ running DietPi (debian based).
The text was updated successfully, but these errors were encountered: