-
-
Notifications
You must be signed in to change notification settings - Fork 371
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
failed to send TCP(read_write) packet (Connection prematurely closed by remote server) #1237
Comments
Hi, this seems to be TCP related. If this is not the environment's configuration (everything over TCP), maybe only select queries are strictly over TCP, like the one from opera I see in the forum? But then the amount of said TCP traffic still pushes Unbound near its configured limits? "Regular" TCP traffic for Unbound (at least for your configuration where I don't see TLS used) would be UDP queries that are retried over TCP because the answer would not fit in the UDP buffer. These would get an immediate cache response because Unbound would already have the data in the cache (it concluded that the answer would not fit in UDP already). Actually I am pretty sure this is what is happening by looking at the log fragments from the forum:
200 msec is the minimum amount of time Unbound will use under pressure. The default (starting) value is 30 secs. So I would bump
Also, I see that this is mostly dnsmasq talking to Unbound? |
Hi @gthess, I can provide full logs - I am not too sure how to share as the unbound log is 215,400 KB and the FTL log from Pi-hole is like 69,021 KB. I could use a shared cloud location. Is there an address whereto I could send the password? From the Pi-hole forum, which perhaps could throw more light on the the issue: and, indeed unbound never sent the reply to the A query downstream: just before the last four lines, we see that the cache is populated with ;; ANSWER SECTION: ;; AUTHORITY SECTION: ;; ADDITIONAL SECTION: Feb 07 13:14:47 unbound[1989:0] info: chased extract ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 0 ;; ANSWER SECTION: ;; AUTHORITY SECTION: ;; ADDITIONAL SECTION: Feb 07 13:14:47 unbound[1989:0] info: chased extract ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 0 ;; ANSWER SECTION: ;; AUTHORITY SECTION: ;; ADDITIONAL SECTION: so unbound indeed has the necessary data to respond to your query. For your question on "why does Unbound get TCP queries in the first place" I presume this is how the solution is put together. I'll post your question to the Pi-hole developers. Likewise for your two last questions: I'll post your questions to the Pi-hole developers. Regards - Steen |
For now the logs are not necessary as I got a glimpse from the forum and I do believe it is the tcp usage that makes Unbound drop connections. You can go ahead and configure Unbound with:
100 is a bit excessive (and it will need more memory), but at least you can try and see if the problem keeps occurring or not.
Then if the problem goes away, the monitoring script should tell you the maximum of incoming tcp connections in your environment. |
Hi, Reloaded and Restarted Unbound, but when running "while true; do date; unbound-control stats | grep total.tcpusage; sleep 1; done" Any help appreciated. |
Sorry for the typo! (Which seemed to happened all over the issue 🤦 ) You are correct, I would assume that Unbound runs as the
The "script" runs in a loop and prints the date and the tcp usage from Unbound.
the script's output will be on the terminal and also written to the 'tcp-usage.log' file. I used |
Hi, So far no error...will continue... |
Hi, |
Good to hear, so it seems it is related to the incoming tcp buffers. Maybe the LAN instance gets more clients, or has a slower connection to the outside, or both? |
Hi,
I wish that would be the case.
The LAN got 6 devices and the dl/ul bandwidth are stable (monitored) 24*7.
No useful from the script as all tcpusage shows 0.
|
That's unfortunate, tcpusage is a value in time that probably with the 1 second granularity and the low number of clients(incoming queries) does not reveal the TCP spikes that could happen.
|
Hi, |
Hi, 2025-02-14 19:46:16.039 WARNING Connection error (127.0.0.1#5335): TCP connection failed while receiving payload length from upstream (Connection prematurely closed by remote server) Fri 14 Feb 19:59:35 GMT 2025 This I got with |
Another user running Unbound who is facing this same "WARNING: Connection error (127.0.0.1#5335): TCP connection failed while receiving payload length from upstream (Resource temporarily unavailable)" error in Pi-Hole v6. System Details: Pi-Hole was updated from v5 to v6 several days ago. Ever since then been receiving a Pi-Hole Diagnosis Error from time to time indicting (for example) the following:
Like others with this issue since moving to Pi-Hole v6, have been experimenting with the incoming-num-tcp: (number) value in the /etc/unbound/unbound.conf.d/pi-hole.conf file. Restarting Unbound each time the incoming-num-tcp value is changed. Have tried the following values, one at at time: 10, 40, 50, 100. Each time a connection error message eventually shows up in the Pi-Hole Diagnostic Error section. Data from the Pi-Hole FLT.log file:
Are there any additional suggestions to try or troubleshooting steps to take? |
Hi,
My first try to report an issue here.
I am having a couple of Raspberry Pi running Pi-hole and is using Unbound as a recursive DNS server.
I and others have seen this message
WARNING Connection error (127.0.0.1#5335): failed to send TCP(read_write) packet (Connection prematurely closed by remote server)
in the Pi-holes FTL.log, but often just disregarded this.However, a few weeks ago I started a discussion around this at Pi-holes Beta 6 forum. Which lead to a deeper investigation by the people behind Pi-hole and my self (let me just say that I am not a software/developer person).
Yesterday one of the developer concluded this must most likely be an Unbound bug, here the link to the message https://discourse.pi-hole.net/t/two-pi-hole-instances-one-with-failed-to-send/75057/44?u=seh2000.
Unfortunately the log files especial the unbound.log are rather large, however in the link above there are extracts from different log files.
Kind regards - Steen
System:
Unbound version: 1.17.1
OS:
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm
Kernel: Linux 6.6.74+rpt-rpi-v8
Architecture: arm64
HW: Raspberry Pi 4 Model B Rev 1.5
unbound -V
output:Version 1.17.1
Configure line: --build=aarch64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=${prefix}/lib/aarch64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --with-pythonmodule --with-pyunbound --enable-subnet --enable-dnstap --enable-systemd --with-libnghttp2 --with-chroot-dir= --with-dnstap-socket-path=/run/dnstap.sock --disable-rpath --with-pidfile=/run/unbound.pid --with-libevent --enable-tfo-client --with-rootkey-file=/usr/share/dns/root.key --disable-flto --enable-tfo-server
Linked libs: libevent 2.1.12-stable (it uses epoll), OpenSSL 3.0.15 3 Sep 2024
Linked modules: dns64 python subnetcache respip validator iterator
TCP Fastopen feature available
BSD licensed, see LICENSE in source package for details.
Report bugs to unbound-bugs@nlnetlabs.nl or https://github.com/NLnetLabs/unbound/issues
unbound conf:
admin@Pi4-1:~ $ sudo grep -v '#|^$' -R /etc/unbound/unbound.conf*
/etc/unbound/unbound.conf:include-toplevel: "/etc/unbound/unbound.conf.d/*.conf"
/etc/unbound/unbound.conf.d/root-auto-trust-anchor-file.conf:server:
/etc/unbound/unbound.conf.d/root-auto-trust-anchor-file.conf: auto-trust-anchor-file: "/var/lib/unbound/root.key"
/etc/unbound/unbound.conf.d/remote-control.conf:remote-control:
/etc/unbound/unbound.conf.d/remote-control.conf: control-enable: yes
/etc/unbound/unbound.conf.d/remote-control.conf: control-interface: /run/unbound.ctl
/etc/unbound/unbound.conf.d/pi-hole.conf:server:
/etc/unbound/unbound.conf.d/pi-hole.conf: logfile: "/var/log/unbound/unbound.log"
/etc/unbound/unbound.conf.d/pi-hole.conf: log-time-ascii: yes
/etc/unbound/unbound.conf.d/pi-hole.conf: log-queries: yes
/etc/unbound/unbound.conf.d/pi-hole.conf: log-replies: yes
/etc/unbound/unbound.conf.d/pi-hole.conf: verbosity: 5
/etc/unbound/unbound.conf.d/pi-hole.conf: interface: 127.0.0.1
/etc/unbound/unbound.conf.d/pi-hole.conf: port: 5335
/etc/unbound/unbound.conf.d/pi-hole.conf: do-ip4: yes
/etc/unbound/unbound.conf.d/pi-hole.conf: do-udp: yes
/etc/unbound/unbound.conf.d/pi-hole.conf: do-tcp: yes
/etc/unbound/unbound.conf.d/pi-hole.conf: do-ip6: no
/etc/unbound/unbound.conf.d/pi-hole.conf: prefer-ip6: no
/etc/unbound/unbound.conf.d/pi-hole.conf: harden-glue: yes
/etc/unbound/unbound.conf.d/pi-hole.conf: harden-dnssec-stripped: yes
/etc/unbound/unbound.conf.d/pi-hole.conf: use-caps-for-id: no
/etc/unbound/unbound.conf.d/pi-hole.conf: edns-buffer-size: 1232
/etc/unbound/unbound.conf.d/pi-hole.conf: prefetch: yes
/etc/unbound/unbound.conf.d/pi-hole.conf: num-threads: 1
/etc/unbound/unbound.conf.d/pi-hole.conf: so-rcvbuf: 1m
/etc/unbound/unbound.conf.d/pi-hole.conf: private-address: 192.168.0.0/16
/etc/unbound/unbound.conf.d/pi-hole.conf: private-address: 169.254.0.0/16
/etc/unbound/unbound.conf.d/pi-hole.conf: private-address: 172.16.0.0/12
/etc/unbound/unbound.conf.d/pi-hole.conf: private-address: 10.0.0.0/8
/etc/unbound/unbound.conf.d/pi-hole.conf: private-address: fd00::/8
/etc/unbound/unbound.conf.d/pi-hole.conf: private-address: fe80::/10
Additional information
The text was updated successfully, but these errors were encountered: