Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FTL crash when rate-limiting #1189

Closed
disappointingsupernova opened this issue Oct 1, 2021 · 6 comments · Fixed by #1197 or #1199
Closed

FTL crash when rate-limiting #1189

disappointingsupernova opened this issue Oct 1, 2021 · 6 comments · Fixed by #1197 or #1199
Labels
Bug Fixed in next release

Comments

@disappointingsupernova
Copy link

Versions

Pi-hole version is v5.5 (Latest: v5.5)
AdminLTE version is v5.7 (Latest: v5.7)
FTL version is release v5.10.2 (Latest: v5.10.2)

Platform

  • OS and version: Ubuntu 20.04
  • Platform: Hyper-V VM

Actual behavior / bug

Pihole-FTL will crash 1 min after restarting

Steps to reproduce

Its occurs 1m after restarting Pihole-FTL service - This has only occured since updating FTL to 5.10.1

Debug Token

  • URL: offline (behind fw)

Additional context

[2021-10-01 06:35:47.080 82997/T82998] Listening on port 4711 for incoming IPv4 telnet connections
[2021-10-01 06:35:47.080 82997/T82999] Listening on port 4711 for incoming IPv6 telnet connections
[2021-10-01 06:35:47.081 82997/T83000] Listening on Unix socket
[2021-10-01 06:35:47.086 82997M] Reloading DNS cache
[2021-10-01 06:35:48.180 82997/T83001] Compiled 1 whitelist and 4 blacklist regex filters for 10 clients in 35.7 msec
[2021-10-01 06:35:54.089 82997M] Blocking status is enabled
[2021-10-01 06:35:54.206 82997M] Reloading DNS cache
[2021-10-01 06:35:55.237 82997/T83001] SQLite3 message: file renamed while open: /etc/pihole/gravity.db (28)
[2021-10-01 06:35:55.276 82997/T83001] Compiled 1 whitelist and 4 blacklist regex filters for 10 clients in 36.5 msec
[2021-10-01 06:35:59.097 82997M] Blocking status is enabled
[2021-10-01 06:36:35.851 82997M] Resizing "FTL-queries" from 19726336 to (356352 * 56) == 19955712 (/dev/shm: 20.2MB used, 473.3MB total, FTL uses 20.2MB)
[2021-10-01 06:37:33.780 82997M] Rate-limiting 192.168.1.140 for 106 seconds
[2021-10-01 06:37:33.796 82997M] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-10-01 06:37:33.796 82997M] ----------------------------> FTL crashed! <----------------------------
[2021-10-01 06:37:33.796 82997M] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-10-01 06:37:33.796 82997M] Please report a bug at https://github.com/pi-hole/FTL/issues
[2021-10-01 06:37:33.796 82997M] and include in your report already the following details:
[2021-10-01 06:37:33.796 82997M] FTL has been running for 109 seconds
[2021-10-01 06:37:33.796 82997M] FTL branch: v5.10.2
[2021-10-01 06:37:33.796 82997M] FTL version: v5.10.2
[2021-10-01 06:37:33.796 82997M] FTL commit: 2f2b5bf
[2021-10-01 06:37:33.797 82997M] FTL date: 2021-09-30 20:55:27 +0100
[2021-10-01 06:37:33.797 82997M] FTL user: started as pihole, ended as pihole
[2021-10-01 06:37:33.797 82997M] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2021-10-01 06:37:33.797 82997M] Process details: MID: 82997
[2021-10-01 06:37:33.797 82997M] PID: 82997
[2021-10-01 06:37:33.797 82997M] TID: 82997
[2021-10-01 06:37:33.797 82997M] Name: pihole-FTL
[2021-10-01 06:37:33.797 82997M] Received signal: Segmentation fault
[2021-10-01 06:37:33.798 82997M] at address: (nil)
[2021-10-01 06:37:33.798 82997M] with code: SEGV_MAPERR (Address not mapped to object)
[2021-10-01 06:37:33.798 82997M] Backtrace:
[2021-10-01 06:37:33.799 82997M] B[0000]: /usr/bin/pihole-FTL(generate_backtrace+0x41) [0x5610abe22881]
[2021-10-01 06:37:33.802 82997M] L[0000]: N/A (0x5f881 -> addr2line 0x5f881 -e /usr/bin/pihole-FTL)
[2021-10-01 06:37:33.802 82997M] B[0001]: /usr/bin/pihole-FTL(+0x5fd88) [0x5610abe22d88]
[2021-10-01 06:37:33.804 82997M] L[0001]: N/A (0x5fd88 -> addr2line 0x5fd88 -e /usr/bin/pihole-FTL)
[2021-10-01 06:37:33.805 82997M] B[0002]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f321594b3c0]
[2021-10-01 06:37:33.805 82997M] B[0003]: /usr/bin/pihole-FTL(_FTL_make_answer+0x33d) [0x5610abe124cd]
[2021-10-01 06:37:33.807 82997M] L[0003]: N/A (0x4f4cd -> addr2line 0x4f4cd -e /usr/bin/pihole-FTL)
[2021-10-01 06:37:33.807 82997M] B[0004]: /usr/bin/pihole-FTL(receive_query+0x6df) [0x5610abe5561f]
[2021-10-01 06:37:33.810 82997M] L[0004]: N/A (0x9261f -> addr2line 0x9261f -e /usr/bin/pihole-FTL)
[2021-10-01 06:37:33.810 82997M] B[0005]: /usr/bin/pihole-FTL(+0x85df5) [0x5610abe48df5]
[2021-10-01 06:37:33.814 82997M] L[0005]: N/A (0x85df5 -> addr2line 0x85df5 -e /usr/bin/pihole-FTL)
[2021-10-01 06:37:33.814 82997M] B[0006]: /usr/bin/pihole-FTL(main_dnsmasq+0x12ad) [0x5610abe4ac4d]
[2021-10-01 06:37:33.817 82997M] L[0006]: N/A (0x87c4d -> addr2line 0x87c4d -e /usr/bin/pihole-FTL)
[2021-10-01 06:37:33.817 82997M] B[0007]: /usr/bin/pihole-FTL(main+0x127) [0x5610abe09877]
[2021-10-01 06:37:33.820 82997M] L[0007]: N/A (0x46877 -> addr2line 0x46877 -e /usr/bin/pihole-FTL)
[2021-10-01 06:37:33.820 82997M] B[0008]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f321576b0b3]
[2021-10-01 06:37:33.820 82997M] B[0009]: /usr/bin/pihole-FTL(_start+0x2a) [0x5610abe0998a]
[2021-10-01 06:37:33.822 82997M] L[0009]: N/A (0x4698a -> addr2line 0x4698a -e /usr/bin/pihole-FTL)
[2021-10-01 06:37:33.823 82997M] ------ Listing content of directory /dev/shm ------
[2021-10-01 06:37:33.823 82997M] File Mode User:Group Size Filename
[2021-10-01 06:37:33.823 82997M] rwxrwxrwx root:root 280 .
[2021-10-01 06:37:33.823 82997M] rwxr-xr-x root:root 4K ..
[2021-10-01 06:37:33.823 82997M] rw------- pihole:pihole 4K FTL-per-client-regex
[2021-10-01 06:37:33.823 82997M] rw------- pihole:pihole 4K FTL-dns-cache
[2021-10-01 06:37:33.823 82997M] rw------- pihole:pihole 8K FTL-overTime
[2021-10-01 06:37:33.823 82997M] rw------- pihole:pihole 20M FTL-queries
[2021-10-01 06:37:33.823 82997M] rw------- pihole:pihole 20K FTL-upstreams
[2021-10-01 06:37:33.824 82997M] rw------- pihole:pihole 356K FTL-clients
[2021-10-01 06:37:33.824 82997M] rw------- pihole:pihole 25K FTL-domains
[2021-10-01 06:37:33.824 82997M] rw------- pihole:pihole 82K FTL-strings
[2021-10-01 06:37:33.824 82997M] rw------- pihole:pihole 12 FTL-settings
[2021-10-01 06:37:33.824 82997M] rw------- pihole:pihole 240 FTL-counters
[2021-10-01 06:37:33.824 82997M] rw------- pihole:pihole 88 FTL-lock
[2021-10-01 06:37:33.824 82997M] rwx------ root:root 80 multipath
[2021-10-01 06:37:33.825 82997M] ---------------------------------------------------
[2021-10-01 06:37:33.825 82997M] Please also include some lines from above the !!!!!!!!! header.
[2021-10-01 06:37:33.825 82997M] Thank you for helping us to improve our FTL engine!
[2021-10-01 06:37:33.825 82997M] Waiting for threads to join
[2021-10-01 06:37:33.825 82997M] Thread telnet-IPv4 (0) is idle, terminating it.
[2021-10-01 06:37:33.825 82997M] Thread telnet-IPv6 (1) is idle, terminating it.
[2021-10-01 06:37:33.825 82997M] Thread telnet-socket (2) is idle, terminating it.
[2021-10-01 06:37:33.826 82997M] Thread database (3) is idle, terminating it.
[2021-10-01 06:37:33.826 82997M] Thread housekeeper (4) is idle, terminating it.
[2021-10-01 06:37:33.826 82997M] Thread DNS client (5) is idle, terminating it.
[2021-10-01 06:37:33.827 82997M] All threads joined
[2021-10-01 06:37:33.837 82997M] ########## FTL terminated after 1m 49s (code 1)! ##########
@DL6ER
Copy link
Member

DL6ER commented Oct 1, 2021

Sorry für the inconvenience. What does

addr2line 0x5f881 -e /usr/bin/pihole-FTL
addr2line 0x5fd88 -e /usr/bin/pihole-FTL
addr2line 0x4f4cd -e /usr/bin/pihole-FTL

say?

@disappointingsupernova
Copy link
Author

Hey there, no inconvenience at all.

root@pihole2:~# addr2line 0x5f881 -e /usr/bin/pihole-FTL
/root/project/src/signals.c:99

root@pihole2:~# addr2line 0x5fd88 -e /usr/bin/pihole-FTL
/root/project/src/signals.c:242

root@pihole2:~# addr2line 0x4f4cd -e /usr/bin/pihole-FTL
/root/project/src/dnsmasq_interface.c:316 (discriminator 2)

Just an FYI, this is now occurring on 3 of my 4 piholes...

@disappointingsupernova
Copy link
Author

So an update on this; I've disabled rate limiting

cat /etc/pihole/pihole-FTL.conf
PRIVACYLEVEL=0
RATE_LIMIT=0/0

The server now seems be be staying up without issue.

Will report back

@DL6ER
Copy link
Member

DL6ER commented Oct 2, 2021

The bug happens in

FTL/src/dnsmasq_interface.c

Line 316 in 2f2b5bf

const bool hostname = strcmp(blockingreason, HOSTNAME) == 0;

I've disabled rate limiting
[...]
The server now seems be be staying up without issue.

This is a strange workaround... but it sparks an idea.

Could you check if running

pihole checkout ftl fix/rate_limit_crash

prevents crashes for you even with the default rate-limiting settings?

@hudson-thomas
Copy link

I've been having the same issue, and using the fix/rate_limit_crash branch has fixed it! I haven't tested it for too long, so I will report back if the same issue arises.

Thanks for all your hard work, @DL6ER!

@DL6ER DL6ER changed the title ########## FTL terminated after 1m 49s (code 1)! ########## FTL crash when rate-limiting Oct 3, 2021
@DL6ER DL6ER mentioned this issue Oct 3, 2021
Merged
5 tasks
This was linked to pull requests Oct 4, 2021
Fix rare crashes when rate-limiting happens before queries are blocked #1197
Merged
Tweak rate-limiting #1199
Merged
@disappointingsupernova
Copy link
Author

Hey @DL6ER - can confirm that all seems to be good with this patch and rate limiting switched on

Thanks very much for your work on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Fixed in next release
Projects
None yet
Milestone
No milestone
3 participants
@hudson-thomas @DL6ER @disappointingsupernova