PowerDNS-recursor 4.0.4 exiting on signal 11 while using dq:addAnswer Lua code on FreeBSD 10.3

[marcvanhoutum]

Hello,

I've installed PowerDNS-recursor 4.0.4 on a new FreeBSD 10.3 server. I want to use a Lua script and the preresolve function to have the server answer with a different IP address for a record than is configured in it's zone. The Lua code is:

function preresolve(dq)
if(dq.qname:equal("support.customer.com"))
    then
        pdnslog("Got query for "..dq.qname:toString(), pdns.loglevels.All)
        dq:addAnswer(pdns.A, "10.20.30.40")
        return true;
    end
    return false;
end

However, while querying support.customer.com the following behavior is displayed:

Jan 20 07:36:21 dns1 pdns_recursor[12428]: 1 [2/1] question for 'support.customer.com|A' from 127.0.0.1
Jan 20 07:36:21 dns1 pdns_recursor[12428]: Got query for support.customer.com.
Jan 20 07:36:21 dns1 kernel: pid 12428 (pdns_recursor), uid 120: exited on signal 11

It seems that the 'dq:addAnswer' line is causing it to crash, commenting out the line logs the query using the pdnslog function, and the recursor sends an empty answer as expected. Other queries are also working just fine. The daemon is compiled using Poudriere and clang 3.4.1.

I've been unable to generate a core file so I've attached GDB to the running process, but it didn't seem to provide much useful information, see attached file PowerDNS-recursor GDB.txt. A 'truss -f' returns the following after initialising:

24576: SIGNAL 11 (SIGSEGV)
24576: setgid(0x78) ERR#4 'Interrupted system call'
24576: process killed, signal = 11

Running the Lua script on a Linux Debian 8 server seems to work just fine so this seems to be FreeBSD related.

Best regards,
Marc van Houtum

PowerDNS-recursor GDB.txt

[pieterlexis]

Thanks for this. Would you mind creating a new build with -ggdb -O0 and provide us with a GDB stacktrace from that build?

[marcvanhoutum]

Sure. Configure options:

configure: Configuration summary
configure: =====================
configure:
configure: PowerDNS Recursor configured with:  '--sysconfdir=/usr/local/etc/pdns' '--with-lua' '--without-luajit' '--prefix=/usr/local' '--localstatedir=/var' '--mandir=/usr/local/man' '--disable-silent-rules' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd10.3' 'build_alias=amd64-portbld-freebsd10.3' 'CC=cc' 'CFLAGS=-O2 -pipe  -ggdb -O0 -fstack-protector -isystem /usr/local/include -fno-strict-aliasing' 'LDFLAGS=  -fstack-protector' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-isystem /usr/local/include' 'CXX=c++' 'CXXFLAGS=-O2 -pipe -ggdb -O0 -fstack-protector -isystem /usr/local/include -fno-strict-aliasing  -D_GLIBCXX_USE_C99 -ggdb -O0 -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'
configure:
configure: CC: cc
configure: CXX: c++
configure: LD: /usr/bin/ld
configure: CFLAGS:  -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 --param ssp-buffer-size=4 -fstack-protector -O2 -pipe  -ggdb -O0 -fstack-protector -isystem /usr/local/include -fno-strict-aliasing
configure: CPPFLAGS: -isystem /usr/local/include  -pthread
configure: CXXFLAGS:   -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 --param ssp-buffer-size=4 -fstack-protector -O2 -pipe -ggdb -O0 -fstack-protector -isystem /usr/local/include -fno-strict-aliasing  -D_GLIBCXX_USE_C99 -ggdb -O0 -isystem /usr/local/include -std=gnu++11
configure: LDFLAGS: -Wl,-z -Wl,relro -Wl,-z -Wl,now   -fstack-protector
configure: LIBS: -L/usr/local/lib -lboost_thread -lboost_system -pthread
configure: BOOST_CPPFLAGS:  -pthread
configure:
configure: Features enabled
configure: ----------------
configure: Lua: lua-5.2
configure: OpenSSL ECDSA: yes
configure: Protobuf: no
configure: systemd: no
configure: Context library: Boost context

GDB stacktrace:

Jan 24 14:15:26 1 [1/1] question for 'support.customer.com|A' from 127.0.0.1
Jan 24 14:15:26 Got query for support.customer.com.
[New Thread 803407400 (LWP 100423/pdns_recursor)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 803407400 (LWP 100423/pdns_recursor)]
0x000000080298140b in strlen () from /lib/libc.so.7
(gdb)
(gdb) bt
#0  0x000000080298140b in strlen () from /lib/libc.so.7
#1  0x0000000000433f75 in std::__1::char_traits<char>::eq ()
#2  0x0000000000629e2e in json11::Json::~Json ()
#3  0x0000000000629ccb in json11::Json::~Json ()
#4  0x0000000000629a6b in json11::Json::~Json ()
#5  0x0000000000629723 in json11::Json::~Json ()
#6  0x000000000062956b in json11::Json::~Json ()
#7  0x0000000000629518 in json11::Json::~Json ()
#8  0x00000008016c3c45 in lua_getinfo () from /usr/local/lib/liblua-5.2.so
#9  0x00000008016d14bb in lua_close () from /usr/local/lib/liblua-5.2.so
#10 0x00000008016c41e2 in lua_getinfo () from /usr/local/lib/liblua-5.2.so
#11 0x00000008016c36f3 in lua_getinfo () from /usr/local/lib/liblua-5.2.so
#12 0x00000008016c46a8 in lua_yieldk () from /usr/local/lib/liblua-5.2.so
#13 0x00000008016bf4c1 in lua_pcallk () from /usr/local/lib/liblua-5.2.so
#14 0x00000000005cb40b in json11::Json::~Json ()
#15 0x00000000005cb1a2 in json11::Json::~Json ()
#16 0x00000000005caf7d in json11::Json::~Json ()
#17 0x00000000005cae5c in json11::Json::~Json ()
#18 0x00000000005c86e2 in json11::Json::~Json ()
#19 0x00000000005363b4 in json11::Json::~Json ()
#20 0x0000000000537273 in json11::Json::~Json ()
#21 0x000000000073171a in json11::Json::~Json ()
#22 0x000000000079b508 in std::bad_cast::bad_cast ()
#23 0x000000000079b2cd in std::bad_cast::bad_cast ()
#24 0x000000000070555b in json11::Json::~Json ()
#25 0x00000000007053ab in json11::Json::~Json ()
#26 0x00000008014b12bf in make_fcontext () from /usr/local/lib/libboost_context.so.1.63.0
#27 0x0000000000000000 in ?? ()
(gdb)

[Habbie]

Can you please -rebuild- with -O0 -ggdb and then give us an all-thread backtrace?

[marcvanhoutum]

Hi Habbie,

I'm pretty sure I have rebuild with those options in both CFLAGS and CXXFLAGS. I've checked (and attached) the poudriere build log to make sure: Poudiere_powerdns-recursor-4.0.4.txt

Attached is also the new GDB trace, please note the built timestamp: GDB-trace.txt

Would you like me to circumvent Poudriere and build from source directly?

[Habbie]

Hi Marc,

I see the -O0 in there but the gdb output is all wrong, it claims basically everything is Json11. I do wonder if building and testing directly from the source would be better.

[marcvanhoutum]

Hi Habbie,

As requested, I have compiled powerdns-recursor 4.0.4 on the server. I had to apply this patch for it to initialise properly.

There's a lot more information this time, I've put it in the document here: pdns-recursor 4.0.4 GDB.txt

[ozkankirik]

I'm facing same problem also. Environment FreeBSD 10.3-RELEASE-p17 amd64, powerdns-recursor 4.0.4_2 LUA enabled.

[omoerbeek]

This seems to be fixed in

FreeBSD freebsd 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

$ dig @127.0.0.1  support.customer.com

; <<>> DiG 9.16.2 <<>> @127.0.0.1 support.customer.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21085
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;support.customer.com.          IN      A

;; ANSWER SECTION:
support.customer.com.   3600    IN      A       10.20.30.40

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed May 20 12:20:35 UTC 2020
;; MSG SIZE  rcvd: 65