Bug: large number of addDomainSpoof lines break daemon start

[H3ik0]

I try to build an ad- malware- and tracker-blocking dns-resolver.
There are 30807 lines of domains in a domains.txt (1 domain = 1 line)

Merged source files:
https://adaway.org/hosts.txt
http://hosts-file.net/ad_servers.txt
http://www.malwaredomainlist.com/hostslist/hosts.txt
http://pgl.yoyo.org/adservers/serverlist.php
http://someonewhocares.org/hosts/hosts
http://winhelp2002.mvps.org/hosts.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malware.txt

The dnsdist.conf:

newServer("8.8.8.8")
-- tested with and without cache
-- pc = newPacketCache(10000, 86400, 0, 60, 60)
-- getPool(""):setCache(pc)
for line in io.lines("/etc/dnsdist/domains.txt") do addDomainSpoof(line, "127.0.0.1") end

I always get this in syslog:

Failed to start DNS Loadbalancer.
dnsdist.service: Unit entered failed state.
dnsdist.service: Failed with result 'timeout'.
dnsdist.service: Service hold-off time over, scheduling restart.

With a less number of lines lf.e. 200 all is working fine.
Machine is a virtual root server with ssd and multicore.

BTW: there is also no way to reduce lines, because Regex f.e. m/^ads?./ ist not supported for addDomainSpoof.

[Habbie]

This is systemd considering your dnsdist startup failed because it took too long. SSD and multicore don't help here - this all happens on a single CPU core. Perhaps you can increase the timeout in your systemd unit file?

[ahupowerdns]

The bigger thing is that our rules are not intended for having 30000 lines (which we have not documented, by the way). The better way to do this is like this (more or less):

domains=newSuffixMatchNode()
for line in io.lines("/etc/dnsdist/domains.txt") do domains:add(line) end
addAction(SuffixMatchNodeRule(domains), SpoofAction("127.0.0.1"))

This will 1) work 2) be much much faster.
an open question is why we take soo long to add 30000 rules. It should not take that long.

[Habbie]

@ahupowerdns I agree, I was surprised to read the cutoff point might be at 200 already, I would expect us to be able to do thousands if not tens of thousands within the few seconds that is no doubt our systemd timeout default.

[H3ik0]

@ahupowerdns

domains=newSuffixMatchNode();
for line in io.lines("/etc/dnsdist/domains.txt") do domains:add(line) end
addAction(SuffixMatchNodeRule(domains), SpoofAction("127.0.0.1"))

-->

Fatal error: Unable to convert parameter from no value to 7DNSName
Fatal error: Unable to convert parameter from no value to 7DNSName
Control process exited, code=exited status=1` 

dnsdist version 1.1.0-2
Linux server1 4.10.0-24-generic #28-Ubuntu SMP Wed Jun 14 08:14:34 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

To compare:

time systemctl start unbound
real	0m0.329s
user	0m0.000s
sys	0m0.004s

with include: /etc/unbound/unbound-local-block.conf
like: local-data: "02gzx.voluumtrk.com A 127.0.0.1"

I'm not sure how to fix this.

[ahupowerdns]

Hi h3ik0, try: domains:add(newDNSName(line))
Also, we highly recommend unbound if you are not pleased with the help we provide here. Good luck.

[H3ik0]

domains=newSuffixMatchNode(); for line in io.lines("/etc/dnsdist/domains.txt") do domains:add(**newDNSName(line)**) end addAction(SuffixMatchNodeRule(domains), SpoofAction("127.0.0.1"))
successfully fast start :) Thank you.

BTW I want to replace unbound. The time was to show the machine has no performance issue.

No point to be angry please

[ahupowerdns]

Ok, apologies. Glad it worked. We are looking into why adding rules is so slow at larger numbers.

[H3ik0]

Thats great thank you. For your information: dig request time from cache: unbound 6ms dnsdist 2ms. Cool. Good work.