Add a trailing dot to DNS domains by default to speed up discovery when a search domain is configured

[dozernz]

Summary

If a DNS search domain is configured in the system resolver, gobuster will re-issue queries for every NXDOMAIN result with the DNS search domain appended. This occurs even when using a custom resolver. This PR simply adds a trailing dot to the domain if one isn't already present, so the search domain isn't applied. It also adds a flag to revert this behaviour if required.

While it is possible to work around by removing the search domain from resolv, or stipulating a base domain with a trailing dot, this is not something I always remember to do.

Details

The current behaviour results in an extraneous query for each word that doesn't exist if a DNS search domain is configured, and the domain is not supplied with a trailing dot.

For example, with the following /etc/resolv.conf:

# cat /etc/resolv.conf
nameserver 1.1.1.1
nameserver 8.8.8.8
search localdomain

Running a simple test with four subdomains:

# printf 'www\ninvalid\ninvalid2\ninvalid3' | gobuster dns -z -r 8.8.8.8 -d example.com -w -
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Domain:     example.com
[+] Threads:    10
[+] Resolver:   8.8.8.8
[+] Timeout:    1s
[+] Wordlist:   stdin (pipe)
===============================================================
Starting gobuster in DNS enumeration mode
===============================================================
Found: www.example.com

===============================================================
Finished
===============================================================

This ends up sending 20 queries. All queries that returned NXDOMAIN are re-issued with the search domain:

00:27:29.990033 IP <ip>.37548 > 8.8.8.8.53: 28077+ [1au] AAAA? fa79cba0-e7fc-4920-8a40-0be0858a50ae.example.com. (77)
00:27:29.990246 IP <ip>.44477 > 8.8.8.8.53: 61276+ [1au] A? fa79cba0-e7fc-4920-8a40-0be0858a50ae.example.com. (77)
00:27:30.001803 IP <ip>.34009 > 8.8.8.8.53: 23905+ [1au] AAAA? fa79cba0-e7fc-4920-8a40-0be0858a50ae.example.com.localdomain. (89)
00:27:30.001862 IP <ip>.36322 > 8.8.8.8.53: 64374+ [1au] A? fa79cba0-e7fc-4920-8a40-0be0858a50ae.example.com.localdomain. (89)
00:27:30.011556 IP <ip>.37348 > 8.8.8.8.53: 6400+ [1au] AAAA? example.com. (40)
00:27:30.011571 IP <ip>.44812 > 8.8.8.8.53: 65030+ [1au] A? example.com. (40)
00:27:30.013530 IP <ip>.43300 > 8.8.8.8.53: 35768+ [1au] AAAA? www.example.com. (44)
00:27:30.013665 IP <ip>.54630 > 8.8.8.8.53: 52527+ [1au] AAAA? invalid.example.com. (48)
00:27:30.013758 IP <ip>.52362 > 8.8.8.8.53: 63530+ [1au] AAAA? invalid2.example.com. (49)
00:27:30.013868 IP <ip>.46685 > 8.8.8.8.53: 53334+ [1au] AAAA? invalid3.example.com. (49)
00:27:30.013945 IP <ip>.40598 > 8.8.8.8.53: 33567+ [1au] A? www.example.com. (44)
00:27:30.013995 IP <ip>.45437 > 8.8.8.8.53: 63327+ [1au] A? invalid.example.com. (48)
00:27:30.014041 IP <ip>.55010 > 8.8.8.8.53: 18785+ [1au] A? invalid2.example.com. (49)
00:27:30.014092 IP <ip>.54091 > 8.8.8.8.53: 6625+ [1au] A? invalid3.example.com. (49)
00:27:30.023172 IP <ip>.38643 > 8.8.8.8.53: 132+ [1au] AAAA? invalid2.example.com.localdomain. (61)
00:27:30.023242 IP <ip>.45824 > 8.8.8.8.53: 13507+ [1au] A? invalid2.example.com.localdomain. (61)
00:27:30.023310 IP <ip>.51039 > 8.8.8.8.53: 24261+ [1au] AAAA? invalid.example.com.localdomain. (60)
00:27:30.023393 IP <ip>.40031 > 8.8.8.8.53: 17467+ [1au] A? invalid.example.com.localdomain. (60)
00:27:30.027140 IP <ip>.39405 > 8.8.8.8.53: 18923+ [1au] AAAA? invalid3.example.com.localdomain. (61)
00:27:30.027284 IP <ip>.56024 > 8.8.8.8.53: 23436+ [1au] A? invalid3.example.com.localdomain. (61)

If we either remove the search domain from resolv, or supply a domain with a trailing dot (-d example.com.) , this is cut down to 12 queries:

00:31:23.982011 IP <ip>.37986 > 8.8.8.8.53: 11869+ [1au] AAAA? 85d5c497-988b-492f-a63a-68c9cf26282b.example.com. (77)
00:31:23.982059 IP <ip>.40365 > 8.8.8.8.53: 37511+ [1au] A? 85d5c497-988b-492f-a63a-68c9cf26282b.example.com. (77)
00:31:23.994870 IP <ip>.34752 > 8.8.8.8.53: 9401+ [1au] AAAA? example.com. (40)
00:31:23.994909 IP <ip>.43539 > 8.8.8.8.53: 47663+ [1au] A? example.com. (40)
00:31:23.996597 IP <ip>.36751 > 8.8.8.8.53: 33425+ [1au] AAAA? www.example.com. (44)
00:31:23.996754 IP <ip>.36513 > 8.8.8.8.53: 10848+ [1au] AAAA? invalid.example.com. (48)
00:31:23.996883 IP <ip>.60591 > 8.8.8.8.53: 54684+ [1au] AAAA? invalid2.example.com. (49)
00:31:23.997007 IP <ip>.59325 > 8.8.8.8.53: 26004+ [1au] AAAA? invalid3.example.com. (49)
00:31:23.997071 IP <ip>.57571 > 8.8.8.8.53: 23502+ [1au] A? www.example.com. (44)
00:31:23.997144 IP <ip>.44323 > 8.8.8.8.53: 3719+ [1au] A? invalid.example.com. (48)
00:31:23.997240 IP <ip>.57715 > 8.8.8.8.53: 26541+ [1au] A? invalid2.example.com. (49)
00:31:23.997308 IP <ip>.33131 > 8.8.8.8.53: 21877+ [1au] A? invalid3.example.com. (49)

On a larger wordlist, eliminating these extraneous queries results in a significant speedup:

$ cat /usr/share/dict/american-english  | shuf -n10000  > wl
$ for i in {1..5}; do time gobuster dns -z -r 8.8.8.8 -d example.com -w wl ;done 2>&1 | grep real
real    0m14.862s
real    0m14.643s
real    0m14.365s
real    0m14.282s
real    0m14.136s
#reshuffle wordlist
$ cat /usr/share/dict/american-english  | shuf -n10000  > wl
# note the trailing dot on the domain
$ for i in {1..5}; do time gobuster dns -z -r 8.8.8.8 -d example.com. -w wl ;done 2>&1 | grep real 
real    0m8.190s
real    0m7.896s
real    0m7.648s
real    0m7.459s
real    0m7.359s

With the code from the PR applied we don't need to supply the trailing dot by default.:

$ cat /usr/share/dict/american-english  | shuf -n10000  > wl
$ for i in {1..5}; do time ./gobuster/gobuster dns -z -r 8.8.8.8 -d example.com -w wl ;done 2>&1 | grep real
real    0m8.254s
real    0m8.030s
real    0m7.823s
real    0m7.591s
real    0m7.491s

[firefart]

nice catch thanks!