Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add dns flag #350

Merged
merged 8 commits into from
Sep 19, 2023
Merged

add dns flag #350

merged 8 commits into from
Sep 19, 2023

Conversation

dogancanbakir
Copy link
Member

Closes #345.

CLI output:
$ go run . -u google.com -dns
  

  _____ _    _____  __
 |_   _| |  / __\ \/ /
   | | | |__\__ \>  < 
   |_| |____|___/_/\_\  v1.1.4

                projectdiscovery.io

[INF] Current tlsx version v1.1.4 (latest)
google.com:443
googleoptimize-cn.com
g.doubleclick.cn
safenup.googlesandbox-cn.com
youtubeeducation.com
google.pt
ampproject.org.cn
googleadapis.com
app-measurement-cn.com
ytimg.com
goo.gl
google.co.in
google.com.br
recaptcha.net.cn
googlesandbox-cn.com
gvt2.com
ggpht.cn
developers.android.google.cn
google.co.uk
google.com.ar
googlecnapps.cn
recaptcha-cn.net
googlecommerce.com
source.android.google.cn
origin-test.bdn.dev
google.com.au
google.nl
gcpcdn.gvt1.com
googletagmanager-cn.com
gvt1.com
www.goo.gl
developer.android.google.cn
ampproject.net.cn
googletraveladservices-cn.com
youtube-nocookie.com
googleapps-cn.com
doubleclick.cn
google.ca
google.it
google-analytics-cn.com
googlevads-cn.com
2mdn-cn.net
g.cn
google.com
cloud.google.com
youtubekids.com
youtu.be
youtube.com
google.es
google.pl
googlevideo.com
gkecnapps.cn
widevine.cn
admob-cn.com
crowdsource.google.com
google.com.mx
android.com
urchin.com
metric.gstatic.com
gcp.gvt2.com
g.doubleclick-cn.net
googlesyndication-cn.com
googleflights-cn.net
flash.android.com
google.fr
gstatic.cn
google.com.co
google.com.tr
gvt2-cn.com
google-analytics.com
appengine.google.com
bdn.dev
fls.doubleclick.cn
googletagservices-cn.com
safeframe.googlesyndication-cn.com
google.com.vn
doubleclick-cn.net
g.co
yt.be
datacompute.google.com
gstatic.com
gvt1-cn.com
android.clients.google.com
google.hu
fls.doubleclick-cn.net
googleadservices-cn.com
dartsearch-cn.net
url.google.com
google.cl
googleapis.cn
gstatic-cn.com
googledownloads.cn
googleapis-cn.com
google.co.jp
google.de
[INF] Connections made using crypto/tls: 1, zcrypto/tls: 0, openssl: 0
JSON output:
$ go run . -u google.com -dns -j | jq .
  

  _____ _    _____  __
 |_   _| |  / __\ \/ /
   | | | |__\__ \>  < 
   |_| |____|___/_/\_\  v1.1.4

                projectdiscovery.io

[INF] Current tlsx version v1.1.4 (latest)
[INF] Connections made using crypto/tls: 1, zcrypto/tls: 0, openssl: 0
{
  "timestamp": "2023-09-14T07:43:53.197154836Z",
  "host": "google.com",
  "ip": "172.217.18.14",
  "port": "443",
  "probe_status": true,
  "tls_version": "tls13",
  "cipher": "TLS_AES_128_GCM_SHA256",
  "not_before": "2023-08-14T08:16:28Z",
  "not_after": "2023-11-06T08:16:27Z",
  "subject_dn": "CN=*.google.com",
  "subject_cn": "*.google.com",
  "subject_an": [
    "*.google.com",
    "*.appengine.google.com",
    "*.bdn.dev",
    "*.origin-test.bdn.dev",
    "*.cloud.google.com",
    "*.crowdsource.google.com",
    "*.datacompute.google.com",
    "*.google.ca",
    "*.google.cl",
    "*.google.co.in",
    "*.google.co.jp",
    "*.google.co.uk",
    "*.google.com.ar",
    "*.google.com.au",
    "*.google.com.br",
    "*.google.com.co",
    "*.google.com.mx",
    "*.google.com.tr",
    "*.google.com.vn",
    "*.google.de",
    "*.google.es",
    "*.google.fr",
    "*.google.hu",
    "*.google.it",
    "*.google.nl",
    "*.google.pl",
    "*.google.pt",
    "*.googleadapis.com",
    "*.googleapis.cn",
    "*.googlevideo.com",
    "*.gstatic.cn",
    "*.gstatic-cn.com",
    "googlecnapps.cn",
    "*.googlecnapps.cn",
    "googleapps-cn.com",
    "*.googleapps-cn.com",
    "gkecnapps.cn",
    "*.gkecnapps.cn",
    "googledownloads.cn",
    "*.googledownloads.cn",
    "recaptcha.net.cn",
    "*.recaptcha.net.cn",
    "recaptcha-cn.net",
    "*.recaptcha-cn.net",
    "widevine.cn",
    "*.widevine.cn",
    "ampproject.org.cn",
    "*.ampproject.org.cn",
    "ampproject.net.cn",
    "*.ampproject.net.cn",
    "google-analytics-cn.com",
    "*.google-analytics-cn.com",
    "googleadservices-cn.com",
    "*.googleadservices-cn.com",
    "googlevads-cn.com",
    "*.googlevads-cn.com",
    "googleapis-cn.com",
    "*.googleapis-cn.com",
    "googleoptimize-cn.com",
    "*.googleoptimize-cn.com",
    "doubleclick-cn.net",
    "*.doubleclick-cn.net",
    "*.fls.doubleclick-cn.net",
    "*.g.doubleclick-cn.net",
    "doubleclick.cn",
    "*.doubleclick.cn",
    "*.fls.doubleclick.cn",
    "*.g.doubleclick.cn",
    "dartsearch-cn.net",
    "*.dartsearch-cn.net",
    "googletraveladservices-cn.com",
    "*.googletraveladservices-cn.com",
    "googletagservices-cn.com",
    "*.googletagservices-cn.com",
    "googletagmanager-cn.com",
    "*.googletagmanager-cn.com",
    "googlesyndication-cn.com",
    "*.googlesyndication-cn.com",
    "*.safeframe.googlesyndication-cn.com",
    "app-measurement-cn.com",
    "*.app-measurement-cn.com",
    "gvt1-cn.com",
    "*.gvt1-cn.com",
    "gvt2-cn.com",
    "*.gvt2-cn.com",
    "2mdn-cn.net",
    "*.2mdn-cn.net",
    "googleflights-cn.net",
    "*.googleflights-cn.net",
    "admob-cn.com",
    "*.admob-cn.com",
    "googlesandbox-cn.com",
    "*.googlesandbox-cn.com",
    "*.safenup.googlesandbox-cn.com",
    "*.gstatic.com",
    "*.metric.gstatic.com",
    "*.gvt1.com",
    "*.gcpcdn.gvt1.com",
    "*.gvt2.com",
    "*.gcp.gvt2.com",
    "*.url.google.com",
    "*.youtube-nocookie.com",
    "*.ytimg.com",
    "android.com",
    "*.android.com",
    "*.flash.android.com",
    "g.cn",
    "*.g.cn",
    "g.co",
    "*.g.co",
    "goo.gl",
    "www.goo.gl",
    "google-analytics.com",
    "*.google-analytics.com",
    "google.com",
    "googlecommerce.com",
    "*.googlecommerce.com",
    "ggpht.cn",
    "*.ggpht.cn",
    "urchin.com",
    "*.urchin.com",
    "youtu.be",
    "youtube.com",
    "*.youtube.com",
    "youtubeeducation.com",
    "*.youtubeeducation.com",
    "youtubekids.com",
    "*.youtubekids.com",
    "yt.be",
    "*.yt.be",
    "android.clients.google.com",
    "developer.android.google.cn",
    "developers.android.google.cn",
    "source.android.google.cn"
  ],
  "hostname": [
    "google.com",
    "google.es",
    "widevine.cn",
    "googlevads-cn.com",
    "fls.doubleclick.cn",
    "googletagmanager-cn.com",
    "googlesyndication-cn.com",
    "gcp.gvt2.com",
    "g.cn",
    "origin-test.bdn.dev",
    "fls.doubleclick-cn.net",
    "google-analytics.com",
    "bdn.dev",
    "googleadservices-cn.com",
    "g.doubleclick-cn.net",
    "doubleclick.cn",
    "2mdn-cn.net",
    "googlesandbox-cn.com",
    "gstatic.com",
    "googlecommerce.com",
    "developer.android.google.cn",
    "google.com.co",
    "google.fr",
    "google.nl",
    "googleadapis.com",
    "googleoptimize-cn.com",
    "googleflights-cn.net",
    "appengine.google.com",
    "crowdsource.google.com",
    "google.cl",
    "google.com.tr",
    "ampproject.org.cn",
    "google-analytics-cn.com",
    "googletagservices-cn.com",
    "google.co.in",
    "google.co.jp",
    "google.co.uk",
    "google.pl",
    "g.doubleclick.cn",
    "goo.gl",
    "youtubekids.com",
    "gkecnapps.cn",
    "safeframe.googlesyndication-cn.com",
    "safenup.googlesandbox-cn.com",
    "metric.gstatic.com",
    "android.com",
    "flash.android.com",
    "www.goo.gl",
    "datacompute.google.com",
    "google.de",
    "google.it",
    "googledownloads.cn",
    "gcpcdn.gvt1.com",
    "yt.be",
    "google.pt",
    "googleapis.cn",
    "ampproject.net.cn",
    "dartsearch-cn.net",
    "googletraveladservices-cn.com",
    "gvt1-cn.com",
    "ytimg.com",
    "g.co",
    "google.com.ar",
    "google.com.br",
    "gstatic-cn.com",
    "recaptcha.net.cn",
    "recaptcha-cn.net",
    "googleapis-cn.com",
    "gvt2.com",
    "url.google.com",
    "source.android.google.cn",
    "google.hu",
    "googlecnapps.cn",
    "gvt2-cn.com",
    "ggpht.cn",
    "gstatic.cn",
    "admob-cn.com",
    "gvt1.com",
    "cloud.google.com",
    "google.ca",
    "youtube-nocookie.com",
    "youtu.be",
    "android.clients.google.com",
    "googlevideo.com",
    "urchin.com",
    "developers.android.google.cn",
    "google.com.au",
    "google.com.mx",
    "google.com.vn",
    "youtube.com",
    "googleapps-cn.com",
    "doubleclick-cn.net",
    "app-measurement-cn.com",
    "youtubeeducation.com"
  ],
  "serial": "37:E9:82:7A:AE:D7:7B:A2:10:C2:A3:FB:D8:57:94:A4",
  "issuer_dn": "CN=GTS CA 1C3, O=Google Trust Services LLC, C=US",
  "issuer_cn": "GTS CA 1C3",
  "issuer_org": [
    "Google Trust Services LLC"
  ],
  "fingerprint_hash": {
    "md5": "9efe46135dafacc1d99786107afe7dba",
    "sha1": "5a485b27a7fb0bd663838e8e80db29b72c72a88e",
    "sha256": "440c58514c737c67daa272298168cdfc51b5796566f055fa55c44530bbdd0982"
  },
  "wildcard_certificate": true,
  "tls_connection": "ctls",
  "sni": "google.com"
}

Copy link
Member

@ehsandeep ehsandeep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$ echo hackerone.com | ./tlsx -silent -cn -san
hackerone.com:443 [hackerone.com]
hackerone.com:443 [www.hackerone.com]
hackerone.com:443 [api.hackerone.com]
$ echo hackerone.com | ./tlsx -silent -dns
hackerone.com:443
api.hackerone.com
hackerone.com
www.hackerone.com

There is an additional line, hackerone.com:443, being appended in the result when used with the -dns option.

Expected is:

$ echo hackerone.com | ./tlsx -silent -dns
api.hackerone.com
hackerone.com
www.hackerone.com

Copy link
Member

@tarunKoyalwar tarunKoyalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • no output when using -j flag (i.e json)
$  go run . -u google.com -dns -j -v
  

  _____ _    _____  __
 |_   _| |  / __\ \/ /
   | | | |__\__ \>  < 
   |_| |____|___/_/\_\	v1.1.4

		projectdiscovery.io

[INF] Current tlsx version v1.1.4 (latest)
[INF] Processing input google.com:443
[INF] Connections made using crypto/tls: 1, zcrypto/tls: 0, openssl: 0

Copy link
Member

@tarunKoyalwar tarunKoyalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm !

$ go run . -u google.com -dns -j | jq .
  

  _____ _    _____  __
 |_   _| |  / __\ \/ /
   | | | |__\__ \>  < 
   |_| |____|___/_/\_\	v1.1.4

		projectdiscovery.io

[INF] Current tlsx version v1.1.4 (latest)
[INF] Connections made using crypto/tls: 1, zcrypto/tls: 0, openssl: 0
{
  "timestamp": "2023-09-18T19:23:26.348362+05:30",
  "host": "google.com",
  "ip": "142.250.199.174",
  "port": "443",
  "probe_status": true,
  "tls_version": "tls13",
  "cipher": "TLS_AES_128_GCM_SHA256",
  "not_before": "2023-08-14T08:16:28Z",
  "not_after": "2023-11-06T08:16:27Z",
  "subject_dn": "CN=*.google.com",
  "subject_cn": "*.google.com",
  "subject_an": [
    "*.google.com",
    "*.appengine.google.com",
    "*.bdn.dev",
    "*.origin-test.bdn.dev",
    "*.cloud.google.com",
    "*.crowdsource.google.com",
    "*.datacompute.google.com",
    "*.google.ca",
    "*.google.cl",
    "*.google.co.in",
    "*.google.co.jp",
    "*.google.co.uk",
    "*.google.com.ar",
    "*.google.com.au",
    "*.google.com.br",
    "*.google.com.co",
    "*.google.com.mx",
    "*.google.com.tr",
    "*.google.com.vn",
    "*.google.de",
    "*.google.es",
    "*.google.fr",
    "*.google.hu",
    "*.google.it",
    "*.google.nl",
    "*.google.pl",
    "*.google.pt",
    "*.googleadapis.com",
    "*.googleapis.cn",
    "*.googlevideo.com",
    "*.gstatic.cn",
    "*.gstatic-cn.com",
    "googlecnapps.cn",
    "*.googlecnapps.cn",
    "googleapps-cn.com",
    "*.googleapps-cn.com",
    "gkecnapps.cn",
    "*.gkecnapps.cn",
    "googledownloads.cn",
    "*.googledownloads.cn",
    "recaptcha.net.cn",
    "*.recaptcha.net.cn",
    "recaptcha-cn.net",
    "*.recaptcha-cn.net",
    "widevine.cn",
    "*.widevine.cn",
    "ampproject.org.cn",
    "*.ampproject.org.cn",
    "ampproject.net.cn",
    "*.ampproject.net.cn",
    "google-analytics-cn.com",
    "*.google-analytics-cn.com",
    "googleadservices-cn.com",
    "*.googleadservices-cn.com",
    "googlevads-cn.com",
    "*.googlevads-cn.com",
    "googleapis-cn.com",
    "*.googleapis-cn.com",
    "googleoptimize-cn.com",
    "*.googleoptimize-cn.com",
    "doubleclick-cn.net",
    "*.doubleclick-cn.net",
    "*.fls.doubleclick-cn.net",
    "*.g.doubleclick-cn.net",
    "doubleclick.cn",
    "*.doubleclick.cn",
    "*.fls.doubleclick.cn",
    "*.g.doubleclick.cn",
    "dartsearch-cn.net",
    "*.dartsearch-cn.net",
    "googletraveladservices-cn.com",
    "*.googletraveladservices-cn.com",
    "googletagservices-cn.com",
    "*.googletagservices-cn.com",
    "googletagmanager-cn.com",
    "*.googletagmanager-cn.com",
    "googlesyndication-cn.com",
    "*.googlesyndication-cn.com",
    "*.safeframe.googlesyndication-cn.com",
    "app-measurement-cn.com",
    "*.app-measurement-cn.com",
    "gvt1-cn.com",
    "*.gvt1-cn.com",
    "gvt2-cn.com",
    "*.gvt2-cn.com",
    "2mdn-cn.net",
    "*.2mdn-cn.net",
    "googleflights-cn.net",
    "*.googleflights-cn.net",
    "admob-cn.com",
    "*.admob-cn.com",
    "googlesandbox-cn.com",
    "*.googlesandbox-cn.com",
    "*.safenup.googlesandbox-cn.com",
    "*.gstatic.com",
    "*.metric.gstatic.com",
    "*.gvt1.com",
    "*.gcpcdn.gvt1.com",
    "*.gvt2.com",
    "*.gcp.gvt2.com",
    "*.url.google.com",
    "*.youtube-nocookie.com",
    "*.ytimg.com",
    "android.com",
    "*.android.com",
    "*.flash.android.com",
    "g.cn",
    "*.g.cn",
    "g.co",
    "*.g.co",
    "goo.gl",
    "www.goo.gl",
    "google-analytics.com",
    "*.google-analytics.com",
    "google.com",
    "googlecommerce.com",
    "*.googlecommerce.com",
    "ggpht.cn",
    "*.ggpht.cn",
    "urchin.com",
    "*.urchin.com",
    "youtu.be",
    "youtube.com",
    "*.youtube.com",
    "youtubeeducation.com",
    "*.youtubeeducation.com",
    "youtubekids.com",
    "*.youtubekids.com",
    "yt.be",
    "*.yt.be",
    "android.clients.google.com",
    "developer.android.google.cn",
    "developers.android.google.cn",
    "source.android.google.cn"
  ],
  "hostname": [
    "google.com.au",
    "google.nl",
    "widevine.cn",
    "google-analytics-cn.com",
    "doubleclick-cn.net",
    "doubleclick.cn",
    "googletraveladservices-cn.com",
    "datacompute.google.com",
    "gstatic.com",
    "gcp.gvt2.com",
    "url.google.com",
    "googletagservices-cn.com",
    "google.hu",
    "recaptcha.net.cn",
    "googlevads-cn.com",
    "app-measurement-cn.com",
    "admob-cn.com",
    "gcpcdn.gvt1.com",
    "goo.gl",
    "google.cl",
    "google.com.mx",
    "gstatic.cn",
    "googledownloads.cn",
    "fls.doubleclick-cn.net",
    "dartsearch-cn.net",
    "gvt1-cn.com",
    "youtube.com",
    "cloud.google.com",
    "google.pt",
    "gkecnapps.cn",
    "ampproject.net.cn",
    "googleoptimize-cn.com",
    "google.es",
    "googleflights-cn.net",
    "googlecnapps.cn",
    "ampproject.org.cn",
    "safenup.googlesandbox-cn.com",
    "gvt1.com",
    "android.clients.google.com",
    "google.com.br",
    "googleapis.cn",
    "googlesyndication-cn.com",
    "safeframe.googlesyndication-cn.com",
    "google-analytics.com",
    "google.com.vn",
    "google.co.jp",
    "google.co.uk",
    "googleadapis.com",
    "googleapis-cn.com",
    "appengine.google.com",
    "google.fr",
    "gstatic-cn.com",
    "g.doubleclick.cn",
    "gvt2.com",
    "developer.android.google.cn",
    "google.com.tr",
    "gvt2-cn.com",
    "googlesandbox-cn.com",
    "ytimg.com",
    "android.com",
    "g.co",
    "youtubeeducation.com",
    "youtubekids.com",
    "google.com",
    "crowdsource.google.com",
    "google.com.co",
    "recaptcha-cn.net",
    "googleadservices-cn.com",
    "flash.android.com",
    "source.android.google.cn",
    "origin-test.bdn.dev",
    "yt.be",
    "developers.android.google.cn",
    "googleapps-cn.com",
    "google.ca",
    "google.co.in",
    "google.de",
    "googlevideo.com",
    "ggpht.cn",
    "youtu.be",
    "bdn.dev",
    "g.doubleclick-cn.net",
    "2mdn-cn.net",
    "youtube-nocookie.com",
    "google.it",
    "fls.doubleclick.cn",
    "googletagmanager-cn.com",
    "metric.gstatic.com",
    "g.cn",
    "www.goo.gl",
    "googlecommerce.com",
    "google.pl",
    "urchin.com",
    "google.com.ar"
  ],
  "serial": "37:E9:82:7A:AE:D7:7B:A2:10:C2:A3:FB:D8:57:94:A4",
  "issuer_dn": "CN=GTS CA 1C3, O=Google Trust Services LLC, C=US",
  "issuer_cn": "GTS CA 1C3",
  "issuer_org": [
    "Google Trust Services LLC"
  ],
  "fingerprint_hash": {
    "md5": "9efe46135dafacc1d99786107afe7dba",
    "sha1": "5a485b27a7fb0bd663838e8e80db29b72c72a88e",
    "sha256": "440c58514c737c67daa272298168cdfc51b5796566f055fa55c44530bbdd0982"
  },
  "wildcard_certificate": true,
  "tls_connection": "ctls",
  "sni": "google.com"
}

@tarunKoyalwar
Copy link
Member

@dogancanbakir , we initially implemented dedupe for projectdiscovery/alterx at https://github.com/projectdiscovery/alterx/blob/main/dedupe.go

in that dedupe implementation , initially all dedupe happens using map datatype upto 100MB and after this threshold is reached it switches to goleveldb ( we should also use bloom filters if not already using ) thus it's faster and also has cleanup to remove residual data

we can move that implementation to utils and use it here as well as in alterx . in followup tickets . what do you think ?

@dogancanbakir
Copy link
Member Author

@tarunKoyalwar,
It makes sense; let's move it to the utils. Can you create a ticket for it? Thanks!

@ehsandeep ehsandeep merged commit 801c8a5 into dev Sep 19, 2023
8 checks passed
@ehsandeep ehsandeep deleted the add_dns_flag branch September 19, 2023 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Option to include / list unique hostname from certificate response
3 participants