Cherry-pick #19321 to 7.x: DNS over TLS support for DNS processor

[marc-gr]

Cherry-pick of PR #19321 to 7.x branch. Original message:

Label: Enhancement

What does this PR do?

Implement DNS over TLS (DoT) handler for the DNS Processor. #16663

Why is it important?

Now you can securely query for reverseDNS lookups across the internet. E.g. cloudflare 1.1.1.1 supports DNS over TLS

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
  I would need a bit of help here. Do I have to touch every beat config, or is the DNS processor page in the docs rendered from the libbeat ascii doc file?
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

1. go to filebeat
2. make
3. edit filebeat config to include the at least the DNS processor

Minimum filebeat config

  - dns:
      type: reverse
      transport: tls
      fields:
        source.ip: source.hostname
        destination.ip: destination.hostname
      nameservers: ["1.1.1.1"]

Full filebeat config

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/e_pkah/Documents/GitHub/pkahr/demodata/*
  json.keys_under_root: true
  json.add_error_key: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
output.elasticsearch:
  hosts: ["localhost:9200"]
processors:
  - decode_json_fields:
      fields: log
      overwrite_keys: true
  - dns:
      type: reverse
      transport: tls
      fields:
        source.ip: source.hostname
        destination.ip: destination.hostname
      nameservers: ["1.1.1.1"]

Example doc

{"source.ip": "1.1.1.1","destination.ip": "8.8.8.8"}

4. replace transport: either with tls, udp. If left empty it will revert to default udp
5. Ingest some file that has a source.ip or destination.ip (e.g. use example doc above)

Related issues

• Closes [*beats] Reverse DNS processor support DNSoverTLS #16663
• Should we create an additional issue that tracks the status of the implementation of DNS over HTTPS in the https://github.com/miekg/dns repo?

Use cases

• Allow for secure reverse DNS lookups across the entire Internet and locally

Logs

TLS

./filebeat -e -d "*"
2020-06-22T19:56:00.082+0200    DEBUG   [processor.dns] dns/dns.go:67   DNS processor config: {CacheConfig:{SuccessCache:{TTL:0s InitialCapacity:1000 MaxCapacity:10000} FailureCache:{TTL:1m0s InitialCapacity:1000 MaxCapacity:10000}} Nameservers:[1.1.1.1] Timeout:500ms Type:reverse Action:append TagOnFailure:[] Fields:{"destination":{"ip":"destination.hostname"},"source":{"ip":"source.hostname"}} transport:tls reverseFlat:map[destination.ip:destination.hostname source.ip:source.hostname]}      {"instance_id": 1}

2020-06-22T20:28:31.155+0200    DEBUG   [processors]    processing/processors.go:187    Publish event: {
  "@timestamp": "2020-06-22T18:28:30.854Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.0.0"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/Users/e_pkah/Documents/GitHub/pkahr/demodata/apache.json"
    }
  },
  "ecs": {
    "version": "1.5.0"
  },
  "destination": {
    "hostname": "dns.google"
  },
  "agent": {
    "id": "6bc4fd89-5e9e-4382-a2fb-5068f5782406",
    "name": "N1890002",
    "type": "filebeat",
    "version": "8.0.0",
    "ephemeral_id": "b8d53727-48f5-4a6a-9c84-735b7d4d0905"
  },
  "source": {
    "hostname": "one.one.one.one"
  },
  "source.ip": "1.1.1.1",
  "destination.ip": "8.8.8.8",
  "input": {
    "type": "log"
  },
  "host": {
    "name": "N1890002"
  }
}

UDP / no config

./filebeat -e -d "*"
2020-06-22T20:13:33.150+0200    DEBUG   [processor.dns] dns/dns.go:67   DNS processor config: {CacheConfig:{SuccessCache:{TTL:0s InitialCapacity:1000 MaxCapacity:10000} FailureCache:{TTL:1m0s InitialCapacity:1000 MaxCapacity:10000}} Nameservers:[1.1.1.1] Timeout:500ms Type:reverse Action:append TagOnFailure:[] Fields:{"destination":{"ip":"destination.hostname"},"source":{"ip":"source.hostname"}} transport:udp reverseFlat:map[destination.ip:destination.hostname source.ip:source.hostname]}      {"instance_id": 1}

2020-06-22T20:28:31.155+0200    DEBUG   [processors]    processing/processors.go:187    Publish event: {
  "@timestamp": "2020-06-22T18:28:30.854Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.0.0"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/Users/e_pkah/Documents/GitHub/pkahr/demodata/apache.json"
    }
  },
  "ecs": {
    "version": "1.5.0"
  },
  "destination": {
    "hostname": "dns.google"
  },
  "agent": {
    "id": "6bc4fd89-5e9e-4382-a2fb-5068f5782406",
    "name": "N1890002",
    "type": "filebeat",
    "version": "8.0.0",
    "ephemeral_id": "b8d53727-48f5-4a6a-9c84-735b7d4d0905"
  },
  "source": {
    "hostname": "one.one.one.one"
  },
  "source.ip": "1.1.1.1",
  "destination.ip": "8.8.8.8",
  "input": {
    "type": "log"
  },
  "host": {
    "name": "N1890002"
  }
}

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[andrewkroh]

Suggested change

	- Add support for DNS over TLS for the dns_processor. {pull}19321[19321]
	- Add support for DNS over TLS for the `dns` processor. {pull}19321[19321]

The processor name is just dns.

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #20090 updated]

• Start Time: 2020-08-11T12:05:45.282+0000

• Duration: 83 min 57 sec

Test stats 🧪

Test	Results
Failed	0
Passed	17782
Skipped	1838
Total	19620