DNS over TLS support for DNS processor

[philippkahr]

Label: Enhancement

What does this PR do?

Implement DNS over TLS (DoT) handler for the DNS Processor. #16663

Why is it important?

Now you can securely query for reverseDNS lookups across the internet. E.g. cloudflare 1.1.1.1 supports DNS over TLS

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
  I would need a bit of help here. Do I have to touch every beat config, or is the DNS processor page in the docs rendered from the libbeat ascii doc file?
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

1. go to filebeat
2. make
3. edit filebeat config to include the at least the DNS processor

Minimum filebeat config

  - dns:
      type: reverse
      transport: tls
      fields:
        source.ip: source.hostname
        destination.ip: destination.hostname
      nameservers: ["1.1.1.1"]

Full filebeat config

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/e_pkah/Documents/GitHub/pkahr/demodata/*
  json.keys_under_root: true
  json.add_error_key: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
output.elasticsearch:
  hosts: ["localhost:9200"]
processors:
  - decode_json_fields:
      fields: log
      overwrite_keys: true
  - dns:
      type: reverse
      transport: tls
      fields:
        source.ip: source.hostname
        destination.ip: destination.hostname
      nameservers: ["1.1.1.1"]

Example doc

{"source.ip": "1.1.1.1","destination.ip": "8.8.8.8"}

4. replace transport: either with tls, udp. If left empty it will revert to default udp
5. Ingest some file that has a source.ip or destination.ip (e.g. use example doc above)

Related issues

• Closes [*beats] Reverse DNS processor support DNSoverTLS #16663
• Should we create an additional issue that tracks the status of the implementation of DNS over HTTPS in the https://github.com/miekg/dns repo?

Use cases

• Allow for secure reverse DNS lookups across the entire Internet and locally

Logs

TLS

./filebeat -e -d "*"
2020-06-22T19:56:00.082+0200    DEBUG   [processor.dns] dns/dns.go:67   DNS processor config: {CacheConfig:{SuccessCache:{TTL:0s InitialCapacity:1000 MaxCapacity:10000} FailureCache:{TTL:1m0s InitialCapacity:1000 MaxCapacity:10000}} Nameservers:[1.1.1.1] Timeout:500ms Type:reverse Action:append TagOnFailure:[] Fields:{"destination":{"ip":"destination.hostname"},"source":{"ip":"source.hostname"}} transport:tls reverseFlat:map[destination.ip:destination.hostname source.ip:source.hostname]}      {"instance_id": 1}

2020-06-22T20:28:31.155+0200    DEBUG   [processors]    processing/processors.go:187    Publish event: {
  "@timestamp": "2020-06-22T18:28:30.854Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.0.0"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/Users/e_pkah/Documents/GitHub/pkahr/demodata/apache.json"
    }
  },
  "ecs": {
    "version": "1.5.0"
  },
  "destination": {
    "hostname": "dns.google"
  },
  "agent": {
    "id": "6bc4fd89-5e9e-4382-a2fb-5068f5782406",
    "name": "N1890002",
    "type": "filebeat",
    "version": "8.0.0",
    "ephemeral_id": "b8d53727-48f5-4a6a-9c84-735b7d4d0905"
  },
  "source": {
    "hostname": "one.one.one.one"
  },
  "source.ip": "1.1.1.1",
  "destination.ip": "8.8.8.8",
  "input": {
    "type": "log"
  },
  "host": {
    "name": "N1890002"
  }
}

UDP / no config

./filebeat -e -d "*"
2020-06-22T20:13:33.150+0200    DEBUG   [processor.dns] dns/dns.go:67   DNS processor config: {CacheConfig:{SuccessCache:{TTL:0s InitialCapacity:1000 MaxCapacity:10000} FailureCache:{TTL:1m0s InitialCapacity:1000 MaxCapacity:10000}} Nameservers:[1.1.1.1] Timeout:500ms Type:reverse Action:append TagOnFailure:[] Fields:{"destination":{"ip":"destination.hostname"},"source":{"ip":"source.hostname"}} transport:udp reverseFlat:map[destination.ip:destination.hostname source.ip:source.hostname]}      {"instance_id": 1}

2020-06-22T20:28:31.155+0200    DEBUG   [processors]    processing/processors.go:187    Publish event: {
  "@timestamp": "2020-06-22T18:28:30.854Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.0.0"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/Users/e_pkah/Documents/GitHub/pkahr/demodata/apache.json"
    }
  },
  "ecs": {
    "version": "1.5.0"
  },
  "destination": {
    "hostname": "dns.google"
  },
  "agent": {
    "id": "6bc4fd89-5e9e-4382-a2fb-5068f5782406",
    "name": "N1890002",
    "type": "filebeat",
    "version": "8.0.0",
    "ephemeral_id": "b8d53727-48f5-4a6a-9c84-735b7d4d0905"
  },
  "source": {
    "hostname": "one.one.one.one"
  },
  "source.ip": "1.1.1.1",
  "destination.ip": "8.8.8.8",
  "input": {
    "type": "log"
  },
  "host": {
    "name": "N1890002"
  }
}

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [marc-gr commented: jenkins run tests]

• Start Time: 2020-07-21T07:59:28.060+0000

• Duration: 89 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	17705
Skipped	1902
Total	19607

Steps errors

Expand to view the steps failures

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 2 min 22 sec

  • Start Time: 2020-07-21T08:21:18.488+0000

  • log

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 2 min 22 sec

  • Start Time: 2020-07-21T08:35:30.595+0000

  • log

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 1 min 27 sec

  • Start Time: 2020-07-21T08:26:38.332+0000

  • log

• Name: Install Go 1.14.4

  • Description: .ci/scripts/install-go.sh

  • Duration: 1 min 57 sec

  • Start Time: 2020-07-21T08:25:06.462+0000

  • log

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 1 min 26 sec

  • Start Time: 2020-07-21T08:35:40.701+0000

  • log

[philippkahr]

Hi,

I am not too happy about all the added duplicated code for the TLS test, however, I could not figure out a way where we can reuse most of the code. Maybe someone can point me in the right direction 17b1c07

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[philippkahr]

Hi @andrewkroh I hope it is ok, that I ping you on this PR. I saw that you originally developed the DNS processor. Could you look into this PR?

[marc-gr]

Hello @philippkahr and thanks for opening this PR!

Couple of things:

Please add the CHANGELOG.next.asciidoc entry to the end of the entry list, and remove it from CHANGELOG-developer.next.asciidoc
I think this will need a rebase and to run mage fmt update to generate the new documentation.

If you wish, we can help you through it or do it ourselves, as you prefer.

[philippkahr]

Hi @marc-gr I am currently on vacation and do not have access to my MacBook. If you could be so kind and perform those changes for me, I would be happy.

[marc-gr]

Hi @marc-gr I am currently on vacation and do not have access to my MacBook. If you could be so kind and perform those changes for me, I would be happy.

Sure no problem! enjoy the rest of your vacations!

[marc-gr]

jenkins run tests

[andrewkroh]

I just a have a few minor requests. Otherwise LGTM. Thanks

[marc-gr]

jenkins run tests

[marc-gr]

jenkins run tests