Packetbeat includes nil IP fields when a DNS query is interrupted

[jsoriano]

There are situations with DNS packets, that Packetbeat includes null IP fields in the documents, causing mapping errors when ingesting in Elasticsearch.

This has been reported with events that contain the message Another query with the same DNS ID from this client was received so this query was closed without receiving a response.
In this case it seems that Packetbeat is including a resolved_ip field with a list with a single nil value ("resolved_ip": [ "<nil>" ]), and also a <nil> value in the related.ip field.

This causes errors like failed to parse field [dns.resolved_ip] of type [ip] in document with id 'EcpU5XQBHte-Y-A36w7t'. Preview of field's value: '<nil>'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'<nil>' is not an IP string literal.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mattheweberger]

Possible revisit this on 7.X roadmap?

[tom-x1]

Just stumbled upon the same issue

[efd6]

@jsoriano Are you able to obtain suitably redacted logs to dig into this further? I have narrowed down the critical region of code, but I need some additional context to proceed. Found.

[efd6]

In the case of an incomplete response, miekg/dns allows an empty IP to be packed into an answer; case 0 below:

func packDataA(a net.IP, msg []byte, off int) (int, error) {
	switch len(a) {
	case net.IPv4len, net.IPv6len:
		// It must be a slice of 4, even if it is 16, we encode only the first 4
		if off+net.IPv4len > len(msg) {
			return len(msg), &Error{err: "overflow packing a"}
		}

		copy(msg[off:], a.To4())
		off += net.IPv4len
	case 0:
		// Allowed, for dynamic updates.
	default:
		return len(msg), &Error{err: "overflow packing a"}
	}
	return off, nil
}

The converse is also true, meaning that we can receive an empty IP in an answer.

This is then rendered as <nil> by dns.rrToString during packetbeat publication.

Given that the record is a dropped repetition, I think it's reasonable to omit the IP from the record. It would be good if someone who has seen this on their installations could confirm that the successful query is actually indexed.

[tom-x1]

This problem does not only occur in case of an dropped repetition, I also see it in other cases but haven't found a explanation yet. But it seems to me that if a clients tries method:UPDATE and he's not authorized for that, that it leads also to this behaviour. For example:

{Timestamp:time.Time{wall:0x1a168720, ext:63769199361, loc:(*time.Location)(0x55f2cb91c640)}, Meta:null, Fields:{"agent":{"ephemeral_id":"8abff7b0-e855-48ff-afc1-03a35e115f10","hostname":"reg-node.example.com","id":"ed1816ac-fc02-43f2-8163-31522f7d6bf2","name":"reg-node.example.com","type":"packetbeat","version":"7.14.0"},"client":{"bytes":512,"ip":"96.xx.xx.230","port":13169},"destination":{"bytes":512,"ip":"176.xx.xx.3","port":53},"dns":{"additionals_count":0,"answers":[{"class":"NONE","data":"\u003cnil\u003e","name":"ORLITMOT-TAXLT6.lithiainc.com","ttl":"0","type":"A"}],"answers_count":1,"authorities_count":1,"flags":{"authentic_data":false,"authoritative":false,"checking_disabled":false,"recursion_available":false,"recursion_desired":false,"truncated_response":false},"header_flags":null,"id":5154,"op_code":"UPDATE","question":{"class":"IN","etld_plus_one":"lithiainc.com","name":"lithiainc.com","registered_domain":"lithiainc.com","top_level_domain":"com","type":"SOA"},"resolved_ip":["\u003cnil\u003e"],"response_code":"REFUSED","type":"answer"},"ecs":{"version":"1.10.0"},"event":{"category":["network_traffic","network"],"dataset":"dns","duration":54000,"end":"2021-10-07T10:29:21.437Z","kind":"event","start":"2021-10-07T10:29:21.437Z","type":["connection","protocol"]},"host":{"name":"reg-node.example.com"},"method":"UPDATE","network":{"bytes":1024,"community_id":"1:V2xZdDRmdG9zkcVzqQkU76W/ulE=","direction":"ingress","protocol":"dns","transport":"udp","type":"ipv4"},"query":"class IN, type SOA, lithiainc.com","related":{"ip":["96.xx.xx.230","176.xx.xx.3","\u003cnil\u003e"]},"resource":"lithiainc.com","server":{"bytes":512,"ip":"176.xx.xx.3","port":53},"source":{"bytes":512,"ip":"96.xx.xx.230","port":13169},"status":"Error","type":"dns"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [dns.resolved_ip] of type [ip] in document with id 'TphMWnwBxS9AVhzk-0no'. Preview of field's value: '<nil>'","caused_by":{"type":"illegal_argument_exception","reason":"'<nil>' is not an IP string literal."}}

I've seen such indexing errors also with the 'related.ip' field. To me it looks like it would be reasonable to remove any field which is mapped as IP adress if does not contain a valid IP address.

[tom-x1]

I don't find the "method" field in the documentation by the way...

[efd6]

Thanks for providing that. The places in that output look like pretty reasonable places to have a non-present IP. The change I have prepared will prevent those from being emitted.

I will look into the documentation.

[efd6]

@tom-x1 You can find the method field described here.