BOT: DNS lookup

[robcza]

We are considering a bot to do a DNS lookup for events, where only fqdn is included. Except the performance issues that will be the same as in the case of reverse_dns expert bot, we have encountered another issue. Forward lookup could return more than one IP address for the domain, what to do with the result?

• split the event in more events and fill the source.ip with the results of the dns lookup
• add new attribute for the results of the dns lookup

What do you suggest?

[aaronkaplan]

On 22 Oct 2015, at 15:03, Robert Šefr notifications@github.com wrote:

We are considering a bot to do a DNS lookup for events, where only fqdn is included. Except the performance issues that will be the same as in the case of reverse_dns expert bot, we have encountered another issue. Forward lookup could return more than one IP address for the domain, what to do with the result?

hmm, good question.

• split the event in more events and fill the source.ip with the results of the dns lookup

I guess that depends on the event.

• add new attribute for the results of the dns lookup
an array?

What do you suggest?

Can you give us an example of such an event?

—
Reply to this email directly or view it on GitHub.

[robcza]

Array is an obvious solution, and although I'm not a fan of adding new attributes, I don't think we should split an event into more. The logic behind the event is "this fqdn is bad" and splitting the event would be misleading on the output. My opinion is - we should add new array attribute.

I do not have any particular event example, but any domain with more IPs will do, sorry google:

Non-authoritative answer:
Name:   google.com
Address: 173.194.123.72
Name:   google.com
Address: 173.194.123.68
Name:   google.com
Address: 173.194.123.73
Name:   google.com
Address: 173.194.123.67
Name:   google.com
Address: 173.194.123.69
Name:   google.com
Address: 173.194.123.65
Name:   google.com
Address: 173.194.123.70
Name:   google.com
Address: 173.194.123.66
Name:   google.com
Address: 173.194.123.71
Name:   google.com
Address: 173.194.123.78
Name:   google.com
Address: 173.194.123.64

[e3rd]

What about 'forward_dns' to be the name? Note that 'reverse_dns' exists just for the opposite case.

[aaronkaplan]

On 22 Oct 2015, at 15:39, Robert Šefr notifications@github.com wrote:

Array is an obvious solution, and although I'm not a fan of adding new attributes, I don't think we should split an event into more. The logic behind the event is "this fqdn is bad" and splitting the event would be misleading on the output. My opinion is - we should add new array attribute.

I do not have any particular event example, but any domain with more IPs will do, sorry google:

Got it. Okay, I came across the same problem once when thinking about pDNS.
So, if we have an array, then potentially we have a very large array.
Still I prefer an array to splitting the event.

Non-authoritative answer:
Name: google.com
Address: 173.194.123.72
Name: google.com
Address: 173.194.123.68
Name: google.com
Address: 173.194.123.73
Name: google.com
Address: 173.194.123.67
Name: google.com
Address: 173.194.123.69
Name: google.com
Address: 173.194.123.65
Name: google.com
Address: 173.194.123.70
Name: google.com
Address: 173.194.123.66
Name: google.com
Address: 173.194.123.71
Name: google.com
Address: 173.194.123.78
Name: google.com
Address: 173.194.123.64

—
Reply to this email directly or view it on GitHub.

[aaronkaplan]

On 22 Oct 2015, at 15:42, e3rd notifications@github.com wrote:

What about 'forward_dns' to be the name? Note that 'reverse_dns' exists just for the opposite case.

Makes sense for me as well.

[e3rd]

I've been trying to implement this, however, when configuring forward_dns in harmonization.conf, I am unable to set the type as "list". I'd like to put it as "list of IPAddress" but this seems to not be possible.
I have to set forward_dns as String and join the list of IPs to the signle String.

There is no "list" type option in harmonization, or am I blind?

[sebix]

@e3rd Yes this is correct. We had a discussion on a similar problem concerning email addresses (abuse contact), see #185. But this is a different case, as data is derived from IP addresses, but not from abuse contacts.

Alternative to lists: Clone the event and create duplicates for each address. I prefer this variant as I see severe issues when saving multiple IPs per event. Small example: Given is a domain used as C&C server or a websites which distributes malware. The FQDN resolves to multiple addresses in different AS' and countries. We do ASN and CC lookups, but how to save these informations? How to filter them? Having non-distinct data in such fundamental fields means events are not unique anymore. With multiple addresses per event, how to save data which depends on it? Lists of CCs, ASNs etc? Needs big adoptions everywhere.

For all others: The forward DNS bot proposal can be found here:

• harmonization: CZ-NIC@fc52c59
• expert: CZ-NIC@f42cdea
• test: CZ-NIC@f91228d

[robcza]

Ok, cloning the events makes sense. I agree with @sebix
@e3rd could you please adjust the bot accordingly?

[e3rd]

Ok, I'll remake it and pull it here.

[e3rd]

Interesting problem that I do not dare solve without your consent.
There may be M ip addresses related to source and N addresses related to the destination.
Should I make:
A) create lot of events: M × event (source.ip = ..., dest.ip = null) + N × event (source.ip = null, dest.ip = ...)
B) randomly map source to destination: M × event (source.ip = ..., dest.ip = ...) + (M-N) × event (source.ip = ..., dest.ip = null)
C) ...?

Did you get the point of the problem, am I clear?

[aaronkaplan]

On 25 Nov 2015, at 12:28, e3rd notifications@github.com wrote:

Interesting problem that I do not dare solve without your consent.
There may be M ip addresses related to source and N addresses related to the destination.
Should I make:
A) create lot of events: M × event (source.ip = ..., dest.ip = null) + N × event (source.ip = null, dest.ip = ...)
B) randomly map source to destination: M × event (source.ip = ..., dest.ip = ...) + (M-N) × event (source.ip = ..., dest.ip = null)
C) ...?

Did you get the point of the problem, am I clear?

I understand and I am not sure if it’s solvable.
I believe we are missing information here (e.g. net flow data which would prove which src ip talked to which dst ip).
Inventing an event is probably not a good solution.

It’s a bigger architecture discussion IMHO. Let’s have a call on this.

Best,
a.

[aaronkaplan]

will be looked at in Release-2 or -3

[e3rd]

I think this is partly covered by Gethostbyname bot and we may close this issue. In most of the cases, single IP address produced from a FQDN is enough.