[maybe later] Propose "minimum" option for defanging

[geekscrapy]

Great library!

It'd be great to see an option for defanging that would defang in a lesser intrusive manner. Potentially with the option to prioritise towards the left or right side. This may assist in the readability of some of the output strings (but maybe slightly more risky, hence making it optional!).

For example:

www.example.com = www[.]example[.]com (current)
www.example[.]com (minimum flag used, prioritise right by default)
www[.]example.com (minimum flag used, prioritise left)

http://www.google.com/this/is/bad.html = hXXp://www[.]google[.]com/this/is/bad[.]html (current)
http://www.google.com/this/is/bad.html = hXXp://www.google[.]com/this/is/bad[.]html (minimum flag used, prioritise right by default)
http://www.google.com/this/is/bad.html = hXXp://www[.]google.com/this/is/bad[.]html (minimum flag used, prioritise left)

192.168.4.2 = 192[.]168[.]4.2 (current)
192.168.4[.]2 (minimum flag used, prioritise right by default)
192[.]168.4.2 (minimum flag used, prioritise left)

[fhightower]

Thanks @geekscrapy ! Great idea, I hadn't thought of that. I'll work on getting this implemented over the next month or so.

[geekscrapy]

Hey, hate to bug - any progress with this? I may have some downtime soon to look at it of you haven't already

[fhightower]

Hey @geekscrapy ... thanks for the follow up and I apologize it has been so long. If you have time to take a look, that would be great! Otherwise, it is high time I finally get around to it, so I can prioritize this higher in the near future.

[fhightower]

I'm going to work on this today. Implementation details:

• Add a "minimum_defanging" option to the defanging function (and cli)
• If "minimum_defanging" is set, there will be an option to prioritize right or prioritize left (prioritize right will be default)

[fhightower]

@geekscrapy : I did some investigation into how to implement this feature request and I do not think I'm going to do it for two reasons:

Reason 1

I don't think it is possible to implement this feature request without significant changes to how this package works.

The fundamental difficulty is that there is no way for this package to know which instances of a certain character to defang. For example, if the defang function is given www.example.com and we want to minimally defang (with priority on the right), we would expect to get www.example[.]com back from the function. The problem is that there is no way for the package to know that the first . should be left alone while the second . should be defanged. This issue becomes more poignant if one of the functions is given a large blob of text with multiple indicators in it. Does that make sense?

From what I can see, implementing this feature would require a significant over-haul to the way this package works (it seems to me we would have to update the package to first parse indicators and then defang them appropriately). This is certainly possible, but not something I'm interested in doing right now.

I'm open to suggestions on how I can implement this more easily, but I don't see any easy, non-invasive way to implement this.

Reason 2

In your original comment, you mentioned that using this feature would be risky (and I think you are correct); after thinking a bit more on this, I'm not comfortable introducing a feature with even a small amount of risk. When it comes to handling indicators of compromise, I'm comfortable compromising readability to maintain safety and security.

Conclusion

Does all of this make sense? I still think it is a great idea and I would love to implement it someday/somewhere, but I think this would require more extensive changes to this package than I'm willing to make right now. Again, I'm open to any ideas anyone has one this issue (I'll close it out in a few weeks - but I'm comfortable reopening it if a good implementation strategy arises).

[geekscrapy]

You are spot on with your reasoning, and I was having the same thoughts about overall parsing being more complex, and therefore risky - but remember, this should be an optional flag?

My thought is that the library would probably need to understand the IOC type before it can make a decision on what to prioritise. This is made simpler, of course if you are dealing with a list of IOCs rather than a block of text.

Question: do people use this to convert blocks of text with IOCs in it? I would assume that IOCs are defanged "at source", during initial analysis/lookups and are therefore just a list (but we obviously can't ignore other use cases!) My particular use case is taking IOCs from analysis and pushing them to chat/emails etc. and so wouldn't be caught out by this.

Overall though, what type of IOCs would need to be recognised to implement this? As far as I see, it would just be two:

• IP
• Domain

Other considerations:

• Email @ sign should come before a domain
• URLs would only need the domain part converted so is covered above

So as far as I see, there would only be those two types that need to be recognised.

Thoughts?

[fhightower]

To your question about people defanging blocks of text: Yes, there are use cases I'm aware of where people are defanging blocks of text and I wouldn't want to place any restrictions on the input format.

I agree that the system would have to be aware of IOC type and that is a great point that implementing this feature request really only requires us to handle IP addresses and domains.

Identifying IOC types is possible (I already do this here: https://github.com/fhightower/ioc-finder), but that is not a step I'm going to take right now. I'll keep this issue open, but will not be working on it in the near future.