Address #1622, add securitytxt module

[colin-stubbs]

Addresses issue #1622

Adds new securitytxt module.

Full test suite runs without error.

[codecov]

Codecov Report

Attention: Patch coverage is 92.85714% with 5 lines in your changes missing coverage. Please review.

Project coverage is 93%. Comparing base (16fc5fa) to head (beb48a8).

Files	Patch %	Lines
bbot/modules/securitytxt.py	91%	5 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #1628   +/-   ##
=====================================
- Coverage     93%     93%   -0%     
=====================================
  Files        341     343    +2     
  Lines      26040   26108   +68     
=====================================
+ Hits       24012   24039   +27     
- Misses      2028    2069   +41     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TheTechromancer]

@colin-stubbs this is some interesting stuff! I'm going to apologize in advance for not getting to it right away, being busy with the 2.0 release. But I should have time next week to give it a full review.

[TheTechromancer]

@colin-stubbs solid module. Made one small tweak, should be almost ready to merge.

One question about the in_scope_only option. Is the idea to restrict out-of-scope emails/urls from being emitted by the module? Or is it for filtering the incoming DNS_NAMEs?

[colin-stubbs]

@colin-stubbs solid module. Made one small tweak, should be almost ready to merge.

One question about the in_scope_only option. Is the idea to restrict out-of-scope emails/urls from being emitted by the module? Or is it for filtering the incoming DNS_NAMEs?

Definitely to filter incoming DNS_NAME's

Basically I noted that in_scope_only is default False in the default BaseModule class, and I think the default behaviour for this module and the others I'm working on should be to have safe defaults to avoid having them recurse too far out beyond the original target(s) unless instructed to do otherwise.

My understanding of BaseModule and in_scope_only == True, and my testing, suggested this is the behaviour I'd want most of the time.

e.g. if target was blacklanternsecurity.notreal we definitely probe that, plus sub.blacklanternsecurity.notreal, but we definitely don't want to probe thirdparty.notreal even if a DNS_NAME for that gets emitted, unless thirdparty.notreal is somehow determined to be in scope due to scan configuration.

[TheTechromancer]

avoid having them recurse too far out beyond the original target(s) unless instructed to do otherwise

Got it. I'm realizing this is a blind spot in our documentation so I'll try and explain how it works here, and you can tell me if it makes sense lol.

By default a module will only accept events with scope distance <= config.scope.search_distance (default 0) + self.scope_distance_modifier (default 0). This means that, by default, a module only accepts in-scope events unless the user cranks up the scan's search_distance.

Usually the only time you want to set in_scope_only is when you don't want the module to automatically scale with the scan's search_distance (i.e., you don't want the user to be able to control it). Most of the subdomain-enum modules use this setting, to avoid enumerating distance-1 domains like microsoft.com.

For this module, it might make sense to leave those at their defaults. This would keep it in-scope-only by default, but let the user dial up the search distance if they wanted to.

[TheTechromancer]

I made a small PR to your PR 😂

If it looks okay, feel free to merge it and it will update this one.