Elastic-friendly Event JSON

[TheTechromancer]

This PR separates the event IDs from the discovery_chain and puts them in their own list:

{
  "type": "DNS_NAME",
  "id": "DNS_NAME:879e47564ff0ed7711b707d3dbecb706ad6af1a3",
  "scope_description": "in-scope",
  "data": "www.blacklanternsecurity.com",
  "host": "www.blacklanternsecurity.com",
  "resolved_hosts": [
    "185.199.108.153",
    "2606:50c0:8003::153",
    "blacklanternsecurity.github.io"
  ],
  "dns_children": {},
  "web_spider_distance": 0,
  "scope_distance": 0,
  "scan": "SCAN:477d1e6b94be928bf85c554b0845985189cfc81d",
  "timestamp": "2024-08-17T03:49:47.906017+00:00",
  "parent": "DNS_NAME:1e57014aa7b0715bca68e4f597204fc4e1e851fc",
  "tags": [
    "cdn-github",
    "subdomain",
    "in-scope"
  ],
  "module": "otx",
  "module_sequence": "otx",
  "discovery_context": "otx searched otx API for \"blacklanternsecurity.com\" and found DNS_NAME: www.blacklanternsecurity.com",
  "discovery_path": [ <-----
    "Scan demonic_jimmy seeded with DNS_NAME: blacklanternsecurity.com",
    "otx searched otx API for \"blacklanternsecurity.com\" and found DNS_NAME: www.blacklanternsecurity.com"
  ],
  "parent_chain": [ <-----
    "DNS_NAME:1e57014aa7b0715bca68e4f597204fc4e1e851fc",
    "DNS_NAME:879e47564ff0ed7711b707d3dbecb706ad6af1a3"
  ]
}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93%. Comparing base (ac08e5f) to head (3d74ac5).
Report is 127 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #1670   +/-   ##
=====================================
- Coverage     93%     93%   -0%     
=====================================
  Files        341     341           
  Lines      25934   25948   +14     
=====================================
- Hits       23931   23923    -8     
- Misses      2003    2025   +22     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[colin-stubbs]

Here's another issue @TheTechromancer ,

WIth BBOT v1.x siem_friendly=True output... .data.SCAN is a string, e.g.

{
  "type": "SCAN",
  "id": "SCAN:725368977d3a680e579707504e59428a7e3acc9d",
  "data": {
    "SCAN": "heinous_hermione (SCAN:725368977d3a680e579707504e59428a7e3acc9d)"
  },
  "scope_distance": 0,
  "scan": "SCAN:725368977d3a680e579707504e59428a7e3acc9d",
  "timestamp": 1709170919.403808,
  "source": "SCAN:725368977d3a680e579707504e59428a7e3acc9d",
  "tags": [
    "in-scope"
  ],
  "module": "TARGET",
  "module_sequence": "TARGET"
}

Whereas with BBOT v2.x siem_friendly=True output .data.SCAN is an object, e.g.

{
  "type": "SCAN",
  "id": "SCAN:b7b249df0e216908b4377509f50ac8092326b36b",
  "scope_description": "in-scope",
  "data": {
    "SCAN": {
      "id": "SCAN:b7b249df0e216908b4377509f50ac8092326b36b",
      "name": "devious_edna",
      "target": {
        "seeds": [
          "blacklanternsecurity.com"
        ],
        "whitelist": [
          "blacklanternsecurity.com"
        ],
        "blacklist": [],
        "strict_scope": false,
        "hash": "cffefd70a4eac5b8389a3c16987fb2ae91328c4c",
        "seed_hash": "29b7be19a3f7633571a48c40f320d465c918c26b",
        "whitelist_hash": "29b7be19a3f7633571a48c40f320d465c918c26b",
        "blacklist_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "scope_hash": "ef4a64d445f60a4ae47d81411d8994e40c3382d1"
      },

Again, I can hack around this in the Elastic ingest pipeline by moving .data.SCAN to something else like .data.SCAN_INIT

But this will probably cause conflicts with other SIEM systems that also expect type consistency.

It won't be a problem if there's no coexistence of v1.x logs alongside v2.x logs, so one other approach here is also to simply say that they're not compatible and that any new version of a SIEM integration should only support v2.x logs...

[TheTechromancer]

Oof yeah. My instinct would be to support only v2 going forward.

[colin-stubbs]

Yeap, I'm inclined to agree right now, backwards compatibility right now might be pointless, though I do think my suggestion in #1672 re using JSON Schema to test output would be really good moving forward to avoid this kind of issue again as BBOT v3.x develops :-)

[TheTechromancer]

@colin-stubbs should we merge this?

[colin-stubbs]

@colin-stubbs should we merge this?

Yeap, this would be good.