MISP feeds cause the growing of ES audit docs

[garanews]

Request Type

Issue

Problem Description

The Hive when connected with a MISP instance with public feeds enabled (ex. alienvault) causes an immense ES audit growth.

Steps to Reproduce

Enable Feeds on MISP
Configure The Hive to sync with MISP
After a while check audit docs in Elasticsearch

Complementary information

Configure feed in MISP:

command to discover the biggest doc in ES audit:

curl -XPOST http://127.0.0.1:9200/the_hive_12/audit/_search -d ' { "size": 1, "_source": "_id", "query": { "match_all": {} }, "sort": { "_script": { "script": { "lang": "groovy", "source": "_source.toString().length()" }, "type": "number", "order": "desc" } } }'

when found (example AWErWiwwJTWaHXxRk2AF) download it
curl 'http://127.0.0.1:9200/the_hive_12/audit/AWErWiwwJTWaHXxRk2AF' -o /tmp/big_audit.json

check size
du -h big_audit.json
68M big_audit.json

If try to open will see a very big single row.
This issue will cause the fault during migration (happened from the_hive_11 to the_hive_12) and growing of ES with not useful data:

curl 'http://localhost:9200/_cat/indices/*?v&s=index'

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open the_hive_12 a1Hn_YUATjSi9m1qKhdVXQ 5 1 105692269 3628968 10.8gb 10.8gb

[garanews]

4GB in 4 days...

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open the_hive_12 a1Hn_YUATjSi9m1qKhdVXQ 5 1 151825871 2759162 14.3gb 14.3gb

[nadouani]

This means that you've fetched 2GB of alerts from the MISP? +2GB of audit logs?

Well, we could get rid of the 2GB of audit logs, but not of the 2GB of alerts, and the problem will still the same

[garanews]

Not many alerts arrived (like 50).
Issue is in audit log, not in alert volume.
The enabled feed contains ~50k domain/ip that are updated regularly.
This feed is set as fixed event and its attributes are changing (ip/domains are added/removed )
So no new alert in the hive is coming because this feed. And this alert is set as "ignored", so no idea why is causing the growth of audit log...

[nadouani]

That's clear, so the issue is that the audit contains the alert data. The fix will be to remove the alert data from the audit logs.

We will come up with a patch in the upcoming days and we will include such a fix.

For now, you know the workaround: clear the alert related audit log :(