Elastic Common Schema (ECS)

[kevinkeeneyjr]

I would suggest migrating to the Elastic Common Schema (ECS) for your output so this data can be integrated with output of other projects like RockNSM that have begun to adopt ECS. I believe ECS begins to solve the Babylon problem, by enabling a common language to talk about things like 'host.name' or 'os.version'

https://github.com/elastic/ecs/

[qmontal]

Hi @kevinkeeneyjr ,

This is a really great idea! I am not sure that we could apply it immediately, as it is still work in progress, but working with them we could make sure that all of our needed fields are included in the schema.

I will close #81 in pro of this issue, as its objective was the same but this already has a defined structure to work with.

Cheers!

[qmontal]

Hey Kevin,

I have given some thoughts to this, and I think the best would be to first come up with a standard vulnerability template for VulnWhisperer, using that template for saving the data from all the scanners locally following the same structure, and use ECS only for elasticsearch, changing the fields on the logstash config file from the VulnWhisperer structure to the ECS one.

This would mean also that ECS would be postponed until we have this VulnWhisperer vulnerability template that all scanners will follow.

What thoughts do you have on that?

Cheers!

[elvarb]

Since #86 is still a work in progress it might be smart to include this change in the 6.0 update. That way it could be a part of a migration process, new dashboards and more. The re index api could also be used to re index old VulnWhisperer data to the ECS format.

[qmontal]

Hi @elvarb,

We are actually implementing that separately: on one side we are doing the upgrade to ELK6; on the other side, as mentioned in #113, we are implementing a new Vulnerability standard in VulnWhisperer.

The ECS would actually be implemented in the second, as with the changes that the new standard imply, it will be needed to create a new logstash configuration file that works with the template (and thus, with all the scanners), as well as adapt the Kibana dashboards to ECS.

Update all the current logstash to the ECS would be extra work that would end trashed when adapting the Vulnerability standard.

Cheers!

[nicpenning]

FYI, Elastic has added the Vulnerability schema to ECS: https://github.com/elastic/ecs/blob/master/schemas/vulnerability.yml

[qmontal]

Hi @nicpenning,
Thank your for making us aware, another VulnWhisperer user also notified us about it through the Slack channel. We need to gather resources to develop the change from 1.8 to 2.0, as even with the standard part solved, there's a big refactoring change that needs to be done.
Cheers!