[Filebeat][meta] Add Suricata module

[andrewkroh]

This is the meta issue to track the task of adding a new Filebeat module that reads the Suricata EVE JSON output.

• Add ECS fields to fields.yml. (@adriansr) Append ECS fields to fields.yml #8313
  • A convenient way to import this would be nice (like simply copying a fields-ecs.yml file to _meta/). Add Beats compatible fields.yml for ECS ecs#108
• For Filebeat 7.0 rename source to ? in accordance with ECS. (@adriansr) Append ECS fields to fields.yml #8313
  • According to the ECS logging use case this field should be source.path. (Is this correct?)
  • Finally decided on log.source to better fit the tcp/syslog inputs.
• Package X-Pack modules with the Elastic licensed Filebeat version. [Filebeat] Package x-pack modules in the Elastic-licensed distribution #8403 Filebeat X-Pack Module Packaging #8615
• Add and test the Suricata module.
  • Moving the Suricata module from its temporary repo to Filebeats #8089
  • Add a way to verify from Go that a JSON document's fields are contained in fields.yml. We will be validating the document after it has gone through the Ingest Node pipeline and is queried from ES. -> Field mapping validator #8476
  • Document that the module is only included in Elastic licensed packages.
• Write documentation for the module that explains how to set it up and use it.
  • Define tested Suricata versions.
• Backport module to 6.x and work-around the conflict of the ECS source.ip with the Filebeat 6.x source field.

[andrewkroh]

@ruflin Can you help me in determining what Filebeat's source should be renamed to.

@andrewvc Could mapval be used to validate that a document's fields are contained in fields.yml as described above? Apart of the python system tests, is anyone aware of any fields.yml validation code in Go?

[ruflin]

source should become file.path I think. I need to update the use case.

@andrewkroh For the field validation I remember @simitt put something together for apm-server quite some time ago.

[simitt]

We have a test PayloadAttrsMatchFields checking that attributes from the payload are indexed except for specific exceptions and a KeywordLimitation test checking that every string indexed as keyword follows the 1024 default length restriction in the json spec. Those tests are applied e.g. for transactions. There you also find more tests regarding allowed/forbidden/required/optional attributes.

[ruflin]

@simitt Didn't you also write something in the past to check if a field is in fields.yml in Go as we still have this in python. I now somehow even remember I added this on our side / copied it over. Have to search for it.

[ruflin]

Update: Looks like I missed the first link from @simitt which contains the code on the apm-server side to check the fields: https://github.com/elastic/apm-server/blob/e0e849c3e94042db07573f0c1e6684439f29b88e/tests/fields.go#L46

[adriansr]

Hi @ruflin, any update on the new source mapping for Filebeat?

You suggest file.source, but what about when input is not from a file? tcp, syslog etc? Currently source is the IP address for example.

[tsg]

Adding the module checklist:

• Supported versions are documented
• Supported operating systems are documented (if applicable)
• Integration tests
• System tests
• Automated checks that all fields are documented
• Documentation
• Example data.json exists and an automated way to generate it exists (go test -data)
• Fields follow naming conventions: https://www.elastic.co/guide/en/beats/devguide/master/event-conventions.html
• Dashboards exists (if applicable)
• Kibana Home Tutorial (if applicable) Add tutorial for the Suricata module kibana#24373

[elasticmachine]

Pinging @elastic/secops