v7.0.0

Breaking

• removed metric file target
• move kafka config options to kafka_config dictionary for confluent_kafka_input and confluent_kafka_output connectors

Features

• add a preprocessor to enrich by systems env variables
• add option to define rules inline in pipeline config under processor configs generic_rules or specific_rules
• add option to field_manager to ignore missing source fields to suppress warnings and failure tags
• add ignore_missing_source_fields behavior to calculator, concatenator, dissector, grokker, ip_informer, selective_extractor
• kafka input connector
  • implemented manual commit behaviour if enable.auto.commit: false
  • implemented on_commit callback to check for errors during commit
  • implemented statistics callback to collect metrics from underlying librdkafka library
  • implemented per partition offset metrics
  • get logs and handle errors from underlying librdkafka library
• kafka output connector
  • implemented statistics callback to collect metrics from underlying librdkafka library
  • get logs and handle errors from underlying librdkafka library

Improvements

• pre_detector processor now adds the field creation_timestamp to pre-detections.
  It contains the time at which a pre-detection was created by the processor.
• add prometheus and grafana to the quickstart setup to support development
• provide confluent kafka test setup to run tests against a real kafka cluster

Bugfix

• fix CVE-2023-37920 Removal of e-Tugra root certificate
• fix CVE-2023-43804 Cookie HTTP header isn't stripped on cross-origin redirects
• fix CVE-2023-37276 aiohttp.web.Application vulnerable to HTTP request smuggling via llhttp HTTP request parser

Details

• Add creation timestamp to pre-detections by @ppcad in #436
• add env variable enrichment preprocessor by @ekneg54 in #448
• Fix CVE 2023 37920 by @ekneg54 in #445
• fix main pipeline by @ekneg54 in #449
• fix memory leak by @ekneg54 in #452
• add grafana and prometheus stack to quickstart by @ekneg54 in #450
• fix CVE-2023-43804 by @ekneg54 in #454
• add inline rule config feature to pipline config by @ekneg54 in #453
• Remove MetricFileTarget by @ekneg54 in #456
• remove process_strategy and mermaid by @ekneg54 in #451
• remove MultiprocessLogHandler by @ekneg54 in #455
• rewrite kafka connector by @ekneg54 in #444
• add option to ignore missing fields in field_manager by @ekneg54 in #457
• prepare release 7.0.0 by @dtrai2 in #461

Full Changelog: v6.8.1...v7.0.0