v10.0.0

v10.0.0

Breaking

• reimplement the logprep CLI, see logprep --help for more information.
• remove feature to reload configuration by sending signal SIGUSR1
• remove feature to validate rules because it is already included in logprep test config

Features

• add a number_of_successful_writes metric to the s3 connector, which counts how many events were successfully written to s3
• make the s3 connector work with the new _write_backlog method introduced by the confluent_kafka commit bugfix in v9.0.0
• add option to Opensearch Output Connector to use parallel bulk implementation (default is True)
• add feature to logprep to load config from multiple sources (files or uris)
• add feature to logprep to print the resulting configruation with logprep print json|yaml <Path to config> in json or yaml
• add an event generator that can send records to Kafka using data from a file or from Kafka
• add an event generator that can send records to a HTTP endpoint using data from local dataset

Improvements

• a do nothing option do dummy output to ensure dummy does not fill memory
• make the s3 connector raise FatalOutputError instead of warnings
• make the s3 connector blocking by removing threading
• revert the change from v9.0.0 to always check the existence of a field for negated key-value based lucene filter expressions
• make store_custom in s3, opensearch and elasticsearch connector not call batch_finished_callback to prevent data loss that could be caused by partially processed events
• remove the schema_and_rule_checker module
• rewrite Logprep Configuration object see documentation for more details
• rewrite Runner
• delete MultiProcessingPipeline class to simplify multiprocesing
• add FDA to the quickstart setup
• bump versions for fastapi and aiohttp to address CVEs

Bugfix

• make the s3 connector actually use the max_retries parameter
• fixed a bug which leads to a FatalOutputError on handling CriticalInputError in pipeline

Details

• add architecture overview by @djkhl in #478
• make parallel_bulk optional by @ekneg54 in #504
• remove tox setup by @ekneg54 in #500
• Adapt s3 connector for kafka fix by @ppcad in #499
• add do nothing option to dummy output by @ekneg54 in #503
• Revise CLI by @dtrai2 in #513
• Fix handling of CriticalInputError exceptions by @clumsy9 in #514
• Revert not exists changes from #480, but keep refactorings in filter_expression by @ppcad in #515
• Set no offsets on store custom by @ppcad in #518
• config from multiple sources by @ekneg54 in #507
• Add load-tester by @ppcad in #487
• Add http connector to quickstart setup by @djkhl in #509
• Add FDA to quickstart setup by @dtrai2 in #477
• Prepare release 10.0.0 by @ekneg54 in #521

New Contributors

• @djkhl made their first contribution in #478 🎆

Full Changelog: v9.0.3...v10.0.0

v9.0.3

v9.0.3

Breaking

Features

• make thread_count, queue_size and chunk_size configurable for parallel_bulk in opensearch output connector

Improvements

Bugfix

• fix parallel_bulk implementation not delivering messages to opensearch

Details

• make threadcount configurable for opensearch output connector by @ekneg54 in #496
• add kafka exporter to quickstart by @dtrai2 in #481
• fix opensearch output connector by @ekneg54 in #498

Full Changelog: v9.0.2...v9.0.3

v9.0.2

Bugfix

• remove duplicate pseudonyms in extra outputs of pseudonymizer

What's Changed

• prevent duplicate pseudonyms in extra output by @dtrai2 in #494
• prepare release by @dtrai2 in #495

Full Changelog: v9.0.1...v9.0.2

v9.0.1

Details

• use opensearch parallel bulk by @ekneg54 in #492
• prepare release 9.0.1 by @ekneg54 in #493

Full Changelog: v9.0.0...v9.0.1

v9.0.0

v9.0.0

Breaking

• remove possibility to inject auth credentials via url string, because of the risk leaking credentials in logs
  • if you want to use basic auth, then you have to set the environment variables
    • :code:LOGPREP_CONFIG_AUTH_USERNAME=<your_username>
    • :code:LOGPREP_CONFIG_AUTH_PASSWORD=<your_password>
  • if you want to use oauth, then you have to set the environment variables
    • :code:LOGPREP_CONFIG_AUTH_TOKEN=<your_token>
    • :code:LOGPREP_CONFIG_AUTH_METHOD=oauth

Features

Improvements

• improve error message on empty rule filter
• reimplemented pseudonymizer processor
  • rewrote tests till 100% coverage
  • cleaned up code
  • reimplemented caching using pythons lru_cache
  • add cache metrics
  • removed max_caching_days config option
  • add max_cached_pseudonymized_urls config option which defaults to 1000
  • add lru caching for peudonymizatin of urls
• improve loading times for the rule tree by optimizing the rule segmentation and sorting
• add support for python 3.12 and remove support for python 3.9
• always check the existence of a field for negated key-value based lucene filter expressions

Bugfix

• fix the rule tree parsing some rules incorrectly, potentially resulting in more matches
• fix confluent_kafka commit issue after kafka did some rebalancing, fixes also negative offsets

Details

• Update publish-release-to-pypi.yml to use trusted publishing by @dtrai2 in #474
• Improve error message on empty rule filter by @dtrai2 in #471
• refactor pseudonymizer by @ekneg54 in #466
• remove url string credential injection by @ekneg54 in #465
• fix changelog by @ekneg54 in #479
• add support python 3.12 and drop support python 3.9 by @ekneg54 in #447
• Improve rule loading times in rule tree by @ppcad in #464
• always expect exists filter by @ppcad in #480
• Fix kafka commit after rebalancing issue by @dtrai2 in #486
• prepare release 9.0.0 by @ekneg54 in #488
• fix release pipeline by @ekneg54 in #489

Full Changelog: v8.0.0...v9.0.0

v8.0.0

v8.0.0

Breaking

• reimplemented metrics so the former metrics configuration won't work anymore
• metric content changed and existent grafana dashboards will break
• new rule id could possibly break configurations if the same rule is used in both rule trees
  • can be fixed by adding a unique id to each rule or delete the possibly redundant rule

Features

• add possibility to convert hex to int in calculator processor with new added function from_hex
• add metrics on rule level
• add grafana example dashboards under quickstart/exampledata/config/grafana/dashboards
• add new configuration field id for all rules to identify rules in metrics and logs
  • if no id is given, the id will be generated in a stable way
  • add verification of rule id uniqueness on processor level over both rule trees to ensure metrics are counted correctly on rule level

Improvements

• reimplemented prometheus metrics exporter to provide gauges, histograms and counter metrics
• removed shared counter, because it is redundant to the metrics
• get exception stack trace by setting environment variable DEBUG

Details

• convert hex to int in calculator by @ekneg54 in #463
• refactor pre_detector by @ekneg54 in #467
• Revise Metrics by @dtrai2 in #460
• prepare release v8.0.0 by @ekneg54 in #472

Full Changelog: v7.0.0...v8.0.0

v7.0.0

Breaking

• removed metric file target
• move kafka config options to kafka_config dictionary for confluent_kafka_input and confluent_kafka_output connectors

Features

• add a preprocessor to enrich by systems env variables
• add option to define rules inline in pipeline config under processor configs generic_rules or specific_rules
• add option to field_manager to ignore missing source fields to suppress warnings and failure tags
• add ignore_missing_source_fields behavior to calculator, concatenator, dissector, grokker, ip_informer, selective_extractor
• kafka input connector
  • implemented manual commit behaviour if enable.auto.commit: false
  • implemented on_commit callback to check for errors during commit
  • implemented statistics callback to collect metrics from underlying librdkafka library
  • implemented per partition offset metrics
  • get logs and handle errors from underlying librdkafka library
• kafka output connector
  • implemented statistics callback to collect metrics from underlying librdkafka library
  • get logs and handle errors from underlying librdkafka library

Improvements

• pre_detector processor now adds the field creation_timestamp to pre-detections.
  It contains the time at which a pre-detection was created by the processor.
• add prometheus and grafana to the quickstart setup to support development
• provide confluent kafka test setup to run tests against a real kafka cluster

Bugfix

• fix CVE-2023-37920 Removal of e-Tugra root certificate
• fix CVE-2023-43804 Cookie HTTP header isn't stripped on cross-origin redirects
• fix CVE-2023-37276 aiohttp.web.Application vulnerable to HTTP request smuggling via llhttp HTTP request parser

Details

• Add creation timestamp to pre-detections by @ppcad in #436
• add env variable enrichment preprocessor by @ekneg54 in #448
• Fix CVE 2023 37920 by @ekneg54 in #445
• fix main pipeline by @ekneg54 in #449
• fix memory leak by @ekneg54 in #452
• add grafana and prometheus stack to quickstart by @ekneg54 in #450
• fix CVE-2023-43804 by @ekneg54 in #454
• add inline rule config feature to pipline config by @ekneg54 in #453
• Remove MetricFileTarget by @ekneg54 in #456
• remove process_strategy and mermaid by @ekneg54 in #451
• remove MultiprocessLogHandler by @ekneg54 in #455
• rewrite kafka connector by @ekneg54 in #444
• add option to ignore missing fields in field_manager by @ekneg54 in #457
• prepare release 7.0.0 by @dtrai2 in #461

Full Changelog: v6.8.1...v7.0.0

v6.8.1

Bugfix

• Fix writing time measurements into the event after the deleter has deleted the event. The bug only
  happened when the metrics.measure_time.append_to_event configuration was set to true.

• Fix memory leak by removing the log aggregation capability

Details

• Fix time_measurement for empty events after deleter deleted the event by @dtrai2 in #437
• Refactor rule tree by @ppcad in #416
• time measurement checks for caller class name now by @dtrai2 in #440
• remove logaggregator by @ekneg54 in #439
• prepare release 6.8.1 by @ekneg54 in #442

Full Changelog: v6.8.0...v6.8.1

v6.8.0

Features

• Add option to repeat input documents for the following connectors: DummyInput, JsonInput,
  JsonlInput. This enables easier debugging by introducing a continues input stream of documents.

Bugfix

• Fix restarting of logprep every time the kafka input connector receives events that aren't valid
  json documents. Now the documents will be written to the error output.
• Fix ProcessCounter to actually print counts periodically and not only once events are processed

Details

• Allow repeating events for dummy inputs by @dtrai2 in #426
• fix crash on invalid json input in kafka input connector by @dtrai2 in #428
• fix and revise logprep quickstart by @dtrai2 in #427
• revise processing counter by @dtrai2 in #431
• Prepare Release v6.8.0 by @dtrai2 in #432

Full Changelog: v6.7.0...v6.8.0

v6.7.0

Improvements

• Print logprep warnings in the rule corpus tester only in the detailed reports instead of the
  summary.

Bugfix

• Fix error when writing too large documents into Opensearch/Elasticsearch
• Fix dissector pattern that end with a dissect, e.g system_%{type}
• Handle long-running grok pattern in the Grokker by introducing a timeout limit of one second
• Fix time handling: If no year is given assume the current year instead of 1900 and convert time
  zone only once

Details

• Fix error in ES/OS output connectors caused by to large documents by @ppcad in #419
• Fix CHANGELOG.md by @dtrai2 in #420
• fix dissect pattern that end with a dissect by @dtrai2 in #417
• Change timestamp conversion by @ppcad in #421
• Handle long-running grok pattern by introducing a timeout by @dtrai2 in #422
• Print warnings in detailed reports only in rule corpus tester by @dtrai2 in #404
• Prepare Release v6.7.0 by @dtrai2 in #425

Full Changelog: v6.6.0...v6.7.0