Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable to select whether to prohibit enhancement from being run on alert information to be written back to ES #2542

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

perceptron01
Copy link

@perceptron01 perceptron01 commented Oct 29, 2019

Background and problem

With running enhancements, the data type of elements in a match could be changed from original one due to the enhancements.
As a result, when the data type that Elasticsearch's writeback_index expects does not match the data type in a match to be written to ES, trying to write alert info to Elasticsearch will fail.

This problem especially tends to occur when using enhancements that change the format of @timestamp (or timestamp_field).
In that case, the following error would appear.

ERROR:root:Error writing alert info to Elasticsearch: RequestError(400, 'mapper_parsing_exception', "failed to parse field [match_time] of type [date] in document with id 'XXXXXXXXXXX'")
Traceback (most recent call last):
  File "/<path_to>/venv/lib/python3.6/site-packages/elastalert/elastalert.py", line 1625, in writeback
    res = self.writeback_es.index(index=index, body=body)
  File "/<path_to>/venv/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/<path_to>/venv/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 364, in index
    "POST", _make_path(index, doc_type, id), params=params, body=body
  File "/<path_to>/venv/lib/python3.6/site-packages/elasticsearch/transport.py", line 350, in perform_request
    timeout=timeout,
  File "/<path_to>/venv/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 156, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/<path_to>/venv/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', "failed to parse field [match_time] of type [date] in document with id 'XXXXXXXXXXX'")

Several related issues have also been reported.
#1047 #2036

Cause

The cause of this problem is that, in the current code, it is the alert info modified by enhancements that will be written back to ES.

Solution

Add an option which realizes that enhancements will be run only on the content of alerts send by Alerters, whereas info about the alerts to be written back into Elasticsearch will not be processed by enhancements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant