Match_body is not parsing

[aniketpant1]

When i ran elastalert_create_index it show me this

Elastic Version: 7.10.1
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
Traceback (most recent call last):
File "/usr/local/bin/elastalert-create-index", line 11, in
load_entry_point('elastalert==0.2.4', 'console_scripts', 'elastalert-create-index')()
File "/usr/local/lib/python3.6/site-packages/elastalert-0.2.4-py3.6.egg/elastalert/create_index.py", line 264, in main
create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index)
File "/usr/local/lib/python3.6/site-packages/elastalert-0.2.4-py3.6.egg/elastalert/create_index.py", line 72, in create_index_mappings
body=es_index_mappings['past_elastalert'], include_type_name=True)
File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/client/indices.py", line 321, in put_mapping
"PUT", _make_path(index, "_mapping", doc_type), params=params, body=body
File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/transport.py", line 318, in perform_request
status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/connection/http_requests.py", line 91, in perform_request
self._raise_error(response.status_code, raw_data)
File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/connection/base.py", line 131, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.TransportError: TransportError(500, 'mapper_exception', "the [enabled] parameter can't be updated for the object mapping [match_body]")

and this is my template
PUT /_template/elastalert
{
"index_patterns": ["elastalert*"],
"settings": {
"index": {
"mapping": {
"total_fields.limit": "10000"
}
},
"number_of_replicas": 0,
"number_of_shards": 3
}
,
"mappings": {
"properties": {
"match_body": {
"properties": {
"z_logstash_pipeline": {
"type": "keyword"
},
"etl_pipeline": {
"type": "keyword"
}
}
}
}
}
}

[nsano-rururu]

I checked with the following version with Docker, but the problem did not reproduce. There is no problem with alert notification.

Docker
・Elasticsearch 7.10.1
・Kibana 7.10.1
・Elast
・praecoapp/elastalert-server:latest
　- ElastAlert 0.2.4 + Bugfix etc
・praecoapp/praeco:latest

Giving Elasticsearch at  time to start...
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!

> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

10:13:31.280Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
10:13:31.281Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
10:13:31.290Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
10:13:31.306Z  INFO elastalert-server: Router:  Listening for GET request on /.
10:13:31.306Z  INFO elastalert-server: Router:  Listening for GET request on /status.
10:13:31.307Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
10:13:31.307Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
10:13:31.312Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
10:13:31.312Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
10:13:31.313Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
10:13:31.313Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
10:13:31.313Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
10:13:31.313Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
10:13:31.317Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
10:13:31.317Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
10:13:31.317Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
10:13:31.317Z  INFO elastalert-server: Router:  Listening for POST request on /test.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for GET request on /config.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for POST request on /config.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for POST request on /download.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
10:13:31.318Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
10:13:31.319Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
10:13:31.319Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
10:13:31.319Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
10:13:31.319Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
10:13:31.320Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
10:13:31.320Z  INFO elastalert-server: Router:  Listening for GET request on /config.
10:13:31.331Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
10:13:31.331Z  INFO elastalert-server: ProcessController:  Creating index
10:13:35.335Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.10.1
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index praeco_elastalert_status created
    Done!
    
10:13:35.335Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
10:13:35.336Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
10:13:35.347Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 43)
10:13:35.350Z  INFO elastalert-server: Server:  Server listening on port 3030
10:13:35.390Z  INFO elastalert-server: Server:  Websocket listening on port 3333
10:13:35.394Z  INFO elastalert-server: Server:  Server started
10:13:57.531Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:14:28.007Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:14:58.405Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:15:28.732Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:15:38.131Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:15:38.145Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
10:15:38.150Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
10:15:38.170Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules'.
10:15:38.172Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/templates'.
10:15:59.042Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:16:23.849Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
10:16:29.358Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:16:45.802Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
10:16:55.418Z  INFO elastalert-server: Routes:  Successfully handled POST request for '/rules/:id'.
10:16:55.499Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
10:16:55.502Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
10:16:55.525Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules'.
10:16:55.528Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/templates'.
10:16:59.705Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:17:30.059Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.

[aniketpant1]

I am using elastalert and sigma for transforming windows event logs to elastalert signature.I am using HELK but the component i am using is ELK+kafka+elastalert+sigma.The rules folder contain

This is working fine

This is my script which i've copied but i made some changes.This script is for transforming sigma rules to elastalert signature
pull_sigma.sh
#!/bin/bash
helk_sigmac=/usr/share/sigma/sigmac/sigmac-config.yml
ESALERT_HOME=/usr/share/elastalert/

*********** Unsupported SIGMA Functions ***************

Unsupported feature "near" aggregation operator not yet implemented SigmaHQ/sigma#209

SIGMAremoveNearRules() {
if grep --quiet -E "\s+condition/\s+.*\s+|\s+near\s+" "$1"; then
echo -e "Skipping incompatible rule $1, reference: SigmaHQ/sigma#209"
#rm "$1"
return 0
else
return 1
fi
}

******* Transforming every Windows SIGMA rule to elastalert rules *******

echo " "
echo "Translating SIGMA rules to Elastalert format.."
echo "------------------------------------------------"
echo " "
rule_counter=0

Windows rules

for rule_category in rules/windows/* ; do
echo " "
echo -e "${HELK_INFO_TAG} Working on Folder: $rule_category:"
echo "-------------------------------------------------------------"
if [[ "$rule_category" == "rules/windows/process_creation" ]]; then
for rule in "${rule_category}"/* ; do
if [[ ${rule} != "rules/windows/process_creation/win_mal_adwind.yml" ]]; then
if SIGMAremoveNearRules "$rule"; then
continue
else
echo "[+++] Processing Windows process creation rule: $rule .."
tools/sigmac -t elastalert -c tools/config/generic/sysmon.yml -c "${helk_sigmac}" --backend-option timestamp_field=etl_processed_time -o "${ESALERT_HOME}"/rules/sigma_sysmon_"$(basename "${rule}")" "$rule"
# Give unique rule name for sysmon
sed -i 's/^name: /name: Sysmon_/' "${ESALERT_HOME}"/rules/sigma_sysmon_"$(basename "${rule}")"
tools/sigmac -t elastalert -c tools/config/generic/windows-audit.yml -c "${helk_sigmac}" --backend-option timestamp_field=etl_processed_time -o "${ESALERT_HOME}"/rules/sigma_"$(basename "${rule}")" "$rule"
rule_counter=$[$rule_counter +1]
fi
fi
done
else
for rule in "${rule_category}"/* ; do
if SIGMAremoveNearRules "$rule"; then
continue
else
echo "[+++] Processing additional Windows rule: $rule .."
tools/sigmac -t elastalert -c "${helk_sigmac}" --backend-option timestamp_field=etl_processed_time -o "${ESALERT_HOME}"/rules/sigma_"$(basename "${rule}")" "$rule"
rule_counter=$[$rule_counter +1]
fi
done
fi
done

But problems comes when i have

This is all 800(approx) rules causes elasticsearch lead to crash
Thats why i move all the 800 rules to another directory
and at the end i have few rules(that above picture which start with helk_) to run which does not lead to elasticsearch to crash
by crash means elasticsearch behave abnormal it consumes full heap size and when i don't run those 800 rules it works perfectly
I am not using docker.

[nsano-rururu]

Why don't you delete the index of elastalert created in Elasticsearch with elastalert-create-index and try to execute it again?

[nsano-rururu]

Is python 3.6?

[aniketpant1]

yes it is python 3.6 it is creating index.

[nsano-rururu]

Is it possible to check what happens when the following pull request changes are reflected?
#3016

[aniketpant1]

What pull request ?
in my log file of elastalert it show me
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/connection/http_requests.py", line 77, in perform_request
response = self.session.send(prepared_request, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='elastic', port=9200): Read timed out. (read timeout=20)
ERROR:root:Error running query: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='elastic', port=9200): Read timed out. (read timeout=20))
WARNING:elasticsearch:GET http://elastic:9200/log-wlb-security-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 [status:N/A request:20.026s]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
socket.timeout: timed ou

[aniketpant1]

i have set 'es_conn_timeout: 1000' parameter in elastalert conf file but i think it is not working

[nsano-rururu]

i have set 'es_conn_timeout: 1000' parameter in elastalert conf file but i think it is not working

look
#2469

Did you restart ElastAlert after adding the settings to config.yaml (if it's a docker container, restart ElastAlert's docker container)

[aniketpant1]

I kill the process of elastalert and then again start the elastalert elastalert --verbose --config /path/to/config

…

On Sat, Jan 16, 2021, 9:06 PM Naoyuki Sano ***@***.***> wrote: i have set 'es_conn_timeout: 1000' parameter in elastalert conf file but i think it is not working look #2469 <#2469> Did you restart ElastAlert after adding the settings to config.yaml (if it's a docker container, restart ElastAlert's docker container) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3093 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJVJXF6FKISY7KS6FSAXQT3S2GW73ANCNFSM4WDTOIMA> .

[aniketpant1]

"mappings": {
"properties": {
"match_body": {
"properties": {
"z_logstash_pipeline": {
"type": "keyword"
},
"etl_pipeline": {
"type": "keyword"
}
}
}
}
}
elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'mapper [match_body.etl_pipeline]
cannot be changed from type [long] to [keyword]')
i've set the template and delete the old indices and recreate the index by elastalert-create-index and run the elastalert