elasticsearch compatibility above 7.0.0 #3066

priamai · 2020-12-11T16:32:23Z

I have noticed that the elast-server is pinned to pip packaege of elasticsearch to 7.0.0.
When I try to run with ES of latest version like 7.4.0 the server fails because doc_type is not supported anymore.
What's the best way to add an argument to match the ES version?
Open to suggestions.

priamai · 2020-12-11T16:33:34Z

Okay I can see there is a TODO comment here:
https://github.com/Yelp/elastalert/blob/ea62cf4bd1a62be3616f40992d088e58e2727f40/elastalert/create_index.py
is it tracked anywhere ?

nsano-rururu · 2020-12-11T17:13:26Z

doc_type is deprecated and will be removed in ES 8

nsano-rururu · 2020-12-11T17:14:28Z

I'm using elasticsearch 7.9.3 with pip packaege of elasticsearch to 7.0.0, but it works fine.

nsano-rururu · 2020-12-11T17:15:26Z

How can I get it stuck?
Do you do something that I can't reproduce?

nsano-rururu · 2020-12-11T17:27:52Z

Environment is docker

praecoapp/elastalert-server
praecoapp/praeco
elasticsearch 7.9.3
kibana 7.9.3

nsano-rururu · 2020-12-11T17:29:14Z

Currently, ElastAlert seems to be out of maintenance, so there are no signs of new features or bug fixes.

priamai · 2020-12-11T17:38:43Z

Yes I noticed this is what I tried to do but fails:

docker run -d --rm -p 9200:9200 -p 9300:9300\
  --network="host" --name="elasticsearch" -e "discovery.type=single-node"\
   docker.elastic.co/elasticsearch/elasticsearch:7.4.0


docker run -d --rm -p 3030:3030 -p 3333:3333 --network="host" \
    -v `pwd`/config/elastalert.yaml:/opt/elastalert/config.yaml \
    -v `pwd`/config/config.json:/opt/elastalert-server/config/config.json \
    -v `pwd`/rules:/opt/elastalert/rules \
    -v `pwd`/rule_templates:/opt/elastalert/rule_templates \
    --name elastalert bitsensor/elastalert:3.0.0-beta.0


docker run --rm -p 8080:8080 --network="host" \
    -v `pwd`/public/praeco.config.json:/var/www/html/praeco.config.json \
    -v `pwd`/nginx_config/nginx.conf:/etc/nginx/nginx.conf \
    -v `pwd`/nginx_config/default.conf:/etc/nginx/conf.d/default.conf \
    --name praeco praecoapp/praeco

Error:

WARNING: Published ports are discarded when using host network mode
Starting Nginx
nginx: [emerg] host not found in upstream "elastalert" in /etc/nginx/conf.d/default.conf:11

nsano-rururu · 2020-12-11T17:59:58Z

bitsensor / elastalert and servercentral / elastalert are not maintained.
The developer of servercentral / elastalert forks and maintains praecoapp / elastalert-server.
praecoapp / elastalert-server and praecoapp / praeco is also a co-maintainer for me.

Docker image name	tag	ElastAlert	Elasticsearch 7 Support	Remarks
bitsensor/elastalert	2.0.1	0.1.39	×	python 2.7
bitsensor/elastalert	lastet	0.1.39	×	python 2.7
bitsensor/elastalert	3.0.0-beta.0	0.2.0b2	〇	python 2.7
bitsensor/elastalert	3.0.0-beta.1	0.2.0b2	〇	python 2.7
servercentral/elastalert	latest	0.2.1	〇	python 3.6 bitsensor/elastalert fork Customize bug fix
daichi703n/elastalert	0.2.1-dev2	0.2.1	〇	python 3.6 servercentral/elastalert fork Customize bugfix
praecoapp/elastalert-server	latest	0.2.4	〇	python 3.8 servercentral/elastalert fork bug fix Library Update

nsano-rururu · 2020-12-11T18:22:12Z

default.conf, nginx.conf, praeco.config.json, api.config.json, elastalert.yaml are from praeco's github

/home/sano/dkwork2/es
|--Dockerfiles
|  |--Dockerfile.elastalert
|--docker-compose.yml
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|--kibana
|  |--config
|  |  |--kibana.yml
|--praeco
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--api.config.json
|  |  |--elastalert.yaml
|  |--nginx_config
|  |  |--default.conf
|  |  |--nginx.conf
|  |--public
|  |  |--praeco.config.json
|  |--rule_templates
|  |--rules
|  |  |--BaseRule.config

Dockerfiles/Dockerfile.elastalert

FROM praecoapp/elastalert-server:latest

USER root

RUN apk update && \
    apk add vim bash curl && \
    rm -rf /var/cache/apk/*

ADD praeco/bin/elastalert-start.sh /usr/local/bin/
ADD praeco/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

praeco/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

praeco/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx512m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.9.3
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile.elastalert
    image: elastalert-server:3.0.0
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./praeco/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./praeco/config/api.config.json:/opt/elastalert-server/config/config.json
      - ./praeco/rules:/opt/elastalert/rules
      - ./praeco/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  praeco:
    container_name: praeco
    image: praecoapp/praeco:latest
    ports:
      - 8080:8080
    depends_on:
      - elastalert
    restart: always
    volumes:
      - ./praeco/public/praeco.config.json:/var/www/html/praeco.config.json
      - ./praeco/nginx_config/nginx.conf:/etc/nginx/nginx.conf
      - ./praeco/nginx_config/default.conf:/etc/nginx/conf.d/default.conf
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:8080 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

chmod -R 777 es/data
chmod -R 777 rules rule_templates
docker-compose up -d

docker logs -f elastalert

Giving Elasticsearch at  time to start...
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!

> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

18:05:17.808Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
18:05:17.810Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
18:05:17.839Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
18:05:17.904Z  INFO elastalert-server: Router:  Listening for GET request on /.
18:05:17.905Z  INFO elastalert-server: Router:  Listening for GET request on /status.
18:05:17.906Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
18:05:17.907Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
18:05:17.910Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
18:05:17.911Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
18:05:17.940Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
18:05:17.941Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
18:05:17.942Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
18:05:17.943Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
18:05:17.943Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
18:05:17.945Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
18:05:17.945Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
18:05:17.946Z  INFO elastalert-server: Router:  Listening for POST request on /test.
18:05:17.949Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
18:05:17.956Z  INFO elastalert-server: Router:  Listening for GET request on /config.
18:05:17.957Z  INFO elastalert-server: Router:  Listening for POST request on /config.
18:05:17.958Z  INFO elastalert-server: Router:  Listening for POST request on /download.
18:05:17.959Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
18:05:17.960Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
18:05:17.961Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
18:05:17.962Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
18:05:17.962Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
18:05:17.967Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
18:05:17.968Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
18:05:17.969Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
18:05:17.969Z  INFO elastalert-server: Router:  Listening for GET request on /config.
18:05:17.977Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
18:05:17.978Z  INFO elastalert-server: ProcessController:  Creating index
18:05:19.206Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.9.3
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    Index praeco_elastalert_status already exists. Skipping index creation.

priamai · 2020-12-11T21:40:36Z

Thanks ver much for your config!
I do notice some FATAL errors if you leave it running long enough on Kibana:

kibana           | 
kibana           |  FATAL  Error: Unknown configuration key(s): "cluster.name", "network.host", "discovery.zen.minimum_master_nodes". Check for spelling errors and ensure that expected plugins are installed.

Have you noticed those?

nsano-rururu · 2020-12-12T03:54:07Z

I woke up now due to lack of sleep.
Wrong copy of kibana.yml settings. I modified it to the actual setting.

priamai · 2020-12-12T07:18:37Z

@nsano-rururu thank you for all the support is working now I also woke up early!

I have some quick questions:

if this is not mantained anymore servercentral / elastalert, does it mean that we will not have any future feature releases but just maybe ES compatibility releases?
for the table praecoapp/elast-server the tickbox for ES7 is not enabled does it mean has not been tested extensively? Do you have some unit tests that you run to make sure the generated ruels are working? I could help with that since I need it.
is this the only tutorial available: https://medium.com/@john_8166/praeco-walkthrough-5aada7e078a9

Thank you very much.

nsano-rururu · 2020-12-12T12:41:22Z

@priamai

if this is not mantained anymore servercentral / elastalert, does it mean that we will not have any future feature releases but just maybe ES compatibility releases?

servercentral / elastalert was what Praeco maintainer John Susek did at work at his previous company. servercentral / elastalert forked servercentral / elastalert johnsusek/elastalert-server because John Susek no longer has administrator privileges. Now I'm moving to another company, and I'm a co-maintainer for maintenance because I can't spend time on that work, including Praeco. Will it be maintained in the future? I don't know It's not something you have to promise. I'm not obsessed with you, and I'm not getting paid for this maintenance. First of all, don't you understand what OSS is like?

About OSS

Manuals may not be organized
The developer is not responsible for fixing the bug
In many cases, the support of the developer cannot be received

for the table praecoapp/elast-server the tickbox for ES7 is not enabled does it mean has not been tested extensively? Do you have some unit tests that you run to make sure the generated ruels are working? I could help with that since I need it.

I don't know what you're talking about. What are ES7 checkboxes? .. Unit tests haven't been done since bitsensor

is this the only tutorial available: https://medium.com/@john_8166/praeco-walkthrough-5aada7e078a9

https://medium.com/@ibrahim.ayadhi/hello-and-welcome-to-our-new-article-which-will-be-covering-the-alerting-part-in-our-socaas-136cf6258c49
https://designetwork.daichi703n.com/entry/2019/03/24/Praeco-ElastAlert-GUI
https://designetwork.daichi703n.com/entry/2020/02/10/praeco-elastalert-issue-with-es7
https://designetwork.daichi703n.com/entry/2020/02/24/praeco-helm-kubernetes

nsano-rururu · 2020-12-12T13:17:52Z

@priamai

I think it is doubtful to understand the relationship between the elasticsearch-py version and the elasticsearch version, so I will explain it.

# Elasticsearch 7.x
elasticsearch>=7.0.0,<8.0.0

# Elasticsearch 6.x
elasticsearch>=6.0.0,<7.0.0

# Elasticsearch 5.x
elasticsearch>=5.0.0,<6.0.0

# Elasticsearch 2.x
elasticsearch>=2.0.0,<3.0.0

See the comments below for the reason why elasticsearch == 7.0.0 is set.
#2593 (comment)

nsano-rururu · 2020-12-12T15:08:05Z

@priamai

Do you understand this figure first?
https://user-images.githubusercontent.com/611996/52892144-90a19300-3155-11e9-8050-cb4a440411a4.png

nsano-rururu · 2020-12-12T15:35:45Z

@priamai

I'm confused, but I don't usually use Elast Alert at all. I'm also not the maintainer of ElastAlert.
I understand that if you only support ElasticSearch 7, you should delete the "doc_type" part like the forked repository below.
jsonar@1243d15
jsonar@20e3c52

nsano-rururu · 2020-12-12T16:19:10Z

@priamai

ElastAlert is a program written in Python.
ElastAlertServer is an API made with Node.js. Another name is ElastAlertAPI.
It cannot be used with ElastAlertServer alone. Image issuing instructions to ElastAlert via API of ElastAlertServer.
First of all, you can't understand from there.

nsano-rururu · 2020-12-12T16:25:06Z

@priamai

Whether or not elasticsearch-py is guaranteed to work with the latest version is up to the maintainers of ElastAlert, and I'm not doing it. Please understand that.

priamai · 2020-12-12T23:53:38Z

Hi @nsano-rururu I am just reading now, this is a gold mine of information to understand the history of the project.
I didn't know about the relation between you and John and I was assuming that your company was sponsoring the project. The advice and support you provided it is certainly above the expectation of the average OSS project as you said and I wasn't expecting such a fast and accurate feedback.
Thank you a lot for your information and I am going to close this issue since you have provided more than enough to understand how the projects and dependencies are related.
I will certainly do my best with my startup to contribute to praeco because we like it very much.
Thanks and regards.

priamai closed this as completed Dec 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

elasticsearch compatibility above 7.0.0 #3066

elasticsearch compatibility above 7.0.0 #3066

priamai commented Dec 11, 2020

priamai commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

priamai commented Dec 11, 2020 •

edited

Loading

nsano-rururu commented Dec 11, 2020 •

edited

Loading

nsano-rururu commented Dec 11, 2020 •

edited

Loading

priamai commented Dec 11, 2020

nsano-rururu commented Dec 12, 2020

priamai commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

priamai commented Dec 12, 2020

elasticsearch compatibility above 7.0.0 #3066

elasticsearch compatibility above 7.0.0 #3066

Comments

priamai commented Dec 11, 2020

priamai commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

nsano-rururu commented Dec 11, 2020

priamai commented Dec 11, 2020 • edited Loading

nsano-rururu commented Dec 11, 2020 • edited Loading

nsano-rururu commented Dec 11, 2020 • edited Loading

priamai commented Dec 11, 2020

nsano-rururu commented Dec 12, 2020

priamai commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

nsano-rururu commented Dec 12, 2020

priamai commented Dec 12, 2020

priamai commented Dec 11, 2020 •

edited

Loading

nsano-rururu commented Dec 11, 2020 •

edited

Loading

nsano-rururu commented Dec 11, 2020 •

edited

Loading