Bugfix and better error handling on zabbix alerter

[david-ns]

Zabbix alerter changes:

• Imports changed to relative imports
• Elastalert logger is now used
• Error handling when ZabbixSender causes an exception to take advantage of elastalert's realerting capabilities
• New warning when the configured host or item doesn't exists
• Some code formatting
• Bugfix on ZabbixClient when 'send_aggregated_metrics' was disabled
• Better handling of timestamps

[alvarolmedo]

+1

[alvarolmedo]

The #2605 has been included in this PR.

[alvarolmedo]

@Qmando what do you thing about this improvement?

[nsano-rururu]

I tried to move it, but an error occurs.

    Traceback (most recent call last):
      File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/opt/elastalert/elastalert/elastalert.py", line 2040, in <module>
        sys.exit(main(sys.argv[1:]))
      File "/opt/elastalert/elastalert/elastalert.py", line 2034, in main
        client = ElastAlerter(args)
      File "/opt/elastalert/elastalert/elastalert.py", line 138, in __init__
        self.rules = self.rules_loader.load(self.conf, self.args)
      File "/opt/elastalert/elastalert/loaders.py", line 125, in load
        raise EAException('Error loading file %s: %s' % (rule_file, e))
    elastalert.util.EAException: Error loading file rules/a.yaml: Error initiating alert ['zabbix']: Could not import module zabbix: not enough values to unpack (expected 2, got 1)

You also need to modify loaders.py described in the following issue.

#2586 (comment)

Was looking into the code and found that the zabbix alert is not in the loaders.py - so possibly is not loaded at all!
I added these lines:
from .zabbix import ZabbixAlerter

and
'zabbix': ZabbixAlerter
to the alerts_mapping function

[david-ns]

@nsano-rururu I've fixed issue #2586, thanks for the reference. Could you please test it again? Thanks!

[nsano-rururu]

I confirmed the operation. It seems to work normally

/home/sano/docker-wk3
|--Dockerfiles
|  |--elastalert-dockerfile
|  |--fluentd-dockerfile
|--data
|  |--fonts
|  |--mysql
|  |--zabbix
|--docker-compose.yml
|--elastalert
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--api.config.json
|  |  |--elastalert.yaml
|  |--rule_templates
|  |--rules
|  |  |--a.yaml
|  |--zabbix
|  |  |--loaders.py
|  |  |--zabbix.py
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|--fluentd
|  |--etc
|  |  |--fluent.conf
|  |  |--mysql_template.json
|--kibana
|  |--config
|  |  |--kibana.yml
|--mariadb
|  |--etc
|  |  |--mymariadb.cnf
|  |--log
|  |  |--error.log
|  |  |--general.log
|  |  |--slow.log

chmod 777 es/data
chmod 777 elastalert/rule_templates
chmod 777 elastalert/rules
chmod 777 mariadb/log
chmod 777 mariadb/log/error.log
chmod 777 mariadb/log/general.log
chmod 777 mariadb/log/slow.log
chmod 777 mariadb/log/error.log
chmod 777 mariadb/log/general.log
chmod 777 mariadb/log/slow.log
docker-compose up -d

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms128m -Xmx256m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.6.2
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  fluentd:
    container_name: fluentd
    build:
      context: .
      dockerfile: Dockerfiles/fluentd-dockerfile
    image: fluentd:1.10.2
    ports:
      - 24224:24224
      - 24224:24224/udp
    environment:
      - FLUENTD_CONF=fluent.conf
    volumes:
      - ./fluentd/etc/fluent.conf:/fluentd/etc/fluent.conf
      - ./fluentd/etc/mysql_template.json:/fluentd/etc/mysql_template.json
      - ./mariadb/log:/var/log/mysql
    user: root
    restart: always

  mariadb:
    container_name: mariadb
    image: mariadb:10.4.11
    ports:
      - 3306:3306
    environment:
      - MYSQL_ROOT_PASSWORD=mariadb
    volumes:
      - ./mariadb/etc:/etc/mysql/conf.d
      - ./mariadb/log:/var/log/mysql
    restart: always
    healthcheck:
        test: ["CMD-SHELL", "mysqladmin -h 'localhost' -u root -pmariadb ping --silent"]
        interval: 30s
        timeout: 30s
        retries: 3

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/elastalert-dockerfile
    image: elastalert:0.2.1
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./elastalert/config/api.config.json:/opt/elastalert-server/config/config.json
      - ./elastalert/rules:/opt/elastalert/rules
      - ./elastalert/rule_templates:/opt/elastalert/rule_templates
      - ./elastalert/zabbix/zabbix.py:/opt/elastalert/elastalert/zabbix.py
      - ./elastalert/zabbix/loaders.py:/opt/elastalert/elastalert/loaders.py 
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  mysql:
    image: mysql:5.7
    restart: always
    command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci
    volumes:
      - ./data/mysql:/var/lib/mysql
    ports:
      - "33060:3306"
    environment:
      MYSQL_ROOT_PASSWORD: docker
      MYSQL_DATABASE: zabbix
      MYSQL_USER: zabbix
      MYSQL_PASSWORD: zabbix

  zabbix-server:
    image: zabbix/zabbix-server-mysql:alpine-3.4-latest
    restart: always
    volumes:
      - ./data/zabbix:/var/lib/zabbix
    environment:
      DB_SERVER_HOST: mysql
      MYSQL_USER: zabbix
      MYSQL_PASSWORD: zabbix
    ports:
      - "10051:10051"
    depends_on:
      - mysql

  zabbix-web:
    image: zabbix/zabbix-web-nginx-mysql:alpine-3.4-latest
    restart: always
    volumes:
      - ./data/fonts:/usr/share/zabbix/fonts
    environment:
      DB_SERVER_HOST: mysql
      MYSQL_USER: zabbix
      MYSQL_PASSWORD: zabbix
      ZBX_SERVER_HOST: zabbix-server
      PHP_TZ: Asia/Tokyo
    ports:
      - "80:80"
    depends_on:
      - mysql
      - zabbix-server

Dockerfiles/elastalert-dockerfile

FROM servercentral/elastalert:latest

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/* 

ADD ./elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD ./elastalert/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh && \
    chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

Dockerfiles/fluentd-dockerfile

FROM fluent/fluentd:v1.10.2-debian-1.0

# Use root account to use apt
USER root

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish
RUN buildDeps="sudo make gcc g++ libc-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install fluent-plugin-elasticsearch -v 4.0.7 \
 && sudo gem install fluent-plugin-mysqlslowquery -v 0.0.9 \
 && sudo gem install elasticsearch-xpack -v 7.6.0 \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent

elastalert/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

elastalert/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

elastalert/config/api.config.json

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "es_username": "",
  "es_password": "",
  "es_ssl": false,
  "writeback_index": "elastalert_status"
}

elastalert/config/elastalert.yaml

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 60

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

skip_invalid: True

elastalert/rules/a.yaml

alert:
  - zabbix
zbx_sender_host: zabbix-server
zbx_sender_port: 10051
zbx_host: "test001"
zbx_key: "sender_load1"
alert_subject: a
alert_subject_args: []
alert_text: b
alert_text_args: []
filter:
  - query:
      query_string:
        query: 'message:Quit'
index: mariadblog-*
is_enabled: true
name: a
num_events: 1
realert:
  minutes: 5
terms_size: 50
timeframe:
  minutes: 1
timestamp_field: '@timestamp'
timestamp_type: iso
type: frequency
use_strftime_index: false

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

kibana/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

fluentd/etc/fluent.conf

<source>
  @type mysql_slow_query
  path /var/log/mysql/slow.log
  pos_file /tmp/mysql/slow.pos
  tag mysqld.slow_query
   <parse>
     @type none
   </parse>
</source>
<source>
  @type tail
  format none
  path /var/log/mysql/general.log
  pos_file /tmp/mysql/general.pos
  tag mysqld.general
</source>
<source>
  @type tail
  format none
  path /var/log/mysql/error.log
  pos_file /tmp/mysql/error.pos
  tag mysqld.error
</source>

<match **.**>
  @type copy
  <store>
    @type stdout
  </store>
  <store>
    @type elasticsearch
    include_tag_key true
    include_timestamp true
    tag_key @log_name
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix mariadblog
  </store>
</match>

fluentd/etc/mysql_template.json

{
    "index_patterns": ["mysql-log-*"], 
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "myisql-policy", 
      "index.lifecycle.rollover_alias": "mysql-log"
    }
  }
  

mariadb/etc/mymariadb.cnf

[mysqld]
general_log
general_log_file=/var/log/mysql/general.log
slow_query_log
slow_query_log_file=/var/log/mysql/slow.log
long_query_time=5
log-queries-not-using-indexes
log-error=/var/log/mysql/error.log

[alvarolmedo]

Thanks for your review @nsano-rururu

[nsano-rururu]

@david-ns

I noticed that I forgot to add "py-zabbix == 1.1.3" to setup.py.
The latest version of py-zabbix is 1.1.7.
I have confirmed that ElastAlert works properly with py-zabbix 1.1.7.
I think you should take the following actions in addition to this pull request.

■ Modification of requirements.txt

py-zabbix==1.1.3
↓
py-zabbix>=1.1.3
  
■ Added to setup.py

'py-zabbix>=1.1.3'

[nsano-rururu]

@david-ns

I noticed that I forgot to add "py-zabbix == 1.1.3" to setup.py.
The latest version of py-zabbix is 1.1.7.
I have confirmed that ElastAlert works properly with py-zabbix 1.1.7.
I think you should take the following actions in addition to this pull request.

■ Modification of requirements.txt

py-zabbix==1.1.3
↓
py-zabbix>=1.1.3
  
■ Added to setup.py

'py-zabbix>=1.1.3'

Other modules seem to be inconsistent in requirements.txt and setup.py.
I plan to issue a pull request.

[nsano-rururu]

Merged into elastalert2.
jertel/elastalert2#26