Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to create a folder - only from GUI #219

Closed
talcoh2x opened this issue Jul 16, 2020 · 15 comments
Closed

failed to create a folder - only from GUI #219

talcoh2x opened this issue Jul 16, 2020 · 15 comments
Labels
bug Something isn't working

Comments

@talcoh2x
Copy link

astalert_1 | (node:17) UnhandledPromiseRejectionWarning: TypeError: invalid options argument
elastalert_1 | at optsArg (/opt/elastalert-server/node_modules/mkdirp/lib/opts-arg.js:13:11)
elastalert_1 | at mkdirp (/opt/elastalert-server/node_modules/mkdirp/index.js:11:10)
elastalert_1 | at /opt/elastalert-server/src/common/file_system.js:84:11
elastalert_1 | (Use node --trace-warnings ... to show where the warning was created)
elastalert_1 | (node:17) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
elastalert_1 | (node:17) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

image
@talcoh2x
Copy link
Author

ELK - 7.6.2
praeco - latest release

@nsano-rururu
Copy link
Collaborator

@talcoh2x

Not enough information.

・Did ElastAlert Server recognize that it used the following Docker image? ..
https://hub.docker.com/r/johnsusek/elastalert-server
・Did ElastAlert Server, Praeco, and Elasticserarch move with Docker? Or is it a direct install? ..

@nsano-rururu
Copy link
Collaborator

@talcoh2x

I understand that there may be a place where I need to fix it so that try {...} catch() {...} can be caught properly and execute reject, but the source code of which part of ElastAlert Server is still there. It is unconfirmed whether it is occurring in.
Could you tell me what kind of operation occurred as a reference when reproducing the error?

@nsano-rururu
Copy link
Collaborator

@talcoh2x

Node.js is not very familiar with it, but at first glance it seems that try {} catch (error) {reject(error);} needs to be supported.

src/common/config/server_config.js
Near Line 137
src/common/file_system.js
Near Line 78
Near Line 98
Near Line 111
Near Line 131

@nsano-rururu
Copy link
Collaborator

nsano-rururu commented Jul 16, 2020 •
edited
Loading

@johnsusek

Corresponding image

Before correspondence

  createDirectoryIfNotExists(pathToFolder) {
    let self = this;

    return new Promise(function (resolve, reject) {
      self.directoryExists(pathToFolder).then(function (exists) {
        if (!exists) {
          mkdirp(pathToFolder, function (error) {
            if (error) {
              reject(error);
            } else {
              resolve();
            }
          });
        } else {
          resolve();
        }
      });
    });
  }

Correspondence example

Added "}).catch((error) => {reject(error);"

  createDirectoryIfNotExists(pathToFolder) {
    let self = this;

    return new Promise(function (resolve, reject) {
      self.directoryExists(pathToFolder).then(function (exists) {
        if (!exists) {
          mkdirp(pathToFolder, function (error) {
            if (error) {
              reject(error);
            } else {
              resolve();
            }
          });
        } else {
          resolve();
        }
      });
    }).catch((error) => {
       reject(error);
    });
  }

@talcoh2x
Copy link
Author

@talcoh2x

Not enough information.

・Did ElastAlert Server recognize that it used the following Docker image? ..
https://hub.docker.com/r/johnsusek/elastalert-server
・Did ElastAlert Server, Praeco, and Elasticserarch move with Docker? Or is it a direct install? ..

im using with the Docker for ElastAlert Server and Praeco and for Elasticsearch as direct install

@nsano-rururu
Copy link
Collaborator

nsano-rururu commented Jul 17, 2020 •
edited
Loading

@talcoh2x

The error could not be reproduced.
It works without problems.

johnsusek/elastalert-server:latest
Elasticsearch 7.6.2
Kibana 7.6.2

/home/sano/dkwork2/es
|--Dockerfiles
|  |--Dockerfile.elastalert
|--docker-compose.yml
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|--kibana
|  |--config
|  |  |--kibana.yml
|--praeco
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--api.config.json
|  |  |--elastalert.yaml
|  |--nginx_config
|  |  |--default.conf
|  |  |--nginx.conf
|  |--public
|  |  |--favicon.ico
|  |  |--index.html
|  |  |--js
|  |  |  |--cron-ui.min.js
|  |  |--praeco.config.json
|  |--rule_templates
|  |--rules

Dockerfiles/Dockerfile.elastalert

FROM johnsusek/elastalert-server:latest 

USER root

RUN apk add --update --no-cache net-snmp net-snmp-tools
RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD praeco/bin/elastalert-start.sh /usr/local/bin/
ADD praeco/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx512m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.6.2
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s
 
  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile.elastalert
    image: elastalert-server:0.2.4
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./praeco/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./praeco/config/api.config.json:/opt/elastalert-server/config/config.json
      - ./praeco/rules:/opt/elastalert/rules
      - ./praeco/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  praeco:
    container_name: praeco
    image: johnsusek/praeco:latest
    ports:
      - 8080:8080
    depends_on:
      - elastalert
    restart: always
    volumes:
      - ./praeco/public/praeco.config.json:/var/www/html/praeco.config.json
      - ./praeco/nginx_config/nginx.conf:/etc/nginx/nginx.conf
      - ./praeco/nginx_config/default.conf:/etc/nginx/conf.d/default.conf
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:8080 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

praeco/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

praeco/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

praeco/config/api.config.json

Default settings

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "es_username": "",
  "es_password": "",
  "es_ssl": false,
  "writeback_index": "praeco_elastalert_status"
}

praeco/config/elastalert.yaml

Default settings

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 60

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: praeco_elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

skip_invalid: True

profile: default

praeco/public/praeco.config.json

Default settings

{
  "appUrl": "http://127.0.0.1:8080",
  "errorLoggerUrl": "",
  "hidePreconfiguredFields": []
}

execute

chmod 777 es/data
chmod 777 praeco/rule_templates
chmod 777 praeco/rules
docker-compose up -d
docker logs -f elastalert

Giving Elasticsearch at  time to start...
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!

> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

13:25:51.777Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
13:25:51.788Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
13:25:51.788Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
13:25:51.801Z  INFO elastalert-server: Router:  Listening for GET request on /.
13:25:51.802Z  INFO elastalert-server: Router:  Listening for GET request on /status.
13:25:51.803Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
13:25:51.804Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
13:25:51.807Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
13:25:51.807Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
13:25:51.808Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
13:25:51.809Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
13:25:51.810Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
13:25:51.811Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
13:25:51.812Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
13:25:51.812Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
13:25:51.813Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
13:25:51.815Z  INFO elastalert-server: Router:  Listening for POST request on /test.
13:25:51.815Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
13:25:51.816Z  INFO elastalert-server: Router:  Listening for GET request on /config.
13:25:51.817Z  INFO elastalert-server: Router:  Listening for POST request on /config.
13:25:51.817Z  INFO elastalert-server: Router:  Listening for POST request on /download.
13:25:51.818Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
13:25:51.819Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
13:25:51.820Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
13:25:51.820Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
13:25:51.823Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
13:25:51.824Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
13:25:51.824Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
13:25:51.824Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
13:25:51.825Z  INFO elastalert-server: Router:  Listening for GET request on /config.
13:25:51.829Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
13:25:51.829Z  INFO elastalert-server: ProcessController:  Creating index
13:25:58.634Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.6.2
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index praeco_elastalert_status created
    Done!
    
13:25:58.635Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
13:25:58.635Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
13:25:58.653Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 289)
13:25:58.656Z  INFO elastalert-server: Server:  Server listening on port 3030
13:25:58.657Z  INFO elastalert-server: Server:  Websocket listening on port 3333
13:25:58.658Z  INFO elastalert-server: Server:  Server started
13:25:58.696Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
13:26:29.056Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
13:26:59.358Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
13:27:18.085Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
13:27:18.099Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
13:27:18.104Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
13:27:18.106Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
13:27:18.136Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules'.
13:27:29.697Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.

キャプチャ

@talcoh2x
Copy link
Author

talcoh2x commented Jul 17, 2020 via email

@nsano-rururu
Copy link
Collaborator

Are you talking about this?

Incorrect permissions inside the Docker container? #101
Rule not found, it may have been moved or deleted. #157

It is needed to stop the docker instance and give the adequate rights to rules and rule_templates.
chmod -R 777 rules rule_templates

@nsano-rururu nsano-rururu added the question Further information is requested label Aug 9, 2020
@nsano-rururu
Copy link
Collaborator

@talcoh2x

Is it possible to provide the executed command, docker-compose.yml, configuration file, etc.?

@nsano-rururu nsano-rururu self-assigned this Aug 9, 2020
@nsano-rururu
Copy link
Collaborator

@talcoh2x

We will lower the priority of the survey as there are other high priority tasks. Please understand.

@nsano-rururu nsano-rururu removed their assignment Aug 15, 2020
@nsano-rururu nsano-rururu added the bug Something isn't working label Aug 18, 2020
@nsano-rururu
Copy link
Collaborator

I closed it because there was no reply and it was difficult to continue the investigation. Will reopen if you have any comments

@talcoh2x
Copy link
Author

@nsano-rururu sorry about that, I really want to fix it. let me share all the details

what I'm trying to do is to create a new folder and there a new rule

I clone the REPO and run:
I have 777 on rules and rule_templates
docker-compose up --force-recreate --build -d

praeco/config/elastalert.yaml

The elasticsearch hostname for metadata writeback

Note that every rule can have its own elasticsearch host

es_host: "?????"

The elasticsearch port

es_port: 9200

This is the folder that contains the rule yaml files

Any .yaml file will be loaded as a rule

rules_folder: rules

How often ElastAlert will query elasticsearch

The unit can be anything from weeks to seconds

run_every:
seconds: 60

ElastAlert will buffer results from the most recent

period of time, in case some log sources are not in real time

buffer_time:
minutes: 1

Optional URL prefix for elasticsearch

#es_url_prefix: elasticsearch

Connect with TLS to elasticsearch

#use_ssl: True

Verify TLS certificates

#verify_certs: True

GET request with body is the default option for Elasticsearch.

If it fails for some reason, you can pass 'GET', 'POST' or 'source'.

See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport

for details

#es_send_get_body_as: GET

Option basic-auth username and password for elasticsearch

es_username: "elastic"
es_password: "?????"

The index on es_host which is used for metadata storage

This can be a unmapped index, but it is recommended that you run

elastalert-create-index to set a mapping

writeback_index: praeco_elastalert_status

If an alert fails for some reason, ElastAlert will retry

sending the alert until this time period has elapsed

alert_time_limit:
days: 2

skip_invalid: True

@nsano-rururu nsano-rururu reopened this Aug 23, 2020
@nsano-rururu nsano-rururu removed the question Further information is requested label Aug 23, 2020
@nsano-rururu
Copy link
Collaborator

@talcoh2x
Released a modified version of Docker image.
https://hub.docker.com/r/praecoapp/elastalert-server

By the way, execute the following command for the rules and rule template directories.

sudo chmod -R 777 praeco/rules/
sudo chmod -R 777 praeco/rule_templates/

@nsano-rururu
Copy link
Collaborator

@talcoh2x

We have released a fixed version of the Docker image because we found a bug that prevented us from creating a rules file.
https://hub.docker.com/r/praecoapp/elastalert-server

Repository owner locked and limited conversation to collaborators Oct 28, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nsano-rururu @talcoh2x