Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't authenticate with Elastiscsearch, Kibana server is not ready yet #680

Closed
antoineco opened this issue Mar 3, 2022 · 16 comments
Closed

Can't authenticate with Elastiscsearch, Kibana server is not ready yet #680

antoineco opened this issue Mar 3, 2022 · 16 comments

Comments

@antoineco
Copy link
Collaborator

antoineco commented Mar 3, 2022 •
edited
Loading

Originally posted by @asgharkhan in #677 (comment)

Hi,
I am using
docker-compose version 1.29.2, build 5becea4c

and docker-compose config output is below

networks:
  elk:
    driver: bridge
services:
  elasticsearch:
    build:
      args:
        ELASTIC_VERSION: 8.0.0
      context: /Library/WebServer/Documents/docker/docker-elk/elasticsearch
    environment:
      ELASTIC_PASSWORD: changeme
      ES_JAVA_OPTS: -Xmx256m -Xms256m
      discovery.type: single-node
    networks:
      elk: null
    ports:
    - published: 9200
      target: 9200
    - published: 9300
      target: 9300
    volumes:
    - /Library/WebServer/Documents/docker/docker-elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,z
    - elasticsearch:/usr/share/elasticsearch/data:z
  kibana:
    build:
      args:
        ELASTIC_VERSION: 8.0.0
      context: /Library/WebServer/Documents/docker/docker-elk/kibana
    depends_on:
      elasticsearch:
        condition: service_started
    environment:
      KIBANA_SYSTEM_PASSWORD: changeme
    networks:
      elk: null
    ports:
    - published: 5601
      target: 5601
    volumes:
    - /Library/WebServer/Documents/docker/docker-elk/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z
  logstash:
    build:
      args:
        ELASTIC_VERSION: 8.0.0
      context: /Library/WebServer/Documents/docker/docker-elk/logstash
    depends_on:
      elasticsearch:
        condition: service_started
    environment:
      LOGSTASH_INTERNAL_PASSWORD: changeme
      LS_JAVA_OPTS: -Xmx256m -Xms256m
    networks:
      elk: null
    ports:
    - published: 5044
      target: 5044
    - protocol: tcp
      published: 5000
      target: 5000
    - protocol: udp
      published: 5000
      target: 5000
    - published: 9600
      target: 9600
    volumes:
    - /Library/WebServer/Documents/docker/docker-elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z
    - /Library/WebServer/Documents/docker/docker-elk/logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
  setup:
    build:
      args:
        ELASTIC_VERSION: 8.0.0
      context: /Library/WebServer/Documents/docker/docker-elk/setup
    environment:
      ELASTIC_PASSWORD: changeme
      KIBANA_SYSTEM_PASSWORD: changeme
      LOGSTASH_INTERNAL_PASSWORD: changeme
    init: true
    networks:
      elk: null
    volumes:
    - setup:/state:Z
version: '3.7'
volumes:
  elasticsearch: {}
  setup: {}
➜  docker-elk git:(main) docker-compose --version
docker-compose version 1.29.2, build 5becea4c
➜  docker-elk git:(main) docker-compose --version
docker-compose version 1.29.2, build 5becea4c
➜  docker-elk git:(main) docker-compose config
networks:
  elk:
    driver: bridge
services:
  elasticsearch:
    build:
      args:
        ELASTIC_VERSION: 8.0.0
      context: /Library/WebServer/Documents/docker/docker-elk/elasticsearch
    environment:
      ELASTIC_PASSWORD: changeme
      ES_JAVA_OPTS: -Xmx256m -Xms256m
      discovery.type: single-node
    networks:
      elk: null
    ports:
    - published: 9200
      target: 9200
    - published: 9300
      target: 9300
    volumes:
    - /Library/WebServer/Documents/docker/docker-elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,z
    - elasticsearch:/usr/share/elasticsearch/data:z
  kibana:
    build:
      args:
        ELASTIC_VERSION: 8.0.0
      context: /Library/WebServer/Documents/docker/docker-elk/kibana
    depends_on:
      elasticsearch:
        condition: service_started
    environment:
      KIBANA_SYSTEM_PASSWORD: changeme
    networks:
      elk: null
    ports:
    - published: 5601
      target: 5601
    volumes:
    - /Library/WebServer/Documents/docker/docker-elk/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z
  logstash:
    build:
      args:
        ELASTIC_VERSION: 8.0.0
      context: /Library/WebServer/Documents/docker/docker-elk/logstash
    depends_on:
      elasticsearch:
        condition: service_started
    environment:
      LOGSTASH_INTERNAL_PASSWORD: changeme
      LS_JAVA_OPTS: -Xmx256m -Xms256m
    networks:
      elk: null
    ports:
    - published: 5044
      target: 5044
    - protocol: tcp
      published: 5000
      target: 5000
    - protocol: udp
      published: 5000
      target: 5000
    - published: 9600
      target: 9600
    volumes:
    - /Library/WebServer/Documents/docker/docker-elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z
    - /Library/WebServer/Documents/docker/docker-elk/logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
  setup:
    build:
      args:
        ELASTIC_VERSION: 8.0.0
      context: /Library/WebServer/Documents/docker/docker-elk/setup
    environment:
      ELASTIC_PASSWORD: changeme
      KIBANA_SYSTEM_PASSWORD: changeme
      LOGSTASH_INTERNAL_PASSWORD: changeme
    init: true
    networks:
      elk: null
    volumes:
    - setup:/state:Z
version: '3.7'
volumes:
  elasticsearch: {}
  setup: {}`

I am trying to login on http://localhost:9200/ with below curl command

➜  docker-elk git:(main) curl -D- http://localhost:9200 -u "elastic:changeme"`
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="security" charset="UTF-8"
WWW-Authenticate: ApiKey
content-type: application/json;charset=utf-8
content-length: 409

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}},"status":401}%

If I access http://localhost:5601/ in the browser then I can see

Kibana server is not ready yet.
@antoineco antoineco mentioned this issue Mar 3, 2022
Closed
@antoineco
Copy link
Collaborator Author

@asgharkhan the config looks good to me.

Please share the status of the stack and its logs:

  • docker-compose ps
  • docker-compose logs setup
  • docker-compose logs elasticsearch

@antoineco
Copy link
Collaborator Author

Closing due to inactivity.

@zakrian07
Copy link

I'm having the same issue
can you help me
I can share my log please see all screens
even I can not change the password it says cluster is red
sometimes elastic password works fine and when I set a wrong password and after that, it never authenticates even with the correct password
server4
server3
server2
server1
Just stuck in this behavior
I'm on ubuntu 20.02

@antoineco
Copy link
Collaborator Author

@zakrian07 the screenshot you shared doesn't show enough Elasticsearch logs for me to see why the cluster is RED.

Please run docker-compose logs elasticsearch (give Elasticsearch about a minute to start first), copy the entire text, press Ctrl-Shift-C to copy to your clipboard, and paste it here.
It's much easier to read through text than screenshots.

@zakrian07
Copy link

zakrian07 commented Jun 6, 2022 via email •
edited by antoineco
Loading

@zakrian07
Copy link

again-changeme-wrong
changem-with-wrong-pwd
changem1
suspected behavior

@zakrian07
Copy link

One more thing
somedays ago I change my password and install manually filebeat/metricbeat/heartbeat
everything was working data was coming in kibana and once I stop the project and start again it never works. even delete all volume /image/containers re setup all the project but I'm unable to run the project
same thing happened with me on my cloud Linux/ubuntu server as well

@zakrian07
Copy link

zakrian07 commented Jun 6, 2022 •
edited by antoineco
Loading

here is my docker compose

networks:
  elk:
    driver: bridge
services:
  elasticsearch:
    build:
      args:
        ELASTIC_VERSION: 8.2.2
      context: /home/singhera/projects/elk/docker-elk/elasticsearch
    environment:
      ELASTIC_PASSWORD: changeme
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      discovery.type: single-node
    networks:
      elk: null
    ports:
    - published: 9200
      target: 9200
    - published: 9300
      target: 9300
    volumes:
    - /home/singhera/projects/elk/docker-elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,z
    - elasticsearch:/usr/share/elasticsearch/data:z
  kibana:
    build:
      args:
        ELASTIC_VERSION: 8.2.2
      context: /home/singhera/projects/elk/docker-elk/kibana
    depends_on:
      elasticsearch:
        condition: service_started
    environment:
      KIBANA_SYSTEM_PASSWORD: changeme
    networks:
      elk: null
    ports:
    - published: 5601
      target: 5601
    volumes:
    - /home/singhera/projects/elk/docker-elk/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z
  logstash:
    build:
      args:
        ELASTIC_VERSION: 8.2.2
      context: /home/singhera/projects/elk/docker-elk/logstash
    depends_on:
      elasticsearch:
        condition: service_started
    environment:
      LOGSTASH_INTERNAL_PASSWORD: changeme
      LS_JAVA_OPTS: -Xms256m -Xmx256m
    networks:
      elk: null
    ports:
    - published: 5044
      target: 5044
    - protocol: tcp
      published: 5000
      target: 5000
    - protocol: udp
      published: 5000
      target: 5000
    - published: 9600
      target: 9600
    volumes:
    - /home/singhera/projects/elk/docker-elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z
    - /home/singhera/projects/elk/docker-elk/logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
  setup:
    build:
      args:
        ELASTIC_VERSION: 8.2.2
      context: /home/singhera/projects/elk/docker-elk/setup
    environment:
      ELASTIC_PASSWORD: changeme
      KIBANA_SYSTEM_PASSWORD: changeme
      LOGSTASH_INTERNAL_PASSWORD: changeme
    init: true
    networks:
      elk: null
    volumes:
    - setup:/state:Z
version: '3.7'
volumes:
  elasticsearch: {}
  setup: {}

@zakrian07
Copy link

zakrian07 commented Jun 6, 2022 •
edited by antoineco
Loading 

singhera@singhera-Latitude-E7470:~/projects/elk/docker-elk$ docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user elastic
WARNING: Owner of file [/usr/share/elasticsearch/config/users] used to be [root], but now is [elasticsearch]
WARNING: Owner of file [/usr/share/elasticsearch/config/users_roles] used to be [root], but now is [elasticsearch]
Failed to determine the health of the cluster. Cluster health is currently RED.
This means that some cluster data is unavailable and your cluster is not fully functional.
The cluster logs (https://www.elastic.co/guide/en/elasticsearch/reference/8.2/logging.html) might contain information/indications for the underlying cause
It is recommended that you resolve the issues with your cluster before continuing
It is very likely that the command will fail when run against an unhealthy cluster.

If you still want to attempt to execute this command against an unhealthy cluster, you can pass the `-f` parameter.

ERROR: Failed to determine the health of the cluster. Cluster health is currently RED.

@zakrian07
Copy link

zakrian07 commented Jun 6, 2022 •
edited by antoineco
Loading 

singhera@singhera-Latitude-E7470:~/projects/elk/docker-elk$ docker-compose up setup
Starting docker-elk_setup_1 ... done
Attaching to docker-elk_setup_1
setup_1          | -------- Mon Jun  6 16:12:15 UTC 2022 --------
setup_1          | [+] Waiting for availability of Elasticsearch
setup_1          |    ⠿ Elasticsearch is running
setup_1          | [+] Role 'logstash_writer'
setup_1          |    ⠿ Creating/updating
docker-elk_setup_1 exited with code 28

@antoineco
Copy link
Collaborator Author

It seems like your data is corrupted.

at
least one primary shard for the index [.security-7] is
unavailable

If you don't have any important data stored in Elasticsearch, the most effective solution is to reset the stack:

docker-compose down -v

@zakrian07
Copy link

zakrian07 commented Jun 6, 2022 via email •
edited by antoineco
Loading

@zakrian07
Copy link

image
even password change working on 7.0 version
is this a stable version?

@antoineco
Copy link
Collaborator Author

Did you rebuild your images after switching branches or versions?
The README states that you need to run docker-compose build every time you do so.

Also, pay close attention to the READMEs on each branch. The way you reset passwords in v7 isn't the same as in v8.

Reset the stack, set your passwords in the env file, run the stack fresh, and you should be good to go.

@zakrian07
Copy link

well
I'm able to fully deploy with version 7.0
https://testinglog.alaskaerp.com/
I have updated the env as well with the above passwords
where I need to update the password
for remote_monitoring_user / beats_systems/ APM systems ?

@antoineco
Copy link
Collaborator Author

Again, you must run docker-compose build after switching branches.
Clearly, your local images are still v7 images, otherwise you wouldn't be able to reset passwords like you did on v7.

Stop the stack, check out the main branch, run docker-compose build, and then your stack will be ready for v8.

This issue is closed and what I've read so far doesn't match the original issue description, so I'm going to lock it for now and encourage you to open a new issue with a clear problem statement in case your issue persists after following the instructions I shared here.

Repository owner locked as off-topic and limited conversation to collaborators Jun 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@antoineco @zakrian07