Default Dashboard not showing any data - possible elasticsearch startup problem

[nmohammad]

I have a new Ubuntu VM and I am trying to follow the get started instructions https://chaoss.github.io/grimoirelab-tutorial/docs/getting-started/setup/.
I have set the virtual memory ,and verified it. My machine as 8 CPUS, 16GB of memory.
I am using the default docker-compose method. I see the panels but no data.
When I browse port 9200 I get the following json response:
{
"name": "9PS5HiZ",
"cluster_name": "docker-cluster",
"cluster_uuid": "SOa1jgmoRea1bM0SGsQJ9g",
"version": {
"number": "6.8.6",
"build_flavor": "oss",
"build_type": "docker",
"build_hash": "3d9f765",
"build_date": "2019-12-13T17:11:52.013738Z",
"build_snapshot": false,
"lucene_version": "7.7.2",
"minimum_wire_compatibility_version": "5.6.0",
"minimum_index_compatibility_version": "5.0.0"
},
"tagline": "You Know, for Search"
}

terminalsession.txt

I have attached a copy of my terminal session in which you will see a clean git clone and command sudo docker-compose up , in case that helps.

Any ideas or suggestions to help get the default dashboard up would be appreciated.
I have tried multiple things like adding github and gitlab keys, but I have been informed that should not be necessary to see something. At the moment I only see panels with "no results found"

[nmohammad]

In case it helps this is the output from log files - taken as text copy from the terminal:
I have also attached two screen shots from the devtools .

nmohammad@github-dashboard-dev:/home/grimoirelab/docker-compose$ docker-compose logs
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 395, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.10/http/client.py", line 1283, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 395, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.10/http/client.py", line 1283, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 557, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/bin/docker-compose", line 33, in
sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 81, in main
command_func()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 200, in perform_command
project = project_from_options('.', options)
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 60, in project_from_options
return get_project(
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 152, in get_project
client = get_client(
File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 41, in get_client
client = docker_client(
File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 170, in docker_client
client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 197, in init
self._version = self._retrieve_server_version()
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
nmohammad@github-dashboard-dev:/home/grimoirelab/docker-compose$

Screenshots from Devtools

[jjmerchante]

Hi @nmohammad,

I can't reproduce the error locally, so it's a bit difficult for me to help. However, based on the logs you shared, I can see two errors:

1. First error:

   docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
   

   From what I found online, others have resolved this issue by adding the current user to the Docker group and restarting the machine. You can follow these instructions to do so: Manage Docker as a non-root user.

   This approach will allow you to avoid running Docker commands with sudo.

2. Second error:

   fatal: unable to access 'https://github.com/chaoss/grimoirelab.git/': Operation timed out after 300001 milliseconds with 0 out of 0 bytes received
   

   You can clone the GrimoireLab repository outside of Docker but not from within Docker. This might indicate an issue with your Docker network configuration. Maybe the previous solution fixes this error.

[nmohammad]

Thanks for the reply - really appreciate it! I shall try your suggestions and let you know.
Thanks!!!

[nmohammad]

I think adding to the docker group has resolved the permissions issue. Still no data.
I also created a group called git, and added my user.
I now don't see those two errors in the logs.
I haven't added any of my github or gitlab API keys to the setup.cfg, I read that I should see some data without touching the cfg file.
I have attached the latest log files - incase anyone sees anything which can be addressed.
I suspect there is something else wrong with my setup - but I am a linux newbie.
newlogs.txt

newlogs.txt

[jjmerchante]

It is not necessary to add the GitHub and GitLab keys for the default configuration.

It takes some time to process the data. I can see when the collection starts:

mordred_1            | 2024-09-06 17:37:24,874 - GrimoireLab - sirmordred.task_collection - INFO - [git] collection starts for https://github.com/chaoss/grimoirelab.git

But I can't see when the collection phase finishes, it takes some minutes to clone the repository.

Could you attach the full logs? Please, wait for up to five minutes to see if any data appears.

[nmohammad]

Hi Jose,
Thanks for looking at the logs. From what I can see - it starts but after 0 entries it finishes :(
I have attached another log file.
I deleted my clone, clean clone, waited about 15mins before accessing webserver via my windows machine - waited about 10mins, and then dumped the log.
I see there were errors accessing the github repo https://github.com/chaoss/grimoirelab.git which is in projects.json, so I even have tried to edit that to https://github.com/chaoss/grimoirelab incase the system didn't like the redirects. Still no joy.

I suspect that there is a problem with the network or firewall settings of the VM I am using. It was setup by my corp IT department.
logfile-sept9.txt

Incase there is a firewall on my vm I looked at the iptables and here is the dump with byte usage after launching docker-compose. There is a strange setting for port 9200 but that rule wasn't invoked - no bytes.

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
81422 11M DOCKER-USER all -- any any anywhere anywhere
81422 11M DOCKER-ISOLATION-STAGE-1 all -- any any anywhere anywhere
79051 11M ACCEPT all -- any br-912e7165fc7b anywhere anywhere ctstate RELATED,ESTABLISHED
1637 97644 DOCKER all -- any br-912e7165fc7b anywhere anywhere
734 149K ACCEPT all -- br-912e7165fc7b !br-912e7165fc7b anywhere anywhere
1565 93900 ACCEPT all -- br-912e7165fc7b br-912e7165fc7b anywhere anywhere
0 0 ACCEPT all -- any br-0bbdefb3e9f8 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any br-0bbdefb3e9f8 anywhere anywhere
0 0 ACCEPT all -- br-0bbdefb3e9f8 !br-0bbdefb3e9f8 anywhere anywhere
0 0 ACCEPT all -- br-0bbdefb3e9f8 br-0bbdefb3e9f8 anywhere anywhere
0 0 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any docker0 anywhere anywhere
0 0 ACCEPT all -- docker0 !docker0 anywhere anywhere
0 0 ACCEPT all -- docker0 docker0 anywhere anywhere

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain DOCKER (3 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- !br-912e7165fc7b br-912e7165fc7b anywhere v300-mdf1-ba.scl-ca.domain.com tcp dpt:9200
0 0 ACCEPT tcp -- !br-912e7165fc7b br-912e7165fc7b anywhere 172.18.0.5 tcp dpt:5601
72 3744 ACCEPT tcp -- !br-912e7165fc7b br-912e7165fc7b anywhere 172.18.0.8 tcp dpt:8000

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
734 149K DOCKER-ISOLATION-STAGE-2 all -- br-912e7165fc7b !br-912e7165fc7b anywhere anywhere
0 0 DOCKER-ISOLATION-STAGE-2 all -- br-0bbdefb3e9f8 !br-0bbdefb3e9f8 anywhere anywhere
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 anywhere anywhere
81422 11M RETURN all -- any any anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (3 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- any br-912e7165fc7b anywhere anywhere
0 0 DROP all -- any br-0bbdefb3e9f8 anywhere anywhere
0 0 DROP all -- any docker0 anywhere anywhere
734 149K RETURN all -- any any anywhere anywhere

Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
81422 11M RETURN all -- any any anywhere anywhere

[nmohammad]

I incase this helps I was testing if I could access port 9200 from the terminal window so I did some curl tests here is cut and paste of two tests, let me know if there are other curl tests or devtool tests which could help.

nmohammad@github-dashboard-dev:/home/grimoirelab/docker-compose$ curl http://localhost:9200
{
"name" : "ssvzQx2",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "GVuCGc18S22pbK1aaPu5Pg",
"version" : {
"number" : "6.8.6",
"build_flavor" : "oss",
"build_type" : "docker",
"build_hash" : "3d9f765",
"build_date" : "2019-12-13T17:11:52.013738Z",
"build_snapshot" : false,
"lucene_version" : "7.7.2",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
nmohammad@github-dashboard-dev:/home/grimoirelab/docker-compose$ curl -X GET 'http://localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .grimoirelab-sigils 5ptbFtYIT_SKU5uzoSWL_w 5 1 23 2 47.8kb 47.8kb
yellow open git-aoc_demo_enriched Hsoi7DyPSGKPhKVClMMNHw 5 1 0 0 1.2kb 1.2kb
green open .kibana_1 yGxr6FvYTlWoPEEIcFIgjg 1 0 125 13 227.4kb 227.4kb
yellow open git_demo_enriched ugu26i55RdKxsXLXRTEf9w 5 1 0 0 1.2kb 1.2kb
yellow open git_demo_raw 9Lc52HaFRtm8CB3fbeF49Q 5 1 0 0 1.2kb 1.2kb
nmohammad@github-dashboard-dev:/home/grimoirelab/docker-compose$

I also was able to successfully access 172.18.0.4:9200 - which is called out as an error in the logs

[jjmerchante]

Hi @nmohammad,

I see the same error in the logs as before:

fatal: unable to access 'https://github.com/chaoss/grimoirelab.git/': Operation timed out after 300041 milliseconds with 0 out of 0 bytes received

I’m not very familiar with the firewall settings on the virtual machines. However, I’m wondering if the following line is dropping packets from Docker, as it includes the word "DROP":

Chain FORWARD (policy DROP 0 packets, 0 bytes)

You are able to access the repository from the virtual machine, as you can clone it, but there's no access from the Docker container inside the VM. I haven't encountered this issue before, so I’m unsure how to proceed.

The output from curl http://localhost:9200/ seems correct, and the /_cat/indices shows that GrimoireLab has imported the dashboards, because it has data in the .kibana index. However, there is no data related to the collection, which is likely due to the error I mentioned earlier—the container can’t access the repository.

[zhquan]

Hi @nmohammad

Could you enter into the mordred container docker exec -it <MORDRED_CONTAINER_ID> bash and run these commands?

git clone https://github.com/chaoss/grimoirelab.git
perceval git https://github.com/chaoss/grimoirelab.git

Also, I found this https://stackoverflow.com/questions/20430371/my-docker-container-has-no-internet.

I hope it helps you

[nmohammad]

Thanks folks. I enabled port forwarding by uncommenting a line #net.ipv4.ip_forward=1 in /etc/sysctl.conf and also executed sudo sysctl -w net.ipv4.ip_forward=1
The logs still show the same fatal error.
When I was looking for the container ID to follow zhquan's suggestions I did fine that one of the containers was reporting "unhealthy" - I have copied and pasted the output below:

nmohammad@github-dashboard-dev:/home/grimoirelab/docker-compose$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4be6cc3b9714 grimoirelab/grimoirelab:latest "/bin/sh -c ${DEPLOY…" 15 minutes ago Up 15 minutes (unhealthy) docker-compose_mordred_1
d33649789b0b nginx:latest "/docker-entrypoint.…" 16 minutes ago Up 16 minutes (healthy) 80/tcp, 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp docker-compose_nginx_1
3ad55cb5fdba grimoirelab/sortinghat "server-entrypoint.sh" 16 minutes ago Up 16 minutes 9314/tcp docker-compose_sortinghat_1
3db63f6a229e grimoirelab/sortinghat-worker "worker-entrypoint.sh" 16 minutes ago Up 16 minutes 9314/tcp docker-compose_sortinghat_worker_1
be94f45252bf bitergia/kibiter:community-v6.8.6-3 "/docker_entrypoint.…" 16 minutes ago Up 16 minutes 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp docker-compose_kibiter_1
925aa442e49f docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.6 "/usr/local/bin/dock…" 16 minutes ago Up 16 minutes 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp docker-compose_elasticsearch_1
cb0c6bb173e3 mariadb:10.6 "docker-entrypoint.s…" 16 minutes ago Up 16 minutes (healthy) 3306/tcp docker-compose_mariadb_1
60d751c3c60c redis:latest "docker-entrypoint.s…" 16 minutes ago Up 16 minutes (healthy) 6379/tcp docker-compose_redis_1

[nmohammad]

I tried the tests suggested above:

nmohammad@github-dashboard-dev:/home/grimoirelab/docker-compose$ docker exec -it 4be6cc3b9714 bash
grimoire@4be6cc3b9714:$ ping google.com
bash: ping: command not found
grimoire@4be6cc3b9714:$ ping https://github.com
bash: ping: command not found
grimoire@4be6cc3b9714:$ git clone https://github.com/chaoss/grimoirelab.git
fatal: destination path 'grimoirelab' already exists and is not an empty directory.
grimoire@4be6cc3b9714:$ perceval git https://github.com/chaoss/grimoirelab.git
[2024-09-10 18:15:36,890] - Sir Perceval is on his quest.

perceval is still running - but no errors or messages for over 5mins. If it changes I will update this post.

[nmohammad]

I did a reboot, and it still didn't improve operations.
I retried connecting to the connector and doing a git, this time creating a tmp directory and starting git from that location. The git operation did create a grimoirelab directory,, but there was not additional content pulled down. So it looks like git was not able to execute inside the container.
Even simple curl commands inside the container via bash, start but do not return - I had to stop using ctrl-Z and then stop the jobs.
perhaps some permission issues.

[nmohammad]

I am currently on Unbuntu 22.04.4 LTS , I can upgrade to 24.04.1 LTS if that would help.

[zhquan]

I found this https://forums.docker.com/t/git-clone-works-from-host-but-not-docker-container/120932 maybe you are using a proxy-server

[nmohammad]

I think we are on to something. I did an experiment, I deleted the docker port redirects in iptables, but then when I launched the dashboard again using docker-compose, the same port forwarding entries appeared again.
I will try once again and "save" the settings.
Is it expected that the container launch will result in iptables changes.

Regarding proxy-server any suggestion on how to check and change?

[nmohammad]

Its seems that from inside the containers http: access works but https: does not, for example curl http://google.com works but curl https://google.com
On the VM both work as expected.
I tried to search for related posts , I found this one: https://serverfault.com/questions/1162126/docker-containers-fail-all-ssl-operations

It seems that I don't have a /etc/docker/daemon.json but creating one with just the entries they provides results in both http and https to fail with errors saying dns could not resolve. So I have deleted that file and rebooted the VM.
In the article there are other suggestions about exposing port 80 and 443, but I am not sure how to do that.
I am still doing some more experiments.