How can I visualise Gitlab data?

[pilgrimage80]

I want to visualise the data of the users in Gitlab, such as how many issues they have opened and how many issues they have closed. According to the document below, I fill in the project.json and setup.cfg sections, but it does not work.

https://chaoss.github.io/grimoirelab-tutorial/docs/data-sources/configurations/#gitlab

When I look at the last line of setup.cfg, I think I need to update the following data according to gitlab. How do I do this?

What could I be missing?

#[enrich_demography:git]

#[enrich_areas_of_code:git]
#in_index = git_demo_raw
#out_index = git-aoc_demo_enriched

#[enrich_onion:git]
#in_index = git
#out_index = git-onion_demo_enriched
#contribs_field = hash

[sduenas]

Can you complete the issue with information about how you are running grimoirelab, the contents of your files, etc?

[pilgrimage80]

I installed everything as docker-compose according to the link below.

https://chaoss.github.io/grimoirelab-tutorial/docs/getting-started/setup/

My Project.json :
{
"XX": {
"gitlab:issue": [
"https://gitlabtest/t/xx.git"

    ]
}

}

setup.cfg
[general]
short_name = GrimoireLab
update = true
min_update_delay = 60
debug = false
logs_dir = /home/grimoire/logs
bulk_size = 100
scroll_size = 100
aliases_file = /home/grimoire/aliases.json

[projects]
projects_file = /home/grimoire/conf/projects.json

[es_collection]
url = http://elasticsearch:9200

[es_enrichment]
url = http://elasticsearch:9200
autorefresh = true

[sortinghat]
host = nginx
user = root
password = root
port = 8000
path = /identities/api/
ssl = false
database = sortinghat_db
autoprofile = [github, git]
matching = [email,name,username]
sleep_for = 100
unaffiliated_group = Unknown
affiliate = true

[panels]
kibiter_time_from = now-90d
kibiter_default_index = git
kibiter_url = http://kibiter:5601
kibiter_version = 6.8.6-3
gitlab-issues = true

[phases]
collection = true
identities = true
enrichment = true
panels = true

[gitlab:issues]
api-token = kSMNn3hKvxWGXaC-bruQ
raw_index = gitlab_issues_demo_raw
enriched_index = gitlab_issues_demo_enriched
no-archive = true
enterprise-url = https://gitlabtest
sleep-for-rate = true

[enrich_onion:gitlab-issue] # (optional)
in_index = gitlab-issues_enriched
out_index = gitlab-issues-onion_enriched
data_source = gitlab-issues

When I open localhost:5601 it says no result found and no data is displayed.

There are some patterns in the discover section from the menu, which one of them should be selected

kibana_1
grimoirelab-sigils
affliations

What am I doing wrong?

[zhquan]

Hi @pilgrimage80

Could you check if you have indexes and aliases?
Go to Dev Tools and run:

GET _cat/indices
GET _cat/aliases

And which version of each component are you using?

[pilgrimage80]

GET _cat/indices
green open .kibana_1 BhACCJh4QxCfbb3FrHy6WQ 1 0 131 7 279.4kb 279.4kb
yellow open .grimoirelab-sigils h9Y2xjjKQWWiYDuHr07VtQ 5 1 22 0 59.6kb 59.6kb

GET _cat/aliases
bana .kibana_1 - - -

mariadb 10.6
grimoirelab/sortinghat-worker latest
grimoirelab/sortinghat latest
grimoirelab/grimoirelab latest
docker.elastic.co/kibana/kibana 8.12.2
docker.elastic.co/elasticsearch/elasticsearch 8.12.2
nginx latest
redis latest
bitergia/kibiter community-v6.8.6-3
docker.elastic.co/elasticsearch/elasticsearch-oss 6.8.6

[zhquan]

You have 2 elasticsearch and 2 kibana I guess grimoirelab is connected to ES 6.8.6.

You don't have any index created by grimoirelab. Could you check the grimoirelab logs if there is any error?

[pilgrimage80]

How can I check the logs on grimoirelab?

[pilgrimage80]

Where are the logs?

[zhquan]

If you are using https://github.com/chaoss/grimoirelab/blob/master/docker-compose/docker-compose.yml#L50 you can find it at /tmp/all.log or docker logs <mordred_container_name>

[pilgrimage80]

Hello, I tried to proceed according to the document below without any errors, but I am having a problem as follows.

docker-compose up -d

https://bitergia.com/blog/opensource/grimoirelab-to-measure-organizations-private-development/

After running the command, docker containers are up but after a while the following container is exited.
945f836d644e grimoirelab/grimoirelab:latest "/bin/sh -c ${DEPLOY…" 19 hours ago Exited (1) 12 minutes ago dockercompose_mordred_1

In addition, I chose the Access token of a project that I want to get internal gitlab data in the form of api and read api, and I chose Guest as the Role. Can you confirm the correctness?

As a result, after logging in to http://localhost:5601, I see things like api, gitlab issue as index patterns. Even if I select any of these as default, the dashboard does not give an error but says no result found, as if it does not reach the data within the company.

Where could I have made a mistake? I need your support on this issue.

[pilgrimage80]

My indexes and aliases?

GET _cat/indices
yellow open .grimoirelab-sigils _xV71ns4SaSCH5wC2ISLCQ 5 1 22 0 68.6kb 68.6kb
green open kibana_sample_data_ecommerce d4rcO8D4Sg-c-lF_jmj4_w 1 0 4675 0 4.8mb 4.8mb
green open .kibana_1 wegrmJ1jS7OAiFM2QtnVqA 1 0 149 5 281.8kb 281.8kb
yellow open api D0PN4l8BS4aKHPGrFetxLA 5 1 1 0 5.4kb 5.4kb

GET _cat/aliases

.kibana .kibana_1 - - -

[pilgrimage80]

When I examine grimoirelab/grimoirelab:latest docker logs, I see the error I shared in the link. How can I solve this problem?

in docker it goes from up to exit.

945f836d644e grimoirelab/grimoirelab:latest "/bin/sh -c ${DEPLOY…" 20 hours ago Exited (1) About an hour ago dockercompose_mordred_1

https://pastebin.com/tpYmxBFz

[jjmerchante]

Thank you @pilgrimage80 for providing the logs.

I think the configuration files you provided don't correspond to the errors. There are two errors in the configuration files.

• This first error indicates that your setup.cfg file is wrong:

  Error while consuming configuration: Wrong type for section param: gitlab:issue studies <class 'str'> should be <class 'list'>
  

  Your setup.cfg in the [gitlab:issue] section has something like

  studies = [enrich_onion:gitlab-issue] # (optional)
  

  or

  studies = enrich_onion:gitlab-issue
  

  In this case you should remove # (optional) or create a list:

  studies = [enrich_onion:gitlab-issue] 
  

• The second error is related to your projects.json file:

  json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 8 column 13 (char 251)
  

  It looks like you are using single quotes (') in your projects.json and must be substituted with double quotes (")

After fixing those errors try to restart the containers:

docker compose down
docker compose up -d

[pilgrimage80]

Now when I check the docker logs mordered I get the following error.

2024-03-22 09:34:08,651 - GrimoireLab - urllib3.connectionpool - WARNING - Retrying (Retry(total=3, connect=3, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc1602f0a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /api/v4/projects/%2F
2024-03-22 09:34:12,679 - GrimoireLab - urllib3.connectionpool - WARNING - Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc15fe3eb0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /api/v4/projects/%2F
2024-03-22 09:34:20,691 - GrimoireLab - urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc15fe3460>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /api/v4/projects/%2F
2024-03-22 09:34:36,717 - GrimoireLab - urllib3.connectionpool - WARNING - Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc15fe3e80>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /api/v4/projects/%2F
2024-03-22 09:34:36,748 - GrimoireLab - grimoire_elk.elk - ERROR - Error feeding raw from gitlab (https://gitlabtest//): HTTPSConnectionPool(host='gitlabtest', port=443): Max retries exceeded with url: /api/v4/projects/%2F (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc168e63d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 72, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 363, in connect
self.sock = conn = self._new_conn()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7efc168e63d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 827, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 827, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 827, in urlopen
return self.urlopen(
[Previous line repeated 2 more times]
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 799, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='gitlabtest', port=443): Max retries exceeded with url: /api/v4/projects/%2F (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc168e63d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/grimoire_elk/elk.py", line 205, in feed_backend
ocean_backend.feed(**params)
File "/usr/local/lib/python3.8/site-packages/grimoire_elk/raw/elastic.py", line 234, in feed
self.feed_items(items)
File "/usr/local/lib/python3.8/site-packages/grimoire_elk/raw/elastic.py", line 250, in feed_items
for item in items:
File "/usr/local/lib/python3.8/site-packages/perceval/backend.py", line 314, in fetch
self.client = self._init_client()
File "/usr/local/lib/python3.8/site-packages/perceval/backends/core/gitlab.py", line 249, in _init_client
return GitLabClient(self.owner, self.repository, self.api_token,
File "/usr/local/lib/python3.8/site-packages/perceval/backends/core/gitlab.py", line 517, in init
self._init_rate_limit()
File "/usr/local/lib/python3.8/site-packages/perceval/backends/core/gitlab.py", line 736, in _init_rate_limit
response = super().fetch(url)
File "/usr/local/lib/python3.8/site-packages/perceval/client.py", line 143, in fetch
response = self._fetch_from_remote(url, payload, headers, method, stream, auth)
File "/usr/local/lib/python3.8/site-packages/perceval/client.py", line 174, in _fetch_from_remote
response = self.session.get(url, params=payload, headers=headers, stream=stream,
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gitlabtest', port=443): Max retries exceeded with url: /api/v4/projects/%2F (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc168e63d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
2024-03-22 09:34:36,798 - GrimoireLab - grimoire_elk.elk - INFO - [gitlab] Done collection for https://gitlabtest//
2024-03-22 09:34:36,799 - GrimoireLab - sirmordred.task_collection - INFO - [gitlab:issues] collection finished for /
2024-03-22 09:34:36,800 - GrimoireLab - sirmordred.task_manager - ERROR - [gitlab:issues] Exception in Task Manager list index out of range
Traceback (most recent call last):

[pilgrimage80]

It says no result found under gitlab issues on the dashboard.

My indices and aliases

GET _cat/indices
yellow open .grimoirelab-sigils _xV71ns4SaSCH5wC2ISLCQ 5 1 22 0 68.6kb 68.6kb
yellow open gitlab_issues_demo_enriched iTpgphdFTwu8oATTf7aORA 5 1 0 0 1.2kb 1.2kb
yellow open gitlab_merges_demo_raw w1PRfqinT7qk_oEBYs4BNA 5 1 0 0 1.2kb 1.2kb
green open kibana_sample_data_ecommerce d4rcO8D4Sg-c-lF_jmj4_w 1 0 4675 0 4.8mb 4.8mb
green open .kibana_1 wegrmJ1jS7OAiFM2QtnVqA 1 0 149 5 281.8kb 281.8kb
green open .kibana_2 1ex7EVW5R0yjQ1A081IF2g 1 0 149 3 244.1kb 244.1kb
yellow open gitlab_issues_demo_raw z2tcWKefS6Wp6v6zJpabRQ 5 1 0 0 1.2kb 1.2kb
yellow open gitlab_merges_demo_enriched CYZsCUoIQAW33DtlExBj-A 5 1 0 0 1.2kb 1.2kb
green open .kibana_4 oveHB8eXTeC2kxoJsXXVWQ 1 0 149 3 243.9kb 243.9kb
green open .kibana_3 Uz-xo88kRAGStxGM9Q1mwQ 1 0 149 3 249.3kb 249.3kb
yellow open api D0PN4l8BS4aKHPGrFetxLA 5 1 1 0 5.4kb 5.4kb

GET _cat/aliases

gitlab-raw gitlab_issues_demo_raw - - -
gitlab_merge_requests-raw gitlab_merges_demo_raw - - -
.kibana .kibana_4 - - -

[jjmerchante]

If your projects.json contains https://gitlabtest/t/xx.git, and your setup.cfg has enterprise-url = https://gitlabtest/, it won't collect anything because that domain doesn't exist.

Consider using a repository URL from which you want to collect data, such as https://gitlab.com/gitlab-org/grafana-dashboards in your projects.json. If it's the official GitLab instance, please remove the enterprise-url line from the setup.cfg, if it is an enterprise instance, replace gitlabtest with the actual domain.