Application not running on M1 pro

[dj13may91]

Trying to run out of the box service but not able to get it running on M1 pro.
Using branch : master

Mordred
Traceback (most recent call last): File "/usr/local/bin/sirmordred", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.8/site-packages/sirmordred/bin/sirmordred.py", line 66, in main logger = setup_logs(logs_dir, debug_mode, short_name) File "/usr/local/lib/python3.8/site-packages/sirmordred/bin/sirmordred.py", line 98, in setup_logs fh = logging.FileHandler(fh_filepath) File "/usr/local/lib/python3.8/logging/__init__.py", line 1147, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/local/lib/python3.8/logging/__init__.py", line 1176, in _open return open(self.baseFilename, self.mode, encoding=self.encoding) PermissionError: [Errno 13] Permission denied: '/home/grimoire/logs/all.log'

kibiter
runtime: failed to create new OS thread (have 2 already; errno=22) fatal error: runtime.newosproc runtime stack: runtime.throw(0x56be06) /usr/src/go/src/runtime/panic.go:491 +0xad runtime.newosproc(0xc208020000, 0xc208030000) /usr/src/go/src/runtime/os_linux.c:170 +0x10a newm(0x430700, 0x0) /usr/src/go/src/runtime/proc.c:1157 +0xc5 runtime.newsysmon() /usr/src/go/src/runtime/proc.c:169 +0x33 runtime.onM(0x57cb30) /usr/src/go/src/runtime/asm_amd64.s:257 +0x68 runtime.mstart() /usr/src/go/src/runtime/proc.c:818

[dj13may91]

Fixed above for now,:
fixes:

Morded --> has /tmp/ mounted which had permission issue

kibiter --> used kibana and ES for now

Now facing below error:
Traceback (most recent call last): File "/usr/local/bin/sirmordred", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.8/site-packages/sirmordred/bin/sirmordred.py", line 64, in main debug_mode = config_dict['general']['debug'] KeyError: 'debug'

[dj13may91]

@zhquan please help with above

[zhquan]

Hi @dj13may91

Your setup.cfg seems incorrect or your volume is not mounted correctly. Could you share your docker-compose.yml and the path of your setup.cfg and docker-compose.yml?

[dj13may91]

I am using the default that is present in project. Expecting this setup to pull data of grimoire lab project.
This is my setup.cfg, no changes : https://github.com/chaoss/grimoirelab/blob/master/default-grimoirelab-settings/setup-opensearch.cfg
Also, using a modified docker compose with default elastic versions, but still not able to get the project up and running.
Docker compose :

version: "3.8"

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local
  sortinghat-static:
  mariadb_data:

networks:
  default:
    name: elastic
    external: false

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: kibana\n"\
          "    dns:\n"\
          "      - kibana\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: [ "CMD-SHELL", "[ -f config/certs/es01/es01.crt ]" ]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      #      - discovery.type=single-node
      - cluster.initial_master_nodes=es01
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      #      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    #    labels:
    #      co.elastic.logs/module: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS="https://es01:9200"
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    #      - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
    #      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
    #      - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
    mem_limit: ${KB_MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  mariadb:
    image: mariadb:10.6
    expose:
      - "3306"
    environment:
      - MYSQL_ROOT_PASSWORD=
      - MYSQL_ALLOW_EMPTY_PASSWORD=yes
    healthcheck:
      test: [ "CMD", "/usr/local/bin/healthcheck.sh", "--su=root", "--connect", "--innodb_initialized" ]
      retries: 5
    volumes:
      - mariadb_data:/var/lib/mysql

  redis:
    image: amd64/redis
    expose:
      - "6379"
    healthcheck:
      test: [ "CMD", "redis-cli", "--raw", "incr", "ping" ]
      retries: 5

  sortinghat:
    restart: on-failure:3
    image: grimoirelab/sortinghat
    environment:
      - SORTINGHAT_SECRET_KEY=secret
      - SORTINGHAT_DB_HOST=mariadb
      - SORTINGHAT_DB_PORT=3306
      - SORTINGHAT_DB_DATABASE=sortinghat_db
      - SORTINGHAT_DB_USER=root
      - SORTINGHAT_DB_PASSWORD=
      - SORTINGHAT_REDIS_HOST=redis
      - SORTINGHAT_REDIS_PASSWORD=
      - SORTINGHAT_SUPERUSER_USERNAME=root
      - SORTINGHAT_SUPERUSER_PASSWORD=root
      - SORTINGHAT_ALLOWED_HOST=sortinghat,nginx,localhost,127.0.0.1,[::1]
      - SORTINGHAT_CORS_ALLOWED_ORIGINS=http://localhost:8000,http://127.0.0.1:8000
    expose:
      - "9314"
    volumes:
      - sortinghat-static:/opt/venv/lib/python3.9/site-packages/sortinghat/static/
    depends_on:
      mariadb:
        condition: service_healthy
      redis:
        condition: service_healthy

  sortinghat_worker:
    image: grimoirelab/sortinghat-worker
    environment:
      - SORTINGHAT_SECRET_KEY=secret
      - SORTINGHAT_DB_HOST=mariadb
      - SORTINGHAT_DB_PORT=3306
      - SORTINGHAT_DB_DATABASE=sortinghat_db
      - SORTINGHAT_DB_USER=root
      - SORTINGHAT_DB_PASSWORD=
      - SORTINGHAT_REDIS_HOST=redis
      - SORTINGHAT_REDIS_PASSWORD=
    depends_on:
      mariadb:
        condition: service_healthy
      redis:
        condition: service_healthy

  mordred:
    restart: on-failure:5
    image: grimoirelab/grimoirelab:latest
    volumes:
      - ../default-grimoirelab-settings/setup-opensearch.cfg:/home/grimoire/conf/setup.cfg
      - ../default-grimoirelab-settings/projects.json:/home/grimoire/conf/projects.json
      #      - /tmp/:/home/grimoire/logs
      - ../docker-compose/logs:/home/grimoire/logs
#    depends_on:
#      nginx:
#        condition: service_healthy
    mem_limit: 4g

  nginx:
    restart: on-failure:3
    image: nginx:latest
    volumes:
      - ../default-grimoirelab-settings/nginx.conf.template:/etc/nginx/templates/default.conf.template
      - ../default-grimoirelab-settings/uwsgi_params:/etc/nginx/uwsgi_params
      - sortinghat-static:/sortinghat:ro
    ports:
      - 8000:8000
    environment:
      - KIBANA_HOST=http://localhost:5601/
    depends_on:
      - sortinghat
    healthcheck:
      test: "curl -s --head http://localhost:8000/identities/api/ | grep Set-Cookie || exit 1"
      retries: 5

[zhquan]

I don't have any issues running on Linux using the default configurations.

$ docker-compose -f docker-compose-opensearch.yml up -d
Starting docker-compose_redis_1                 ... done
Starting docker-compose_opensearch-node1_1      ... done
Starting docker-compose_mariadb_1               ... done
Starting docker-compose_opensearch-dashboards_1 ... done
Starting docker-compose_sortinghat_1            ... done
Starting docker-compose_sortinghat_worker_1     ... done
Starting docker-compose_nginx_1                 ... done
Starting docker-compose_mordred_1               ... done

• Here are my indexes

GET _cat/indices?s=i
yellow open git-aoc_demo_enriched        O2Ku2SnXSe6bikKWdi6Lsg 1 1 3921  0   5.5mb   5.5mb
yellow open git-onion_demo_enriched      4u9miSmfQUeGgnRcYUXIEw 1 1  536  0 123.4kb 123.4kb
yellow open git_demo_enriched            DqgGv0uoQfCP-zWjGLnThg 1 1  754  0   9.6mb   9.6mb
yellow open git_demo_raw                 N3Y7Q6QrRrOUhttRCO8Raw 1 1  754  3 956.7kb 956.7kb

But I can reproduce the error if I change the permission of the file to 600 (docker user cannot read the setup.cfg file)

$ docker logs --tail=10  docker-compose_mordred_1
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sirmordred/bin/sirmordred.py", line 64, in main
    debug_mode = config_dict['general']['debug']
KeyError: 'debug'
Traceback (most recent call last):
  File "/usr/local/bin/sirmordred", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sirmordred/bin/sirmordred.py", line 64, in main
    debug_mode = config_dict['general']['debug']
KeyError: 'debug'

Could you try again but change the permission to 777 for ../default-grimoirelab-settings/setup-opensearch.cfg and ../default-grimoirelab-settings/projects.json?