Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Permission issues with prometheus/grafana #9

Open
samlaf opened this issue Feb 11, 2023 · 0 comments
Open

Permission issues with prometheus/grafana #9

samlaf opened this issue Feb 11, 2023 · 0 comments

Comments

@samlaf
Copy link
Contributor

samlaf commented Feb 11, 2023

Getting these issues when running docker compose up in tag 2.2.0 (same issues in 3.* and 4.*):

Attaching to cdn-up-and-running-backend-1, cdn-up-and-running-edge-1, cdn-up-and-running-nginx_base-1, monitoring_grafana, prometheus
prometheus                       | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus                       | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus                       | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus                       | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus                       | level=info ts=2023-02-11T20:48:25.178Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus                       | level=error ts=2023-02-11T20:48:25.179Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus                       | panic: Unable to create mmap-ed active query log
prometheus                       | 
prometheus                       | goroutine 1 [running]:
prometheus                       | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffdd6a4bed5, 0xb, 0x14, 0x2dcaa00, 0xc000641ef0, 0x2dcaa00)
prometheus                       |      /app/promql/query_logger.go:117 +0x4cd
prometheus                       | main.main()
prometheus                       |      /app/cmd/prometheus/main.go:364 +0x5245
monitoring_grafana               | GF_PATHS_DATA='/var/lib/grafana' is not writable.
monitoring_grafana               | You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944018809Z level=info msg="Starting Grafana" version=9.3.6 commit=978237e7cb branch=HEAD compiled=2023-01-26T14:38:58Z
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944304168Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.9443415Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944349525Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944354946Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.94436247Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944369914Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944376087Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944383461Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944410793Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944417186Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944437886Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944444298Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944465589Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.94447149Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944476841Z level=info msg="Path Home" path=/usr/share/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944482582Z level=info msg="Path Data" path=/var/lib/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944488332Z level=info msg="Path Logs" path=/var/log/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944493442Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944518621Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
monitoring_grafana               | logger=settings t=2023-02-11T20:48:25.944524803Z level=info msg="App mode production"
monitoring_grafana               | logger=sqlstore t=2023-02-11T20:48:25.944647669Z level=info msg="Connecting to DB" dbtype=sqlite3
monitoring_grafana               | logger=sqlstore t=2023-02-11T20:48:25.944678529Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r-----
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:25.967090746Z level=info msg="Starting DB migrations"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:25.971707872Z level=info msg="Executing migration" id="Update is_service_account column to nullable"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:25.97185235Z level=error msg="Executing migration failed" id="Update is_service_account column to nullable" error="attempt to write a readonly database"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:25.971874182Z level=error msg="Exec failed" error="attempt to write a readonly database" sql="ALTER TABLE user ADD COLUMN tmp_service_account BOOLEAN DEFAULT 0;\nUPDATE user SET tmp_service_account = is_service_account;\nALTER TABLE user DROP COLUMN is_service_account;\nALTER TABLE user RENAME COLUMN tmp_service_account TO is_service_account;"
monitoring_grafana               | Failed to start grafana. error: migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana               | migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
prometheus exited with code 0
prometheus                       | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus                       | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus                       | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus                       | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus                       | level=info ts=2023-02-11T20:48:26.475Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus                       | level=error ts=2023-02-11T20:48:26.477Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus                       | panic: Unable to create mmap-ed active query log
prometheus                       | 
prometheus                       | goroutine 1 [running]:
prometheus                       | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffc8a919ed5, 0xb, 0x14, 0x2dcaa00, 0xc000533c80, 0x2dcaa00)
prometheus                       |      /app/promql/query_logger.go:117 +0x4cd
prometheus                       | main.main()
prometheus                       |      /app/cmd/prometheus/main.go:364 +0x5245
monitoring_grafana exited with code 0
prometheus exited with code 0
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243144312Z level=info msg="Starting Grafana" version=9.3.6 commit=978237e7cb branch=HEAD compiled=2023-01-26T14:38:58Z
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243425683Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.24344461Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243451092Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243456643Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243461903Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243852615Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243864027Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.24387099Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243885218Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.24390689Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243918893Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243929613Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243937688Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243947798Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.243973798Z level=info msg="Path Home" path=/usr/share/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.24399027Z level=info msg="Path Data" path=/var/lib/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.244001552Z level=info msg="Path Logs" path=/var/log/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.244015148Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.244022031Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
monitoring_grafana               | logger=settings t=2023-02-11T20:48:28.244033763Z level=info msg="App mode production"
monitoring_grafana               | logger=sqlstore t=2023-02-11T20:48:28.244241694Z level=info msg="Connecting to DB" dbtype=sqlite3
monitoring_grafana               | logger=sqlstore t=2023-02-11T20:48:28.24428607Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r-----
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:28.26636261Z level=info msg="Starting DB migrations"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:28.270713545Z level=info msg="Executing migration" id="Update is_service_account column to nullable"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:28.270845668Z level=error msg="Executing migration failed" id="Update is_service_account column to nullable" error="attempt to write a readonly database"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:28.270865226Z level=error msg="Exec failed" error="attempt to write a readonly database" sql="ALTER TABLE user ADD COLUMN tmp_service_account BOOLEAN DEFAULT 0;\nUPDATE user SET tmp_service_account = is_service_account;\nALTER TABLE user DROP COLUMN is_service_account;\nALTER TABLE user RENAME COLUMN tmp_service_account TO is_service_account;"
monitoring_grafana               | Failed to start grafana. error: migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana               | migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
prometheus                       | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus                       | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus                       | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus                       | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus                       | level=info ts=2023-02-11T20:48:28.386Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus                       | level=error ts=2023-02-11T20:48:28.387Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus                       | panic: Unable to create mmap-ed active query log
prometheus                       | 
prometheus                       | goroutine 1 [running]:
prometheus                       | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffdc58a4ed5, 0xb, 0x14, 0x2dcaa00, 0xc000523ec0, 0x2dcaa00)
prometheus                       |      /app/promql/query_logger.go:117 +0x4cd
prometheus                       | main.main()
prometheus                       |      /app/cmd/prometheus/main.go:364 +0x5245
monitoring_grafana exited with code 1
prometheus exited with code 2
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501095589Z level=info msg="Starting Grafana" version=9.3.6 commit=978237e7cb branch=HEAD compiled=2023-01-26T14:38:58Z
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501408961Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501434972Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501444029Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501453257Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501466552Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501476792Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501486Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.50149648Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501505858Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501515697Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501520757Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501525796Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501531097Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501537048Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501590761Z level=info msg="Path Home" path=/usr/share/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501595431Z level=info msg="Path Data" path=/var/lib/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501599248Z level=info msg="Path Logs" path=/var/log/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501603055Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501606912Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
monitoring_grafana               | logger=settings t=2023-02-11T20:48:29.501611001Z level=info msg="App mode production"
monitoring_grafana               | logger=sqlstore t=2023-02-11T20:48:29.501701715Z level=info msg="Connecting to DB" dbtype=sqlite3
monitoring_grafana               | logger=sqlstore t=2023-02-11T20:48:29.50174059Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r-----
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:29.52129226Z level=info msg="Starting DB migrations"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:29.525709803Z level=info msg="Executing migration" id="Update is_service_account column to nullable"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:29.525819955Z level=error msg="Executing migration failed" id="Update is_service_account column to nullable" error="attempt to write a readonly database"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:29.525834904Z level=error msg="Exec failed" error="attempt to write a readonly database" sql="ALTER TABLE user ADD COLUMN tmp_service_account BOOLEAN DEFAULT 0;\nUPDATE user SET tmp_service_account = is_service_account;\nALTER TABLE user DROP COLUMN is_service_account;\nALTER TABLE user RENAME COLUMN tmp_service_account TO is_service_account;"
monitoring_grafana               | Failed to start grafana. error: migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana               | migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
cdn-up-and-running-edge-1        | 2023/02/11 20:48:29 [error] 7#7: *10 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_75.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_75.ext", host: "localhost:8081"
prometheus                       | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus                       | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus                       | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus                       | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus                       | level=info ts=2023-02-11T20:48:29.763Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus                       | level=error ts=2023-02-11T20:48:29.764Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus                       | panic: Unable to create mmap-ed active query log
prometheus                       | 
prometheus                       | goroutine 1 [running]:
prometheus                       | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff157b3ed5, 0xb, 0x14, 0x2dcaa00, 0xc0007e1ec0, 0x2dcaa00)
prometheus                       |      /app/promql/query_logger.go:117 +0x4cd
prometheus                       | main.main()
prometheus                       |      /app/cmd/prometheus/main.go:364 +0x5245
cdn-up-and-running-edge-1        | 2023/02/11 20:48:29 [error] 7#7: *1 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_24.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_24.ext", host: "localhost:8081"
monitoring_grafana exited with code 1
prometheus exited with code 2
monitoring_grafana exited with code 1
prometheus exited with code 2
cdn-up-and-running-edge-1        | 2023/02/11 20:48:30 [error] 7#7: *8 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_54.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_54.ext", host: "localhost:8081"
cdn-up-and-running-edge-1        | 2023/02/11 20:48:30 [error] 7#7: *6 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_98.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_98.ext", host: "localhost:8081"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988640283Z level=info msg="Starting Grafana" version=9.3.6 commit=978237e7cb branch=HEAD compiled=2023-01-26T14:38:58Z
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988894152Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988913309Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988919411Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988925091Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988930001Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988935532Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988940592Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988947174Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988952996Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988957825Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988962694Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988975259Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988980909Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.988990808Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false"
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.98899674Z level=info msg="Path Home" path=/usr/share/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.989004865Z level=info msg="Path Data" path=/var/lib/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.989009023Z level=info msg="Path Logs" path=/var/log/grafana
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.989016277Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.989020676Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
monitoring_grafana               | logger=settings t=2023-02-11T20:48:30.989030163Z level=info msg="App mode production"
monitoring_grafana               | logger=sqlstore t=2023-02-11T20:48:30.989114927Z level=info msg="Connecting to DB" dbtype=sqlite3
monitoring_grafana               | logger=sqlstore t=2023-02-11T20:48:30.989143101Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r-----
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:31.004832178Z level=info msg="Starting DB migrations"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:31.009256503Z level=info msg="Executing migration" id="Update is_service_account column to nullable"
monitoring_grafana               | Failed to start grafana. error: migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana               | migration failed (id = Update is_service_account column to nullable): attempt to write a readonly database
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:31.009359742Z level=error msg="Executing migration failed" id="Update is_service_account column to nullable" error="attempt to write a readonly database"
monitoring_grafana               | logger=migrator t=2023-02-11T20:48:31.00936888Z level=error msg="Exec failed" error="attempt to write a readonly database" sql="ALTER TABLE user ADD COLUMN tmp_service_account BOOLEAN DEFAULT 0;\nUPDATE user SET tmp_service_account = is_service_account;\nALTER TABLE user DROP COLUMN is_service_account;\nALTER TABLE user RENAME COLUMN tmp_service_account TO is_service_account;"
monitoring_grafana exited with code 1
cdn-up-and-running-edge-1        | 2023/02/11 20:48:31 [error] 7#7: *4 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_60.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_60.ext", host: "localhost:8081"
prometheus                       | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"
prometheus                       | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"
prometheus                       | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:335 host_details="(Linux 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 f15eea78067e (none))"
prometheus                       | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"
prometheus                       | level=info ts=2023-02-11T20:48:31.934Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus                       | level=error ts=2023-02-11T20:48:31.935Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
prometheus                       | panic: Unable to create mmap-ed active query log
prometheus                       | 
prometheus                       | goroutine 1 [running]:
prometheus                       | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffd3c753ed5, 0xb, 0x14, 0x2dcaa00, 0xc0007f6cc0, 0x2dcaa00)
prometheus                       |      /app/promql/query_logger.go:117 +0x4cd
prometheus                       | main.main()
prometheus                       |      /app/cmd/prometheus/main.go:364 +0x5245

I tried reading through prometheus/prometheus#5976, tried a few of the proposed solutions, but nothing worked, so I ended up giving root user permission to prometheus and 1000:1000 user to grafana containers

  prometheus:
    image: prom/prometheus:v2.17.1
    user: root
  ...
  grafana:
    image: grafana/grafana:latest
    user: "1000:1000"
   ...

This seems to have fixed the problem. wrk returns

Running 10m test @ http://localhost:8081
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   145.64ms  213.07ms   1.98s    84.62%
    Req/Sec    49.79     33.66   230.00     70.99%
  Latency Distribution
     50%   35.95ms
     75%  240.63ms
     90%  453.02ms
     99%    1.01s
  29946 requests in 10.00m, 12.32MB read
  Socket errors: connect 0, read 13956, write 19020644, timeout 9
  Non-2xx or 3xx responses: 334
Requests/sec:     49.91
Transfer/sec:     21.03KB

however the edge logs still print

cdn-up-and-running-edge-1        | 2023/02/11 20:54:06 [error] 7#7: *96 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.30.0.1, server: , request: "GET /item_36.ext HTTP/1.1", upstream: "http://172.30.0.2:8080/item_36.ext", host: "localhost:8081"

on every request which looks fishy but I'm not sure what it means.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@samlaf