hydra investigate show-monitor fails with Error message {"message":"Dashboard not found","status":"not-found"}

[yarongilor]

16:49:36 >>> Run hydra command on the new SCT runner w/ public IP: 3.239.252.185
16:49:36
16:49:36 Running on Build Server...
16:49:37 Docker version 19.03.8, build afacb8b
16:49:37 Pull version v0.99 from Docker Hub...
16:49:37 v0.99: Pulling from scylladb/hydra
16:49:37 Digest: sha256:b12d4e4b0930947f5764ee0709b8a1daaeb0fd44ccfe4f7c8bbbb7ad8f57abfd
16:49:37 Status: Image is up to date for scylladb/hydra:v0.99
16:49:37 docker.io/scylladb/hydra:v0.99
16:49:37 Obtaining QA SSH keys...
16:49:39 QA SSH keys obtained.
16:49:39 Making sure the ownerships of results directories are of the user
16:49:47 SCT Runner IP: 3.239.252.185
16:49:47 Agent pid 8067
16:49:47 Identity added: /jenkins/.ssh/scylla-qa-ec2 (fruch@scylladb.com)
16:49:47 Identity added: /jenkins/.ssh/scylla-test (/jenkins/.ssh/scylla-test)
16:49:47 Going to run a Hydra commands on SCT runner '3.239.252.185'...
16:49:47 Syncing /jenkins/slave/workspace/QA-tools/hydra-show-monitor/scylla-cluster-tests to the SCT runner instance...
16:49:47 Host 3.239.252.185 not found in /jenkins/.ssh/known_hosts
16:49:47 Warning: Permanently added '3.239.252.185' (ECDSA) to the list of known hosts.
16:49:49 AWS_* environment variables found and will passed to Hydra container.
16:49:50 Going to run './sct.py investigate show-monitor f4793e2e-4103-4081-9234-54feaf6cb929'...
16:49:59 Unable to find image 'scylladb/hydra:v0.99' locally
16:49:59 v0.99: Pulling from scylladb/hydra
16:49:59 b380bbd43752: Pulling fs layer
16:49:59 14063a2781d6: Pulling fs layer
16:49:59 22dcb0c8da00: Pulling fs layer
16:49:59 37d4c4df1df1: Pulling fs layer
16:49:59 9be0c12d73f6: Pulling fs layer
16:49:59 202db2b999fe: Pulling fs layer
16:49:59 dc5f39ee644f: Pulling fs layer
16:49:59 5d6964fa9c4e: Pulling fs layer
16:49:59 554dd09a0e32: Pulling fs layer
16:49:59 37d4c4df1df1: Waiting
16:49:59 9be0c12d73f6: Waiting
16:49:59 202db2b999fe: Waiting
16:49:59 dc5f39ee644f: Waiting
16:49:59 5d6964fa9c4e: Waiting
16:49:59 554dd09a0e32: Waiting
16:49:59 14063a2781d6: Verifying Checksum
16:49:59 14063a2781d6: Download complete
16:50:00 37d4c4df1df1: Verifying Checksum
16:50:00 37d4c4df1df1: Download complete
16:50:00 22dcb0c8da00: Verifying Checksum
16:50:00 22dcb0c8da00: Download complete
16:50:00 202db2b999fe: Verifying Checksum
16:50:00 202db2b999fe: Download complete
16:50:00 b380bbd43752: Verifying Checksum
16:50:00 b380bbd43752: Download complete
16:50:00 9be0c12d73f6: Verifying Checksum
16:50:00 9be0c12d73f6: Download complete
16:50:00 5d6964fa9c4e: Verifying Checksum
16:50:00 5d6964fa9c4e: Download complete
16:50:02 554dd09a0e32: Verifying Checksum
16:50:02 554dd09a0e32: Download complete
16:50:02 dc5f39ee644f: Verifying Checksum
16:50:02 dc5f39ee644f: Download complete
16:50:04 b380bbd43752: Pull complete
16:50:07 14063a2781d6: Pull complete
16:50:07 22dcb0c8da00: Pull complete
16:50:07 37d4c4df1df1: Pull complete
16:50:09 9be0c12d73f6: Pull complete
16:50:09 202db2b999fe: Pull complete
16:50:16 dc5f39ee644f: Pull complete
16:50:18 5d6964fa9c4e: Pull complete
16:50:38 554dd09a0e32: Pull complete
16:50:38 Digest: sha256:b12d4e4b0930947f5764ee0709b8a1daaeb0fd44ccfe4f7c8bbbb7ad8f57abfd
16:50:38 Status: Downloaded newer image for scylladb/hydra:v0.99
16:50:41 Obtaining QA SSH keys...
16:50:42 Completed 3.3 KiB/7.3 KiB (45.2 KiB/s) with 3 file(s) remaining
download: s3://scylla-qa-keystore/scylla-qa-ec2 to ../.ssh/scylla-qa-ec2
16:50:42 Completed 3.3 KiB/7.3 KiB (45.2 KiB/s) with 2 file(s) remaining
Completed 5.7 KiB/7.3 KiB (63.7 KiB/s) with 2 file(s) remaining
download: s3://scylla-qa-keystore/scylla-test to ../.ssh/scylla-test
16:50:42 Completed 5.7 KiB/7.3 KiB (63.7 KiB/s) with 1 file(s) remaining
Completed 7.3 KiB/7.3 KiB (77.1 KiB/s) with 1 file(s) remaining
download: s3://scylla-qa-keystore/support to ../.ssh/support
16:50:43 QA SSH keys obtained.
16:50:43 Installing Argus pip package...
16:50:43 [TODO] Remove this once Argus stabilizes
16:50:58 Package installed.
16:51:00 Get Docker Hub credentials
16:51:00 Login to Docker Hub as `scyllaqatest'
16:51:00 New directory created: /home/ubuntu/sct-results/20211115-145100-004702-investigate-show-monitor
16:51:00 Search monitoring stack archive files for test id f4793e2e-4103-4081-9234-54feaf6cb929 and restoring...
16:51:00 Checking that docker is available...
16:51:00 Docker is available
16:51:00 Restoring monitoring stack from archive f4793e2e-4103-4081-9234-54feaf6cb929/20211114_111511/monitor-set-f4793e2e.tar.gz
16:51:00 Download file https://cloudius-jenkins-test.s3.amazonaws.com/f4793e2e-4103-4081-9234-54feaf6cb929/20211114_111511/monitor-set-f4793e2e.tar.gz to directory /tmp/tmpymp1s9ws
16:51:00 Downloading f4793e2e-4103-4081-9234-54feaf6cb929/20211114_111511/monitor-set-f4793e2e.tar.gz from cloudius-jenkins-test
16:51:01 Downloaded finished
16:52:52 Docker containers for monitoring stack are started
16:52:52 Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message {"message":"Dashboard not found","status":"not-found"}
16:52:52 Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message {"message":"Dashboard not found","status":"not-found"}
16:53:09 Uploading sct dashboard [try #1]
16:53:09 Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message {"message":"Dashboard not found","status":"not-found"}
16:53:09 Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message {"message":"Dashboard not found","status":"not-found"}
16:53:29 Uploading sct dashboard [try #2]
16:53:29 Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message {"message":"Dashboard not found","status":"not-found"}
16:53:29 Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message {"message":"Dashboard not found","status":"not-found"}
16:53:49 'restore_sct_dashboards': Number of retries exceeded!
16:53:49 Error during uploading sct monitoring data Error uploading dashboard /tmp/tmpymp1s9ws/scylla-monitoring-src/sct_monitoring_addons/scylla-dash-per-server-nemesis.master.json. Error message {"message":"Dashboard not found","status":"not-found"}
16:53:49 Errors were found when restoring Scylla monitoring stack
16:53:49 Killing Grafana service
16:53:49 Killing Prometheus service
16:53:49 Killing Alert service
16:53:49 Cleaning SSH agent
16:53:49 Agent pid 8067 killed

[yarongilor]

@roydahan , can you please advise for anyone who can help with that?

[dkropachev]

@yarongilor , how to reproduce it ?

[yarongilor]

@yarongilor , how to reproduce it ?

just run hydra investigate show-monitor with a recent job. or rerun this job: https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/27/

[vponomaryov]

It was fixed here: #4144
Closing it.