-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Most metrics have disappeared after version v0.22.0 #3236
Comments
Thanks for reporting. Could you incrementally go up/down to see when the metrics stopped coming in? |
SeverSide: repos:
- id: '/.*/'
apply_requirements:
- mergeable
- approved
allowed_overrides:
- apply_requirements
- workflow
allow_custom_workflows: false
metrics:
prometheus:
endpoint: /metrics Working in: Failed on: This error was fixed here:
{
"level": "error",
"ts": "2023-03-17T11:39:25.075+0100",
"caller": "events/command_runner.go:427",
"msg": "PANIC: potential tally.Scope() vs Prometheus usage contract mismatch: if this occurs after using Scope.Tagged(), different metric names must be used than were registered with the parent scope: a previously registered descriptor with the same fully-qualified name as Desc{fqName: \"test_project_execution_success\", help: \"test_project_execution_success counter\", constLabels: {}, variableLabels: [workspace base_repo pr_number project project_path terraform_version]} has different label names or a different help string\n/home/runner/go/pkg/mod/github.com/uber-go/tally@v3.5.0+incompatible/prometheus/reporter.go:306 (0x10559579b)\n/home/runner/go/pkg/mod/github.com/uber-go/tally@v3.5.0+incompatible/prometheus/reporter.go:366 (0x10559623f)\n/home/runner/go/pkg/mod/github.com/uber-go/tally@v3.5.0+incompatible/scope.go:280 (0x105492d2f)\n/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:58 (0x105938273)\n/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:35 (0x105937dbb)\n/home/runner/work/atlantis/atlantis/server/events/project_command_pool_executor.go:48 (0x105948727)\n/home/runner/work/atlantis/atlantis/server/events/plan_command_runner.go:216 (0x10593d767)\n/home/runner/work/atlantis/atlantis/server/events/plan_command_runner.go:251 (0x10593de6b)\n/home/runner/work/atlantis/atlantis/server/events/command_runner.go:296 (0x10592a423)\n/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_arm64.s:1172 (0x105020303)\n",
"json": {
"repo": "albertorm95/atlantis-test",
"pull": "7"
},
"stacktrace": "github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\t/home/runner/work/atlantis/atlantis/server/events/command_runner.go:427\nruntime.gopanic\n\t/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/panic.go:890\ngithub.com/uber-go/tally/prometheus.NewReporter.func1\n\t/home/runner/go/pkg/mod/github.com/uber-go/tally@v3.5.0+incompatible/prometheus/reporter.go:306\ngithub.com/uber-go/tally/prometheus.(*reporter).AllocateCounter\n\t/home/runner/go/pkg/mod/github.com/uber-go/tally@v3.5.0+incompatible/prometheus/reporter.go:366\ngithub.com/uber-go/tally.(*scope).Counter\n\t/home/runner/go/pkg/mod/github.com/uber-go/tally@v3.5.0+incompatible/scope.go:280\ngithub.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:58\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Plan\n\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:35\ngithub.com/runatlantis/atlantis/server/events.runProjectCmds\n\t/home/runner/work/atlantis/atlantis/server/events/project_command_pool_executor.go:48\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).run\n\t/home/runner/work/atlantis/atlantis/server/events/plan_command_runner.go:216\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\t/home/runner/work/atlantis/atlantis/server/events/plan_command_runner.go:251\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\t/home/runner/work/atlantis/atlantis/server/events/command_runner.go:296"
} Failed on: Only: So more likely to be the addition of init metrics:
cc: @Fabianoshz |
Do you think this is something @albertorm95 or @Fabianoshz can fix or should we revert the prs #2767 and #2847 ? |
What do you think @nitrocode? I think having metrics is more useful than having non-init metrics but thats from my perspective. I really would like to fix the thing, but I really don't have time now for looking into it. |
Also not sure if related but for me all the metrics of type
I'm running the latest version |
I'll try to fix this. edit 1: edit 2: edit 3:
|
Hello!
When we used
v0.21.0
we had theses metrics:atlantis_cmd_comment_plan_project_execution_success
atlantis_cmd_comment_plan_project_execution_failure
Then we updated to
v0.22.3
and stop seeing them, now we just have:atlantis_cmd_comment_plan_execution_time
Basically only
atlantis_cmd_comment_%s_execution_time
metricsLooking at the drift between v0.21.0 and v0.22.3 is kind of rough, have you notice this?
Have you context on what could have happened?
NOTE: we just tested with
v0.23.2
and same behaviourThe text was updated successfully, but these errors were encountered: