Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

consul_autopilot_healthy metric is NaN #9274

Closed
lawliet89 opened this issue Nov 25, 2020 · 2 comments · Fixed by #9905
Closed

consul_autopilot_healthy metric is NaN #9274

lawliet89 opened this issue Nov 25, 2020 · 2 comments · Fixed by #9905
Assignees
@zalimeni
Labels
theme/telemetry Anything related to telemetry or observability type/docs Documentation needs to be created/updated/clarified

Comments

@lawliet89
Copy link
Contributor

lawliet89 commented Nov 25, 2020 •
edited
Loading

Overview of the Issue

After upgrading to 1.9, the metric consul_autopilot_healthy is NaN. Might be related to #9208

image

The autopilot health endpoint reports the cluster is healthy.

image

Reproduction Steps

Steps to reproduce this issue, eg:

  1. Upgrade to 1.9
  2. Go to /v1/agent/metrics?format=prometheus

Operating system and Environment details

Linux AMD 64 running on Kubernetes

For additional context, this did not present itself in 1.9-beta1. It might be related to the changes introduced in #9198
@lawliet89 lawliet89 changed the title consul_autopilot_healthy is nan consul_autopilot_healthy metric is NaN Nov 25, 2020
@mkeeler
Copy link
Member

mkeeler commented Nov 30, 2020

@lawliet89 When you used the v1/agent/metrics endpoint did you query the leader? That metric is only ever accurate on the current leader.

@mikemorris mikemorris added the theme/telemetry Anything related to telemetry or observability label Nov 30, 2020
@lawliet89
Copy link
Contributor Author

lawliet89 commented Dec 1, 2020 •
edited
Loading

Thanks. You're right. 1 is only returned from a single instance which is the leader.

image

I went back to 1.8.4 to check and only the leader would return 1 and the rest of the agents would return nothing.

If this is expected, please close the issue, although this is a change in behaviour that might trip others up.

@jsosulska jsosulska added the type/docs Documentation needs to be created/updated/clarified label Mar 16, 2021
jsosulska added a commit that referenced this issue Mar 19, 2021
@jsosulska
Fixes #9274-Addresses NaN output in autopilot
5ee4a40
@jsosulska jsosulska mentioned this issue Mar 19, 2021
Merged
jsosulska added a commit that referenced this issue Mar 22, 2021
@jsosulska
Update telemetry docs (#9905)
277de5a 
* Fixes #2379-Improve interval explanation in the telemetry doc

* Fixes #4734-Update consul memory metrics

* Fixes #4836-Removed node.deregistration as that isn't in state.go

* Fixes #8986 partially-Trim redundant language

* Fixes #9087-Adds helpful details to telemetry on autopilot

* Fixes #9274-Addresses NaN output in autopilot
hashicorp-ci pushed a commit that referenced this issue Mar 22, 2021
@jsosulska @hashicorp-ci
Update telemetry docs (#9905)
060a614 
* Fixes #2379-Improve interval explanation in the telemetry doc

* Fixes #4734-Update consul memory metrics

* Fixes #4836-Removed node.deregistration as that isn't in state.go

* Fixes #8986 partially-Trim redundant language

* Fixes #9087-Adds helpful details to telemetry on autopilot

* Fixes #9274-Addresses NaN output in autopilot
hashicorp-ci pushed a commit that referenced this issue Mar 22, 2021
@jsosulska @hashicorp-ci
Update telemetry docs (#9905)
d8be22a 
* Fixes #2379-Improve interval explanation in the telemetry doc

* Fixes #4734-Update consul memory metrics

* Fixes #4836-Removed node.deregistration as that isn't in state.go

* Fixes #8986 partially-Trim redundant language

* Fixes #9087-Adds helpful details to telemetry on autopilot

* Fixes #9274-Addresses NaN output in autopilot
dizzyup pushed a commit that referenced this issue Apr 21, 2021
@jsosulska @dizzyup
Update telemetry docs (#9905)
6dbb324 
* Fixes #2379-Improve interval explanation in the telemetry doc

* Fixes #4734-Update consul memory metrics

* Fixes #4836-Removed node.deregistration as that isn't in state.go

* Fixes #8986 partially-Trim redundant language

* Fixes #9087-Adds helpful details to telemetry on autopilot

* Fixes #9274-Addresses NaN output in autopilot
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zalimeni zalimeni

Labels
theme/telemetry Anything related to telemetry or observability type/docs Documentation needs to be created/updated/clarified
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update telemetry docs hashicorp/consul
5 participants
@lawliet89 @mikemorris @mkeeler @zalimeni @jsosulska