Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return and log detailed services information on /ready #2055

Merged
merged 2 commits into from
Jun 8, 2022

Conversation

dimitarvdimitrov
Copy link
Contributor

@dimitarvdimitrov dimitarvdimitrov commented Jun 8, 2022

Signed-off-by: Dimitar Dimitrov dimitar.dimitrov@grafana.com

What this PR does

The /ready will now include each service's state when there is at least one non-Running service (e.g. Failed or Starting). The same will be logged at debug level.

Previously output:

Some services are not Running:
Running: 11
Starting: 1

Current output:

Some services are not Running:
distributor-service: Starting

Which issue(s) this PR fixes or relates to

Fixes #1780

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

This helps debug starting services more easily.

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
}

http.Error(w, msg.String(), http.StatusServiceUnavailable)
strMsg := msg.String()
level.Debug(util_log.Logger).Log(strMsg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need to debug-log this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My original thinking was that this should make it easier to see why components aren't ready - no need to port-forward to pods and curl endpoints - it will be in the logs.

Do you think that would be useful?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have logs for each starting/started module, which can be used to find out which modules aren't ready.

If we change this log line to only log non-ready modules, I think it would be more useful than logging ALL modules.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we change this log line to only log non-ready modules, I think it would be more useful than logging ALL modules.

agreed, pushed a change in d214659

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@pstibrany pstibrany enabled auto-merge (squash) June 8, 2022 12:20
@pstibrany pstibrany merged commit 716d0c6 into main Jun 8, 2022
@pstibrany pstibrany deleted the dimitar/readiness-verbosity branch June 8, 2022 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Increase verbosity around readiness
2 participants