Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingester circuit breaker deactivate() function #9743

Merged
merged 6 commits into from
Oct 28, 2024
Merged

Conversation

pr00se
Copy link
Contributor

@pr00se pr00se commented Oct 26, 2024

What this PR does

Add a deactivate() method to the ingester circuit breakers to allow them to be disabled and re-enabled dynamically.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@pr00se pr00se requested a review from a team as a code owner October 26, 2024 00:28
@pr00se pr00se marked this pull request as draft October 26, 2024 00:28
@pr00se pr00se marked this pull request as ready for review October 28, 2024 16:17
@pr00se pr00se changed the title Deactivate circuit breaker Allow ingester circuit breakers to be disabled dynamically Oct 28, 2024
@pr00se pr00se changed the title Allow ingester circuit breakers to be disabled dynamically Ingester circuit breaker deactivate() function Oct 28, 2024
@pr00se pr00se changed the title Ingester circuit breaker deactivate() function Ingester circuit breaker deactivate() function Oct 28, 2024
cb.active.Store(true)
})
}

func (cb *circuitBreaker) deactivate() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this PR basically adds a support to deactivate circuit breakers, but it doesn't actually deactivate them anywhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct -- I wanted to keep the changes small, so will add the invocation in another PR. The plan is to call it from https://github.com/grafana/mimir/blob/main/pkg/ingester/downscale.go when we change the read-only state.

Copy link
Contributor

@duricanikolic duricanikolic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pr00se pr00se merged commit 219f278 into main Oct 28, 2024
29 checks passed
@pr00se pr00se deleted the deactivate-circuit-breaker branch October 28, 2024 16:46
pr00se added a commit that referenced this pull request Oct 28, 2024
* Log request type in circuit breaker state change messages

* Test blocking reads if push circuit breaker is open

* Add circuit breaker deactivate()

* Don't set active state twice when initial delay is 0

* Log when breakers are activated/deactivated

* Only log when actually changing circuit breaker state

(cherry picked from commit 219f278)

Co-authored-by: Patryk Prus <patryk@grafana.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants