Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark a service as unhealthy for a certain time period. #5976

Closed
WouterBaeyens opened this issue Jun 17, 2019 · 2 comments
Closed

Mark a service as unhealthy for a certain time period. #5976

WouterBaeyens opened this issue Jun 17, 2019 · 2 comments

Comments

@WouterBaeyens
Copy link

Feature Description

I want to be able to mark a service as unhealthy for a certain time period.

This could be accomplished in a number of ways:

  • a TTL mechanism that flips the client out of maintenance mode
  • a TTL check that defaults to healthy.

Use Case(s)

If I can predict my service will go down, I want to avoid directing users to this failing service.

Currently I have no good way to avoid a percentage of the user-base going to this unhealthy service until another health-check picks up on it.

  • current health checks have a small delay
  • maintenance mode has to be manually disabled.
    If the request to disable maintenance mode fails, the service stays unreachable.
  • a TTL check introduces the burden of converting the service to constantly send it's current status.

Introducing a TTL for maintenance was part of a suggestion in #345 and I don't think a workaround for my use-case currently exists.
(see my question on Stack Overflow)

@pearkes
Copy link
Contributor

pearkes commented Jun 26, 2019

If I understand correctly, my guess is we wouldn't want to add some sort of "unhealthy time period" feature. We'd recommend explicitly transitioning a service to healthily and not during maintenance.

So:

  1. Enable maintenance mode for a service
  2. Perform whatever operation
  3. Unset maintenance mode

You could do this from a 3rd system that has a timer to automate in a way that you specify. However, typically the use-case for people is that maintenance could take an unknown amount of time, and choosing and end time before maintenance begins could pose other risks.

I hope that makes sense. Thank you for opening an issue!

@pearkes pearkes closed this as completed Jun 26, 2019
@WouterBaeyens
Copy link
Author

My original idea was to send a ttl that defaults to healthy from an onShutdown event of my server.
This would remove the delay for Consul to notice the service being unhealthy if I were to only use pings.

I agree, that in the use-case of planned maintenance a timer might pose certain risks.
If the use-cases for this feature are indeed to small in comparison to the needed effort, it saddens me a little, but I understand.

Thank you for your reply!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants