Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In scaled out environments: define a policy to report stopped heartbeats only when instances count goes below a threshold #3317

Closed
mauroservienti opened this issue Dec 13, 2022 · 11 comments
Labels

Comments

@mauroservienti
Copy link
Member

mauroservienti commented Dec 13, 2022

Allow defining a policy that sets/configs some minimum instance count and only reports when the number of instances goes below it. Instead of reporting stopped heartbeats for every instance.

@eirikurharaldsson
Copy link

Is there any update on this?

@mauroservienti
Copy link
Member Author

Thanks for chiming in, @eirikurharaldsson. At the moment, there isn't an ETA or an update. Being labeled as candidate-for-next-release, it might be picked up anytime soon.

@eirikurharaldsson
Copy link

Is this supported in ServiceControl 5?

@eirikurharaldsson
Copy link

Can i override the host identifier, so that the name is stable like dev1, dev2, dev3, test1, test2...
(make it unique per container stage (dev, test, prod).
https://docs.particular.net/nservicebus/hosting/override-hostid

I noticed this issue. Do you know if heartbeat plugin support override of the hostid?
Particular/NServiceBus#7026

@awright18
Copy link
Contributor

awright18 commented Sep 19, 2024

The latest version of the platform (ServicePulse 1.42.1 and ServiceControl 5.9) included this fix. Can you let us know if that helps in your scenario?

@johnsimons
Copy link
Member

@mauroservienti I think we can close this one now that we handle scaled-out environments better, thoughts?

@mauroservienti
Copy link
Member Author

I'm not sure about it, @johnsimons. This is more about defining a lower bound threshold. The feedback was more about "I don't want to be bothered by an alarm if my instances count is still greater than X."

@johnsimons
Copy link
Member

@mauroservienti regarding "I don't want to be bothered by an alarm if my instances count is still greater than X."

So, the new functionality enables this scenario,

I don't want to be bothered by an alarm if my instances count is still greater than 1.

It does not allow customers to set a >1 lower bound, which we discussed in the internal RFC.
The decision was that it felt like we would be reinventing what K8s does out of the box, and it would also have the risk of possible maintenance burden on customers to ensure the replica number is kept in sync with what is in K8s.

So, given we now handle scaled-out environments better, I feel we can close this issue, and if this surfaces again, we can re-evaluate it at that time, thoughts?

@johnsimons
Copy link
Member

Based on my previous comment, I am closing this one.

@udidahan
Copy link
Member

udidahan commented Nov 3, 2024

@eirikurharaldsson if you feel that we missed something, please do let us know and re-open the issue. Thanks.

@eirikurharaldsson
Copy link

If this only raises an alarm if no instance of the same endpoint is running,
and does not leave dead replicas in inactive heartbeat endpoints, then it´s fine by us.
We will upgrade to the new version of ServiceControl asap.
Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants