try to avoid alert flaps.. #486
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lionel-kaufmann-claranet
requested review from
BzSpi,
cvauvarin,
haedri,
JMHAUTBOIS-Claranet,
pdecat,
Shr3ps and
tchernomax
as code owners
lionel-kaufmann-claranet
added
bug
lionel-kaufmann-claranet
assigned lionel-kaufmann-claranet, pdecat, tchernomax and cvauvarin and unassigned lionel-kaufmann-claranet
pdecat
reviewed
Sep 14, 2023
lionel-kaufmann-claranet
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Splunk otel agent use supervisor's RPC interface to collect status.
That's why in https://docs.splunk.com/Observability/gdi/monitors-hosts/supervisor.html for supervisor.state metric they directly link the gauge values to supervisor codes : http://supervisord.org/subprocess.html#process-states
Now consider a particular supervisor code 10 : Starting.
Stage 1: the PRODUCTION Way
"What ? oh ! this is legacy code, Well, it was developed by a guy who left company 12 years ago.... wait...who was the guy ? Did not remember...But no matter the guy !
Do do not worry ! We have solve stability issues by restarting all this process every hours..."
next ...
Stage 2 : the DEVEL Way
"What ? oh ! this is legacy code, with out-of-life libraries, with out-of-life design patterns, with npe, with home made libraries, with memory leaks and sometime a lock blocking condition...did i forget something ?
Do do not worry ! We have solve stability issues by restarting all this process every hours..."
next...
Stage 3: the BUDGET Way ( endly )
"What ? oh ! this is legacy code, there is an incredible tech debt there ! Refactoring and rebuild price is very high with no added value...Therefore the line in the budget approval list is top bottom.
Do do not worry ! We have solve stability issues by restarting all this process every hours..."
Frequency : ~ 24 times per day
It's a bit noisy to have major alert there...
^_^
By the way... all this noise for a RTFM :
Should we consider removing var.process_state_threshold_major ?