-
Notifications
You must be signed in to change notification settings - Fork 576
stats: add bootstrap annotations for stats flush and eviction #3562
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Change-Id: I6c64ddd71aee82b570c7df25f13e2d710a046558 Signed-off-by: Kuat Yessenov <kuat@google.com>
😊 Welcome @kyessenov! This is either your first contribution to the Istio api repo, or it's been You can learn more about the Istio working groups, Code of Conduct, and contribution guidelines Thanks for contributing! Courtesy of your friendly welcome wagon. |
@@ -191,6 +191,25 @@ annotations: | |||
resources: | |||
- Pod | |||
|
|||
- name: sidecar.istio.io/statsFlushInterval | |||
featureStatus: Alpha | |||
description: Specifies the flush interval for push-based stat sinks, e.g. OTLP. Default interval is `5s`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
qq: if this is only for push-based stats sinks, what about prometheus (pull based)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Flush interval doesn't matter for prometheus. However, eviction piggy backs on flush timer, and eviction works for prometheus.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't get it. if it doesn't matter, but we evict when we flush, then it does matter...? And if the default is 5s, isn't that going to break prometheus?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Envoy always runs a stats flush timer, regardless of prometheus. During that timer, server stats are published (e.g. memory allocations, etc), and some other work is done, e.g. eviction.
Prometheus endpoint can snip stats are any point through, in between flushes.
So flush impacts prometheus only indirectly, e.g. by updating/removing some metrics.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe more concretely... I send 1 request to foo.com
. I get an istio_request_total metric for it.
When, with prometheus, will this metric stop showing up in /stats/prometheus with/without this annotation set?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For flush annotation: it will show up immediately and stay forever.
For eviction annotation: same, but it will disappear after 2*eviction interval unless the metric gets another hit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Sorry I was being a bit dense, github UI was hiding part of the PR
/retest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, Istio don't have first class API for bootstrap, this's the best option for now.
Change-Id: I245a8b8e70c54ecb4fd36a9d8fffdd0008bc78e9 Signed-off-by: Kuat Yessenov <kuat@google.com>
Change-Id: I6c64ddd71aee82b570c7df25f13e2d710a046558