-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Reporting] Add output size stats to telemetry metrics #112037
[Reporting] Add output size stats to telemetry metrics #112037
Conversation
Pinging @elastic/kibana-reporting-services (Team:Reporting Services) |
Pinging @elastic/kibana-app-services (Team:AppServices) |
@@ -161,7 +175,7 @@ export async function getReportingUsage( | |||
jobTypes: { | |||
terms: { field: JOB_TYPES_FIELD, size: DEFAULT_TERMS_SIZE }, | |||
aggs: { | |||
appNames: { terms: { field: OBJECT_TYPES_FIELD, size: DEFAULT_TERMS_SIZE } }, // NOTE Discover/CSV export is missing the 'meta.objectType' field, so Discover/CSV results are missing for this agg |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is no longer valid as meta.objectType
is added for all export types in the RequestHandler class.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did not test locally, but these changes look good to me 🍻
I have one open question that I'd like to get your thoughts on, happy to re-review if you agree with the proposal but I won't block on it.
output_size: { | ||
max: sizeMax?.value ?? null, | ||
min: sizeMin?.value ?? null, | ||
avg: sizeAvg?.value ?? null, | ||
}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder whether a percentile based aggregation would not provide better insight into the questions we may have regarding output size (es docs).
Would it be more useful to know that the average output size is, e.g., 500kb
or that 99% of outputs are below 700kb
?
Percentile based aggregation can be more expensive to calcuate, which might disqualify it, but from what I can tell it looks pretty safe (also considering we are working with report documents so not sure there will be millions).
Something like:
output_size: {
25th: ...,
50th: ...,
75th: ...,
95th: ...,
99th: ..., // outliers
}
Let me know what you think @tsullivan !
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
++ I am in favor of this. At the time I implemented it with avg
, I was hoping to use median instead, but there isn't an actual median
aggregation. With percentiles, we get the median from the 50th percentile. This is good!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Core changes lgtm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
schema changes LGTM
@elasticmachine merge upstream |
3895259
to
9d6b579
Compare
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
* [Reporting] Add output size stats to telemetry metrics * fix types * add output_size for each jobtype * add size metrics for each job type * use more mock data in unit tests * clean up test * update test snapshots * update telemetry mapping * SizeMetrics => SizePercentiles * DocCount interface * fix tests * Update get_export_stats.ts * update snapshots
…3057) * [Reporting] Add output size stats to telemetry metrics * fix types * add output_size for each jobtype * add size metrics for each job type * use more mock data in unit tests * clean up test * update test snapshots * update telemetry mapping * SizeMetrics => SizePercentiles * DocCount interface * fix tests * Update get_export_stats.ts * update snapshots
Summary
This PR adds size statistics on the reports in the cluster.