Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[APM] Performance fix for 'cardinality' telemetry task #144061

Merged
merged 3 commits into from
Oct 27, 2022
Merged

[APM] Performance fix for 'cardinality' telemetry task #144061

merged 3 commits into from
Oct 27, 2022

Conversation

basepi
Copy link
Contributor

@basepi basepi commented Oct 26, 2022 •
edited by kibanamachine
Loading

Summary

Someone found an instance where this telemetry task took 29 hours. We should figure out why the timeout argument didn't stop that earlier in that instance, but I also noticed that we're not narrowing the indices at all on these queries. We now only search the traces indices which should (hopefully) improve the performance on this dramatically.

/cc @felixbarny @dgieselaar

For maintainers

Closes #143819.

@basepi basepi requested a review from dgieselaar October 26, 2022 18:41
@basepi basepi requested a review from a team as a code owner October 26, 2022 18:41
@botelastic botelastic bot added the Team:APM All issues that need APM UI Team support label Oct 26, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/apm-ui (Team:APM)

@basepi basepi added the release_note:skip Skip the PR/issue when compiling release notes label Oct 26, 2022
@basepi
Copy link
Contributor Author

basepi commented Oct 26, 2022

I also wonder if it would be worthwhile to make this 1h instead of 1d? A lot less data to sift through.

sorenlouv
sorenlouv reviewed Oct 27, 2022
View reviewed changes
x-pack/plugins/apm/server/lib/apm_telemetry/collect_data_telemetry/tasks.ts
@@ -1058,6 +1059,7 @@ export const tasks: TelemetryTask[] = [
});

const rumAgentCardinalityResponse = await search({
index: [indices.transaction],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably should make index required

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got a fix for that in the works, just waiting on @basepi's permission to push it to this PR. That will also close #143819

@dgieselaar dgieselaar added v8.6.0 v8.5.1 bug Fixes for quality problems that affect the customer experience labels Oct 27, 2022
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@basepi basepi added the backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) label Oct 27, 2022
@dgieselaar
Copy link
Member

@basepi I think it's good to default to 24h in most cases, because e.g. a 1hr time range for rum data leads to very different results if the job runs at midnight.

@basepi basepi merged commit a5c8ebe into elastic:main Oct 27, 2022
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 27, 2022
[APM] Performance fix for 'cardinality' telemetry task (elastic#144061)
d874d57 
* Performance fix for 'cardinality' telemetry task

* Make timeout/index required for telemetry searches

* Fix tests

Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
(cherry picked from commit a5c8ebe)
@kibanamachine kibanamachine mentioned this pull request Oct 27, 2022
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.5

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@basepi basepi mentioned this pull request Oct 27, 2022
Closed
1 task
kibanamachine added a commit that referenced this pull request Oct 27, 2022
@kibanamachine
[APM] Performance fix for 'cardinality' telemetry task (#144061) (#14…
d9d2438 
…4124)

* Performance fix for 'cardinality' telemetry task

* Make timeout/index required for telemetry searches

* Fix tests

Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
(cherry picked from commit a5c8ebe)

Co-authored-by: Colton Myers <colton@myers.fm>
jloleysens added a commit to jloleysens/kibana that referenced this pull request Oct 27, 2022
@jloleysens
Merge branch 'main' into files-move-to-src
1b2d913 
* main: (24 commits)
  [Files] Add file upload to file picker (elastic#143969)
  [Security solution] Guided onboarding, alerts & cases (elastic#143598)
  [APM] Critical path for a single trace (elastic#143735)
  skip failing test suite (elastic#143933)
  [Fleet] Update GH Projects automation (elastic#144123)
  [APM] Performance fix for 'cardinality' telemetry task (elastic#144061)
  [Enterprise Search] Attach ML Inference Pipeline - Pipeline re-use (elastic#143979)
  [8.5][DOCS] Add support for differential logs (elastic#143242)
  Bump nwsapi from v2.2.0 to v2.2.2 (elastic#144001)
  [APM] Add waterfall to dependency operations (elastic#143257)
  [Shared UX] Add deprecation message to kibana react Markdown (elastic#143766)
  [Security Solution][Endpoint] Adds RBAC API checks for Blocklist (elastic#144047)
  Improve `needs-team` auto labeling regex (elastic#143787)
  [Reporting/CSV Export] _id field can not be formatted (elastic#143807)
  Adds SavedObjectsWarning to analytics results pages. (elastic#144109)
  Bump chromedriver to 107 (elastic#144073)
  Update cypress (elastic#143755)
  [Maps] nest security layers in layer group (elastic#144055)
  [Lens][Agg based Heatmap] Navigate to Lens Agg based Heatmap. (elastic#143820)
  Added support of saved search (elastic#144095)
  ...
@mistic mistic removed the v8.5.0 label Nov 1, 2022
@mistic
Copy link
Member

mistic commented Nov 1, 2022

Added v8.5.1 tag has the backport for 8.5 didn't make the v8.5.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sorenlouv sorenlouv sorenlouv approved these changes

@dgieselaar dgieselaar Awaiting requested review from dgieselaar

Assignees
No one assigned
Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) bug Fixes for quality problems that affect the customer experience release_note:skip Skip the PR/issue when compiling release notes Team:APM All issues that need APM UI Team support v8.5.1 v8.6.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[APM] Make sure telemetry queries are scoped and have a timeout
7 participants
@basepi @elasticmachine @kibana-ci @dgieselaar @kibanamachine @mistic @sorenlouv