[ML] Transforms/Data Frame Analytics: Fix freezing wizard for indices with massive amounts of fields. #98259
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
walterra
added
bug
|
Pinging @elastic/ml-ui (:ml) |
|
Tested this on a filebeat index with around 1300 fields, and there is a big performance improvement when opening the Transform wizard. Could the same fix be applied to the data frame analytics wizard, which also suffers from the same issue with the source index preview grid? Also is there anything we can do for the Transform preview grid, when using a |
qn895
reviewed
Apr 26, 2021
peteharverson
approved these changes
Apr 27, 2021
💚 Build SucceededMetrics [docs]Async chunks
Page load bundle
History
To update your PR or re-run it, just comment with: cc @walterra |
alvarezmelissa87
approved these changes
Apr 27, 2021
walterra
changed the title
walterra
added a commit
to walterra/kibana
that referenced
this pull request
Apr 28, 2021
… with massive amounts of fields. (elastic#98259) The transform wizard can become very slow when used with indices with e.g. 1000+ fields. This PR fixes it by prefetching 500 random documents to create a list of populated/used fields and passes those on to the data grid component instead of all available fields from the list derived via Kibana index patterns. For example, for an out of the box metricbeat index, this reduces the list of passed on fields from 3000+ to ~120 fields. Previously, the page would freeze on load for tens of seconds and would freeze again on every rerender. With the applied update, the page loads almost instantly again and remains responsive. Note this fix of reducing available fields is only applied to the data grid preview component. All fields are still available to create the configuration in the UI for groups and aggregations. These UI components, e.g. the virtualized dropdowns, can handle large lists of fields.
walterra
added a commit
to walterra/kibana
that referenced
this pull request
Apr 28, 2021
… with massive amounts of fields. (elastic#98259) The transform wizard can become very slow when used with indices with e.g. 1000+ fields. This PR fixes it by prefetching 500 random documents to create a list of populated/used fields and passes those on to the data grid component instead of all available fields from the list derived via Kibana index patterns. For example, for an out of the box metricbeat index, this reduces the list of passed on fields from 3000+ to ~120 fields. Previously, the page would freeze on load for tens of seconds and would freeze again on every rerender. With the applied update, the page loads almost instantly again and remains responsive. Note this fix of reducing available fields is only applied to the data grid preview component. All fields are still available to create the configuration in the UI for groups and aggregations. These UI components, e.g. the virtualized dropdowns, can handle large lists of fields.
walterra
added a commit
that referenced
this pull request
Apr 28, 2021
… with massive amounts of fields. (#98259) (#98571) The transform wizard can become very slow when used with indices with e.g. 1000+ fields. This PR fixes it by prefetching 500 random documents to create a list of populated/used fields and passes those on to the data grid component instead of all available fields from the list derived via Kibana index patterns. For example, for an out of the box metricbeat index, this reduces the list of passed on fields from 3000+ to ~120 fields. Previously, the page would freeze on load for tens of seconds and would freeze again on every rerender. With the applied update, the page loads almost instantly again and remains responsive. Note this fix of reducing available fields is only applied to the data grid preview component. All fields are still available to create the configuration in the UI for groups and aggregations. These UI components, e.g. the virtualized dropdowns, can handle large lists of fields.
walterra
added a commit
that referenced
this pull request
Apr 28, 2021
… with massive amounts of fields. (#98259) (#98572) The transform wizard can become very slow when used with indices with e.g. 1000+ fields. This PR fixes it by prefetching 500 random documents to create a list of populated/used fields and passes those on to the data grid component instead of all available fields from the list derived via Kibana index patterns. For example, for an out of the box metricbeat index, this reduces the list of passed on fields from 3000+ to ~120 fields. Previously, the page would freeze on load for tens of seconds and would freeze again on every rerender. With the applied update, the page loads almost instantly again and remains responsive. Note this fix of reducing available fields is only applied to the data grid preview component. All fields are still available to create the configuration in the UI for groups and aggregations. These UI components, e.g. the virtualized dropdowns, can handle large lists of fields.
walterra
added a commit
to walterra/kibana
that referenced
this pull request
Apr 28, 2021
… with massive amounts of fields. (elastic#98259) The transform wizard can become very slow when used with indices with e.g. 1000+ fields. This PR fixes it by prefetching 500 random documents to create a list of populated/used fields and passes those on to the data grid component instead of all available fields from the list derived via Kibana index patterns. For example, for an out of the box metricbeat index, this reduces the list of passed on fields from 3000+ to ~120 fields. Previously, the page would freeze on load for tens of seconds and would freeze again on every rerender. With the applied update, the page loads almost instantly again and remains responsive. Note this fix of reducing available fields is only applied to the data grid preview component. All fields are still available to create the configuration in the UI for groups and aggregations. These UI components, e.g. the virtualized dropdowns, can handle large lists of fields.
1 task
walterra
added a commit
that referenced
this pull request
Apr 28, 2021
… with massive amounts of fields. (#98259) (#98573) The transform wizard can become very slow when used with indices with e.g. 1000+ fields. This PR fixes it by prefetching 500 random documents to create a list of populated/used fields and passes those on to the data grid component instead of all available fields from the list derived via Kibana index patterns. For example, for an out of the box metricbeat index, this reduces the list of passed on fields from 3000+ to ~120 fields. Previously, the page would freeze on load for tens of seconds and would freeze again on every rerender. With the applied update, the page loads almost instantly again and remains responsive. Note this fix of reducing available fields is only applied to the data grid preview component. All fields are still available to create the configuration in the UI for groups and aggregations. These UI components, e.g. the virtualized dropdowns, can handle large lists of fields.
1 task
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #78590.
Inspired by the approach taken by Kibana Discover + Lens.
The transform wizard can become very slow when used with indices with e.g. 1000+ fields.
This PR fixes it by prefetching 500 random documents to create a list of populated/used fields and passes those on to the data grid component instead of all available fields from the list derived via Kibana index patterns.
For example, for an out of the box metricbeat index, this reduces the list of passed on fields from 3000+ to ~120 fields. Previously, the page would freeze on load for tens of seconds and would freeze again on every rerender. With the applied update, the page loads almost instantly again and remains responsive.
Note this fix of reducing available fields is only applied to the data grid preview component. All fields are still available to create the configuration in the UI for groups and aggregations. These UI components, e.g. the virtualized dropdowns, can handle large lists of fields.
Checklist