[Autocomplete] Support useTimeFilter option #81515

lizozom · 2020-10-22T17:36:45Z

Summary

Temporarily patches #46054
Related to #15887

This PR applies the active timefilter onto the autocomplete requests, following a request from the Elasticsearch team, to fix the problem mentioned in #46054.

This solution is temporary, as Elasticsearch plans to introduce a new and more efficient autocomplete API. Open question: Will the new ES API use the entire timerange? Or will it be also limited to the timerange?

While this approach reduces the time needed to return autocomplete results for shorter time ranges, when the time range is long or if an index is not time based, it approaches the same performance of the current behavior on master.

It's worth mentioning here, that even on master frozen indices are not being queried for autocomplete suggestions.

Changes with this approach

Unavailable results

On master, autocomplete results are constructed from the entire time range. In this branch only a subset of the results is returned to the user. This is emphasized by the fact that the current UI\UX, doesn't allow the user to see the current time range, while editing the query.

If an organization is unhappy with this behavior they can turn off the autocomplete:useTimerange advanced setting to revert to the previous behavior.

master

New behavior

Inconsistent results between components \ solutions

After much deliberation with @elastic-jb , we decided to keep the drop-downs in the FilterBar populated with the full list of results (i.e. not filter by time) while the Query autocomplete suggestions will be filtered. So a user might see different suggestions in both places.

Relative time range memoization

Relative time ranges will be rounded to the nearest minute, in order to avoid memoization issues.

Impact on other solutions

The autocomplete suggestions API is used by multiple consumers:

APM
Beats
Infra
Ingest
Uptime
Maps

I think it's worth hearing their opinions about this change as well, and whether it has any implications on the solutions.

Release Notes

This PR applies the active timefilter onto the autocomplete requests, to fix performance issues mentioned in #46054. This behavior will be enabled by default, but can be turned off by disabling the autocomplete:useTimerange advanced settings.

This solution is temporary, as Elasticsearch plans to introduce a new and more efficient autocomplete API.

…r-in-autocomplete

lukasolson · 2020-10-29T15:15:24Z

src/plugins/data/public/autocomplete/providers/value_suggestion_provider.ts


-function resolver(title: string, field: IFieldType, query: string, boolFilter: any) {
+function resolver(title: string, field: IFieldType, query: string, filters: any[]) {


Can this be filters: Filter[] in this whole file?

These are not filters of type Filter, but an array of whatever type BoolQuery.filter is ( src/plugins/data/server/search/es_search/elasticsearch.ts).

We could fix that, but I wouldn't want to do it in this PR.

…r-in-autocomplete

kobelb · 2020-10-30T18:12:35Z

I think it's reasonable for us to add an option for the time filter to be applied to the autocomplete suggestions. However, if we've been ignoring the time filter for autocomplete for the entirety of 7.x, or since its implementation mid 7.x, changing this behavior is a breaking change. Is my understanding correct that we've had enough users experience issues with the performance/health of their ES clusters that we should make this breaking change in a minor?

@lizozom, I really like the detailed analysis that you performed in #46054 (comment). Is my understanding correct that without these changes, we're in the "TERMS w/totals, wo/timerange, wo/sort" column? Is there a column that represents the performance with only these changes? I don't think it's absolutely pertinent for us to benchmark this if we have the expert guidance from the ES team, I'm primarily asking out of curiosity.

lizozom · 2020-10-30T18:39:03Z

@kobelb the nearest benchmark would be the second column. Adding sorting didn't improve the results much.
I should mention that these benchmarks were done without terminateAfter, to avoid a cut off.

Since this is a serious problem, I guess fixing it in a minor would make sense, but I do think we need to at least find a way to resolve the inconsistencies I mentioned above as well as the memoization problem.

…r-in-autocomplete

stacey-gammon · 2020-11-03T17:43:39Z

Thanks so much for being so explicit with the corner cases of this approach Liza! @elastic-jb - do you know what the original reason was for removing the time filter from auto complete results? I hope we aren't repeating history, if there were legit reasons to remove it.

I think there are two ways people use autocomplete. One is for fast type-ahead. I start typing, I see what I want, I click it or press tab. The other reason I personally use type ahead is for searching. For example, I'm in Discover and I have an issues-* index. I want to filter on issues created by a certain person but I can't quite remember their name. I just know it started with a J. This issue was created awhile ago, because this person doesn't work at Elastic anymore. I start to type my query hoping that auto complete will provide me with the rest of their name, because it's one of those "I'll know it when I see it". I type author: j. If my time filter happens to be too short, I won't see their name. I don't think "oh I don't see their name because the issue they created isn't in my time range", instead I think something is broken. My first thought was to filter - I don't even know what the time range is. That will be my second thought if no results show up. "Oh, I don't see anything here, let me expand the time range"

I wouldn't expect the results to be filtered by time range and I would think something is broken. If we want to filter by time range to optimize for the fast typeahead use case, I think that's okay, but I think ideally it will be made clear in the UI. Like "results limited to time x - y".

Also to the point about:

. This is emphasized by the fact that the current UI\UX, doesn't allow the user to see the current time range, while editing the query.

💯 I literally was just trying this out and thought my index pattern didn't have a time range and was confused why I wasn't seeing any results. I didn't realize it was limited to last 15 minutes.

Update autocomplete tests and docs

lizozom · 2020-11-04T15:57:10Z

Copying here the original PR text, as I update it with the chosen solutions:

Potential issues with this approach

Unavailable results

On master, autocomplete results are constructed from the entire time range. In this branch only a subset of the results is returned to the user. This is emphasized by the fact that the current UI\UX, doesn't allow the user to see the current time range, while editing the query.

master

New behavior

Inconsistent results between various screens

This may occur under a few different cases, with similar effects:

If there are two index patterns for the same index - one without a time field.
When switching from Discover to the Context (single document) view
When using autocomplete from the query bar vs. from the filter bar vs. from a custom field selector.

For example, if we restrict the QueryBar input with the time range, but choose to have all suggestions available in the FilterBar, the user will see different suggestions in both.

Alternatively, if we choose to apply the time range filtering to both the QueryBar and FilterBar, we would still see different results in the Context view in Discover, as it doesn't use the time range at all.

Relative time range memoization

On master autocomplete results are always memoized. On this branch relative time ranges or any time range that includes now, are not memoized.

master

New behavior

shahzad31 · 2020-11-04T16:37:16Z

uptime already does this by passing bool , range filter https://github.com/elastic/kibana/blob/master/x-pack/plugins/uptime/public/components/overview/kuery_bar/kuery_bar.tsx#L108

src/plugins/data/public/autocomplete/providers/value_suggestion_provider.ts

useTimeFilter in uptime

x-pack/plugins/uptime/public/components/overview/kuery_bar/kuery_bar.tsx

…r-in-autocomplete

lizozom · 2020-11-05T14:30:32Z

Autocomplete short term fix

shahzad31 · 2020-11-06T13:05:32Z

Code looks good to me for uptime, just need to verify functionality in uptime perhaps @Kerry350 or @andrewvc can do it whilst i am on PTO.

lizozom · 2020-11-09T09:50:23Z

@elasticmachine merge upstream

…r-in-autocomplete

…ibana into fix/timefilter-in-autocomplete

…r-in-autocomplete

src/plugins/data/common/constants.ts

Co-authored-by: Lukas Olson <olson.lukas@gmail.com>

lizozom · 2020-11-10T13:11:26Z

@elasticmachine merge upstream

…r-in-autocomplete

…ibana into fix/timefilter-in-autocomplete

kibanamachine · 2020-11-10T16:18:19Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 08058f0

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	3.1MB	3.1MB	-195.0B
`data`	235.0KB	235.0KB	+19.0B
`uptime`	983.1KB	983.0KB	-140.0B
total			-316.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	963.2KB	964.2KB	+1.0KB
`dataEnhanced`	28.3KB	28.3KB	+52.0B
total			+1.1KB

History

💔 Build #87063 failed 582f512
💚 Build #86693 succeeded 302329d
💚 Build #86673 succeeded d38c5bd
💚 Build #86337 succeeded 812e22b
💚 Build #86096 succeeded de3941b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* pass timefilter to autocomplete * ignoreTimeRange advanced setting * Show all results in filter bar autocomplete * Round timerange to minute for autocomplete memoization * Change useTimeFilter param name Update autocomplete tests and docs * Fix maps test useTimeFilter in uptime * docs * useTimeRange in apm * remove date validation * Update src/plugins/data/common/constants.ts Co-authored-by: Lukas Olson <olson.lukas@gmail.com> * docs Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Lukas Olson <olson.lukas@gmail.com>

kibanamachine · 2020-11-12T18:54:39Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

kibanamachine · 2020-11-13T19:04:39Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

* pass timefilter to autocomplete * ignoreTimeRange advanced setting * Show all results in filter bar autocomplete * Round timerange to minute for autocomplete memoization * Change useTimeFilter param name Update autocomplete tests and docs * Fix maps test useTimeFilter in uptime * docs * useTimeRange in apm * remove date validation * Update src/plugins/data/common/constants.ts Co-authored-by: Lukas Olson <olson.lukas@gmail.com> * docs Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Lukas Olson <olson.lukas@gmail.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Lukas Olson <olson.lukas@gmail.com>

pass timefilter to autocomplete

a8831cd

lizozom self-assigned this Oct 22, 2020

Merge branch 'master' of github.com:elastic/kibana into fix/timefilte…

3362863

…r-in-autocomplete

lizozom added Team:AppArch v8.0.0 labels Oct 29, 2020

lukasolson reviewed Oct 29, 2020

View reviewed changes

Liza K added 3 commits October 29, 2020 18:54

ignoreTimeRange advanced setting

f8b1be9

Merge branch 'master' of github.com:elastic/kibana into fix/timefilte…

d870eac

…r-in-autocomplete

Show all results in filter bar autocomplete

0e674b7

timroes added v7.11.0 and removed v7.11 labels Oct 30, 2020

lizozom requested review from kobelb and stacey-gammon October 30, 2020 16:45

Liza K added 2 commits November 3, 2020 15:23

Merge branch 'master' of github.com:elastic/kibana into fix/timefilte…

07e3d6a

…r-in-autocomplete

Round timerange to minute for autocomplete memoization

17084b4

Change useTimeFilter param name

895c6cc

Update autocomplete tests and docs

lizozom mentioned this pull request Nov 4, 2020

Remove kibana yml settings workaround in input control vis #67464

Closed

sorenlouv reviewed Nov 4, 2020

View reviewed changes

src/plugins/data/public/autocomplete/providers/value_suggestion_provider.ts Show resolved Hide resolved

Fix maps test

87f0357

useTimeFilter in uptime

shahzad31 reviewed Nov 5, 2020

View reviewed changes

x-pack/plugins/uptime/public/components/overview/kuery_bar/kuery_bar.tsx Show resolved Hide resolved

Liza K added 2 commits November 5, 2020 14:53

Merge branch 'master' of github.com:elastic/kibana into fix/timefilte…

c1af5ad

…r-in-autocomplete

docs

de3941b

lizozom marked this pull request as ready for review November 5, 2020 14:50

Kerry350 self-requested a review November 6, 2020 13:30

Kerry350 approved these changes Nov 6, 2020

View reviewed changes

lizozom requested a review from lukasolson November 6, 2020 16:46

kibanamachine and others added 4 commits November 9, 2020 04:50

Merge branch 'master' into fix/timefilter-in-autocomplete

d38c5bd

Merge branch 'master' of github.com:elastic/kibana into fix/timefilte…

e28ae05

…r-in-autocomplete

Merge branch 'fix/timefilter-in-autocomplete' of github.com:lizozom/k…

302329d

…ibana into fix/timefilter-in-autocomplete

Merge branch 'master' of github.com:elastic/kibana into fix/timefilte…

f702bfd

…r-in-autocomplete

stacey-gammon removed their request for review November 9, 2020 15:38

lukasolson approved these changes Nov 10, 2020

View reviewed changes

src/plugins/data/common/constants.ts Outdated Show resolved Hide resolved

Update src/plugins/data/common/constants.ts

775632c

Co-authored-by: Lukas Olson <olson.lukas@gmail.com>

kibanamachine and others added 4 commits November 10, 2020 08:11

Merge branch 'master' into fix/timefilter-in-autocomplete

582f512

Merge branch 'master' of github.com:elastic/kibana into fix/timefilte…

c512b19

…r-in-autocomplete

Merge branch 'fix/timefilter-in-autocomplete' of github.com:lizozom/k…

9af6054

…ibana into fix/timefilter-in-autocomplete

docs

08058f0

lizozom merged commit 8842e9b into elastic:master Nov 10, 2020

lizozom mentioned this pull request Nov 10, 2020

[7.x] [Autocomplete] Support useTimeFilter option (#81515) #83090

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 12, 2020

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 15, 2020

cauemarcondes mentioned this pull request Feb 16, 2021

[APM] Kql Search Bar suggests values outside the selected time range #90166

Closed

lukasolson mentioned this pull request Mar 3, 2021

Filter bar value suggestions - filter by time #15887

Closed

Dosant mentioned this pull request Apr 20, 2021

Filter value suggestions takes long time to load #97512

Closed

lizozom mentioned this pull request Jul 20, 2021

Autocomplete for search ( and filter panel) is not working for cross cluster search index patterns #104515

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Autocomplete] Support useTimeFilter option #81515

[Autocomplete] Support useTimeFilter option #81515

lizozom commented Oct 22, 2020 •

edited

Loading

lukasolson Oct 29, 2020

lizozom Nov 6, 2020

kobelb commented Oct 30, 2020 •

edited

Loading

lizozom commented Oct 30, 2020

stacey-gammon commented Nov 3, 2020 •

edited

Loading

lizozom commented Nov 4, 2020

shahzad31 commented Nov 4, 2020

lizozom commented Nov 5, 2020

shahzad31 commented Nov 6, 2020

lizozom commented Nov 9, 2020

lizozom commented Nov 10, 2020

kibanamachine commented Nov 10, 2020

kibanamachine commented Nov 12, 2020

kibanamachine commented Nov 13, 2020


		function resolver(title: string, field: IFieldType, query: string, boolFilter: any) {
		function resolver(title: string, field: IFieldType, query: string, filters: any[]) {

[Autocomplete] Support useTimeFilter option #81515

[Autocomplete] Support useTimeFilter option #81515

Conversation

lizozom commented Oct 22, 2020 • edited Loading

Summary

Changes with this approach

Unavailable results

Inconsistent results between components \ solutions

Relative time range memoization

Impact on other solutions

Release Notes

lukasolson Oct 29, 2020

Choose a reason for hiding this comment

lizozom Nov 6, 2020

Choose a reason for hiding this comment

kobelb commented Oct 30, 2020 • edited Loading

lizozom commented Oct 30, 2020

stacey-gammon commented Nov 3, 2020 • edited Loading

lizozom commented Nov 4, 2020

Potential issues with this approach

Unavailable results

Inconsistent results between various screens

Relative time range memoization

shahzad31 commented Nov 4, 2020

lizozom commented Nov 5, 2020

shahzad31 commented Nov 6, 2020

lizozom commented Nov 9, 2020

lizozom commented Nov 10, 2020

kibanamachine commented Nov 10, 2020

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

History

kibanamachine commented Nov 12, 2020

kibanamachine commented Nov 13, 2020

lizozom commented Oct 22, 2020 •

edited

Loading

kobelb commented Oct 30, 2020 •

edited

Loading

stacey-gammon commented Nov 3, 2020 •

edited

Loading