[Security Solutions] Fixes performance issue of track_total_hits not having an effect when used with search strategy #91068
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FrankHassanabad
changed the title
XavierM
approved these changes
Feb 11, 2021
💚 Build SucceededMetrics [docs]
To update your PR or re-run it, just comment with: |
dplumlee
approved these changes
Feb 11, 2021
FrankHassanabad
added a commit
that referenced
this pull request
Feb 11, 2021
…arch strategy (#91068) (#91076) ## Summary Moves `track_total_hits` from body messages of our queries into the params section of our queries. Several of our `track_total_hits: false` were not taking effect and instead were being set to `track_total_hits: true` when being executed within the Kibana search strategy vs. previously when they were regular Elasticsearch queries and always took effect. When teams port over their searches to the search strategies provided by Kibana, they are required to move any and all `track_total_hits` from their `body` sections of their code into the `params` part of their code. The reason for this is that the search strategy maintains a backwards compatibility with earlier versions of searches before Elasticsearch introduced the `track_total_hits`. However, the code does not detect if you put the `track_total_hits` in your body, it only checks the params section and forces it to `true` if it is not found in the params section. If the search strategy does not see a `track_total_hits` within the params section of the query, it will force add one and that one will override any within the body of the query. For example, if you had a `track_total_hits` in your body and not in the params section, then search strategy would execute the query like so: ```ts GET someindex-*/_search?track_total_hits=true { // some query here "track_total_hits": false } ``` The forced parameter of `?track_total_hits=true` overrides the `track_total_hits: false` within the body of your query regardless of what the `track_total_hits` is set to and you always get the true. This bug has existed since 7.10.0 when we ported over queries to search strategy. You can see the code which sets this parameter if you do not here for master, 7.11, 7.10: https://github.com/elastic/kibana/blob/master/src/plugins/data/server/search/es_search/request_utils.ts#L31 https://github.com/elastic/kibana/blob/7.11/src/plugins/data/server/search/es_search/request_utils.ts#L31 https://github.com/elastic/kibana/blob/7.10/src/plugins/data/server/search/es_search/get_default_search_params.ts#L42 Comments about the behavior from 7.10: #75728 (review) When running this code you can open dev tools and inspect the data and now notice when the total hits does not get set vs. before when it was getting set: before fix where total shows up for queries with `track_total_hits` in the body: <img width="1370" alt="event_view_before" src="https://user-images.githubusercontent.com/1151048/107594265-bfc92e80-6bce-11eb-8526-8a9aa24e7b3a.png"> after fix where total no longer shows up for queries with `track_total_hits` moved to the params section: <img width="1309" alt="event_view_after" src="https://user-images.githubusercontent.com/1151048/107594274-c5bf0f80-6bce-11eb-9d8e-698ed430c953.png"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Feb 11, 2021
* master: (44 commits) [APM] Add experimental support for Data Streams (elastic#89650) [Search Session] Control "Kibana / Search Sessions" management section by privileges (elastic#90818) [Lens] Median as default function (elastic#90952) Implement custom global header banner (elastic#87438) [Fleet] Reduce permissions. (elastic#90302) Update dependency @elastic/charts to v24.5.1 (elastic#89822) [Create index pattern] Can't create single character index without wildcard (elastic#90919) [ts/build_ts_refs] add support for --clean flag (elastic#91060) Don't clean when running e2e tests (elastic#91057) Fixes track_total_hits in the body not having an effect when using search strategy (elastic#91068) [Security Solution][Detections] Adds list plugin Saved Objects to Security feature privilege (elastic#90895) Removing the code plugin entirely for 8.0 (elastic#77940) chore(NA): move the instruction to remove yarn global bazelisk package into the first place on install bazel tools (elastic#91026) [jest/ci] remove max-old-space-size override to use 4gb default (elastic#91020) [Fleet] Restrict integration changes for managed policies (elastic#90675) [CI] Fix auto-backport condditions so that it doesn't trigger for other labels (elastic#91042) [DOCS] Uses variable to refer to query profiler (elastic#90976) [App Search] Relevance Tuning logic listeners (elastic#89461) [Metrics UI] Fix saving/loading saved views from URL (elastic#90216) Limit cardinality of transaction.name (elastic#90955) ...
timroes
added
the
Team: SecuritySolution
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Moves
track_total_hitsfrom body messages of our queries into the params section of our queries.Several of our
track_total_hits: falsewere not taking effect and instead were being set totrack_total_hits: truewhen being executed within the Kibana search strategy vs. previously when they were regular Elasticsearch queries and always took effect.When teams port over their searches to the search strategies provided by Kibana, they are required to move any and all
track_total_hitsfrom theirbodysections of their code into theparamspart of their code. The reason for this is that the search strategy maintains a backwards compatibility with earlier versions of searches before Elasticsearch introduced thetrack_total_hits. However, the code does not detect if you put thetrack_total_hitsin your body, it only checks the params section and forces it totrueif it is not found in the params section.If the search strategy does not see a
track_total_hitswithin the params section of the query, it will force add one and that one will override any within the body of the query. For example, if you had atrack_total_hitsin your body and not in the params section, then search strategy would execute the query like so:The forced parameter of
?track_total_hits=trueoverrides thetrack_total_hits: falsewithin the body of your query regardless of what thetrack_total_hitsis set to and you always get the true. This bug has existed since 7.10.0 when we ported over queries to search strategy.You can see the code which sets this parameter if you do not here for master, 7.11, 7.10:
https://github.com/elastic/kibana/blob/master/src/plugins/data/server/search/es_search/request_utils.ts#L31
https://github.com/elastic/kibana/blob/7.11/src/plugins/data/server/search/es_search/request_utils.ts#L31
https://github.com/elastic/kibana/blob/7.10/src/plugins/data/server/search/es_search/get_default_search_params.ts#L42
Comments about the behavior from 7.10:
#75728 (review)
When running this code you can open dev tools and inspect the data and now notice when the total hits does not get set vs. before when it was getting set:
before fix where total shows up for queries with
track_total_hitsin the body:after fix where total no longer shows up for queries with
track_total_hitsmoved to the params section:Checklist