Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Discover][Main] Split single query into 2 queries for faster results #104818

Merged

Conversation

kertal
Copy link
Member

@kertal kertal commented Jul 8, 2021

Summary

This PR splits the single query for documents in Discover main view into 2 queries sent to ES.

  1. Query for documents
  2. Query for total hits & Query for chart (if needed, which is usually the case)

The purpose of this PR is to present results to the user as quick as possible, if e.g. the query for the chart takes a bit longer the user should already be able to view the documents and vice versa.

There are now 4 instead of 1 BehaviorSubjects the UI subscribes to get messages about the state of data fetching

// Observable to be notified about the progress of the overall data fetching
// When 1 of the 2 data fetches succeeds,`FetchStatus.PARTIAL` is sent, and the UI switches from displaying the loading screen, to display total hits, chart and document table (1 remaining loading state are displayed in UI).
export type DataMain$ = BehaviorSubject<DataMainMsg>;

// Observable to be notified about the progress of the document data fetching
export type DataDocuments$ = BehaviorSubject<DataDocumentsMsg>;

// Observable to be notified about the progress of the total hits data fetching
export type DataTotalHits$ = BehaviorSubject<DataTotalHitsMsg>;

// Observable to be notified about the progress of the chart data fetching
export type DataCharts$ = BehaviorSubject<DataChartsMessage>;

Changes in UI

When data fetching is started and no data was fetched previously, there are no changes, it looks like this:

Bildschirmfoto 2021-07-07 um 08 33 00

When the first of the 2 requests is returned, and it contains data, the spinners of the remaining request are displayed (in the screen all requests are still loading, but it's just mocked):

Bildschirmfoto 2021-07-07 um 08 33 00

When the first of the 2 request is returned, and it contains no data/documents, the following screen is displayed initially, without waiting for the other requests to be finished

Bildschirmfoto 2021-07-07 um 08 49 46

Requests sent to ES

Documents

{
          "body": {
            "size": 500,
            "sort": [
              {
                "timestamp": {
                  "order": "desc",
                  "unmapped_type": "boolean"
                }
              }
            ],
            "fields": [
              {
                "field": "*",
                "include_unmapped": "true"
              },
              {
                "field": "timestamp",
                "format": "strict_date_optional_time"
              }
            ],
            "version": false,
            "script_fields": {},
            "stored_fields": [
              "*"
            ],
            "runtime_mappings": {
              "hour_of_day": {
                "type": "long",
                "script": {
                  "source": "emit(doc['timestamp'].value.hourOfDay);"
                }
              }
            },
            "_source": false,
            "query": {
              "bool": {
                "must": [],
                "filter": [
                  {
                    "range": {
                      "timestamp": {
                        "gte": "2021-07-07T13:29:48.665Z",
                        "lte": "2021-07-07T13:44:48.665Z",
                        "format": "strict_date_optional_time"
                      }
                    }
                  }
                ],
                "should": [],
                "must_not": []
              }
            },
            "highlight": {
              "pre_tags": [
                "@kibana-highlighted-field@"
              ],
              "post_tags": [
                "@/kibana-highlighted-field@"
              ],
              "fields": {
                "*": {}
              },
              "fragment_size": 2147483647
            }
          },
          "track_total_hits": false,
          "preference": 1625665487687
        }

Total hits

{
         "request":{
            "params":{
               "index":"logstash*",
               "body":{
                  "size":0,
                  "fields":[],
                  "script_fields":{
                     
                  },
                  "stored_fields":[
                     "*"
                  ],
                  "runtime_mappings":{
                     
                  },
                  "_source":{
                     "excludes":[
                        
                     ]
                  },
                  "query":{
                     "bool":{
                        "must":[
                           
                        ],
                        "filter":[
                           {
                              "range":{
                                 "@timestamp":{
                                    "gte":"2021-07-22T07:28:14.364Z",
                                    "lte":"2021-07-29T07:28:14.364Z",
                                    "format":"strict_date_optional_time"
                                 }
                              }
                           }
                        ],
                        "should":[
                           
                        ],
                        "must_not":[
                           
                        ]
                     }
                  }
               },
               "track_total_hits":true,
               "preference":1627543690175
            }
         }

Total Hits and Chart

{
               "index":"logstash*",
               "body":{
                  "size":0,
                  "aggs":{
                     "2":{
                        "date_histogram":{
                           "field":"@timestamp",
                           "fixed_interval":"3h",
                           "time_zone":"Europe/Vienna",
                           "min_doc_count":1
                        }
                     }
                  },
                  "fields":[
                     {
                        "field":"@timestamp",
                        "format":"date_time"
                     },
                     {
                        "field":"relatedContent.article:modified_time",
                        "format":"date_time"
                     },
                     {
                        "field":"relatedContent.article:published_time",
                        "format":"date_time"
                     },
                     {
                        "field":"utc_time",
                        "format":"date_time"
                     }
                  ],
                  "script_fields":{
                     
                  },
                  "stored_fields":[
                     "*"
                  ],
                  "runtime_mappings":{
                     
                  },
                  "_source":{
                     "excludes":[
                        
                     ]
                  },
                  "query":{
                     "bool":{
                        "must":[
                           
                        ],
                        "filter":[
                           {
                              "range":{
                                 "@timestamp":{
                                    "gte":"2021-07-22T07:31:13.769Z",
                                    "lte":"2021-07-29T07:31:13.769Z",
                                    "format":"strict_date_optional_time"
                                 }
                              }
                           }
                        ],
                        "should":[
                           
                        ],
                        "must_not":[
                           
                        ]
                     }
                  }
               },
               "track_total_hits":true,
               "preference":1627543869067
            }
         }

More changes in this PR:

  • The version field is added to the JSON tab of the doc viewer Bildschirmfoto 2021-08-02 um 08 35 14

Out of scope of this PR:

When displaying the cart was toggled to a state when the chart is displayed, we currently request all the data, we could now switch to just request the chart data, which would be a nice follow up PR. (Same when changing the chart interval in UI)

Resolves #69134
Part of #101041
Also fixes parts of #69049
Fixes #37519 (since we have an aria-live on the result count now)

Checklist

…b.com:kertal/kibana into kertal-pr-2020-06-08-discover-split-queries-2
Copy link
Contributor

@andreadelrio andreadelrio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sass changes LGTM.

@kertal
Copy link
Member Author

kertal commented Aug 2, 2021

@elasticmachine merge upstream

@kertal kertal requested a review from timroes August 2, 2021 06:37
const [from, to] = x;
this.props.timefilterUpdateHandler({ from, to });
const formatXValue = (val: string) => {
const xAxisFormat = chartData!.xAxisFormat.params!.pattern;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const xAxisFormat = chartData!.xAxisFormat.params!.pattern;
const xAxisFormat = chartData.xAxisFormat.params!.pattern;

Copy link
Contributor

@timroes timroes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the pre version change, all the queries seem to work as expected. Reviewed code, LGTM

@timroes
Copy link
Contributor

timroes commented Aug 2, 2021

@elasticmachine merge upstream

(github failure on doc build)

Copy link
Contributor

@Dosant Dosant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code review src/plugins/data/common/search/search_source/mocks.ts

@kertal
Copy link
Member Author

kertal commented Aug 2, 2021

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
discover 406 421 +15

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
discover 495.9KB 509.2KB +13.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
charts 88.5KB 88.7KB +222.0B
discover 93.4KB 93.5KB +11.0B
total +233.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kertal

@kertal kertal added the auto-backport Deprecated - use backport:version if exact versions are needed label Aug 2, 2021
@kertal kertal merged commit 47f5f81 into elastic:master Aug 2, 2021
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Aug 2, 2021
@kibanamachine
Copy link
Contributor

💚 Backport successful

Status Branch Result
7.x

This backport PR will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Aug 2, 2021
…#104818) (#107392)

Co-authored-by: Tim Roes <tim.roes@elastic.co>

Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Tim Roes <tim.roes@elastic.co>
streamich pushed a commit to vadimkibana/kibana that referenced this pull request Aug 8, 2021
@kertal kertal added the Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. label May 23, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@kertal kertal removed the Team:Visualizations Visualization editors, elastic-charts and infrastructure label May 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed Feature:Discover Discover Application performance release_note:enhancement Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. v7.15.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Split Discover query into three (Accessibility) Discover Page Load
6 participants