Fixes track_total_hits in the body not having an effect when using se…

…arch strategy (#91068) (#91076)

## Summary

Moves `track_total_hits` from body messages of our queries into the params section of our queries.

Several of our `track_total_hits: false` were not taking effect and instead were being set to `track_total_hits: true` when being executed within the Kibana search strategy vs. previously when they were regular Elasticsearch queries and always took effect.  

When teams port over their searches to the search strategies provided by Kibana, they are required to move any and all `track_total_hits` from their `body` sections of their code into the `params` part of their code. The reason for this is that the search strategy maintains a backwards compatibility with earlier versions of searches before Elasticsearch introduced the `track_total_hits`. However, the code does not detect if you put the `track_total_hits` in your body, it only checks the params section and forces it to `true` if it is not found in the params section.

If the search strategy does not see a `track_total_hits` within the params section of the query, it will force add one and that one will override any within the body of the query. For example, if you had a `track_total_hits` in your body and not in the params section, then search strategy would execute the query like so:

```ts
GET someindex-*/_search?track_total_hits=true
{
  // some query here
  "track_total_hits": false
}
``` 

The forced parameter of `?track_total_hits=true` overrides the `track_total_hits: false` within the body of your query regardless of what the `track_total_hits` is set to and you always get the true. This bug has existed since 7.10.0 when we ported over queries to search strategy.

You can see the code which sets this parameter if you do not here for master, 7.11, 7.10:
https://github.com/elastic/kibana/blob/master/src/plugins/data/server/search/es_search/request_utils.ts#L31
https://github.com/elastic/kibana/blob/7.11/src/plugins/data/server/search/es_search/request_utils.ts#L31
https://github.com/elastic/kibana/blob/7.10/src/plugins/data/server/search/es_search/get_default_search_params.ts#L42

Comments about the behavior from 7.10:
#75728 (review)


When running this code you can open dev tools and inspect the data and now notice when the total hits does not get set vs. before when it was getting set:

before fix where total shows up for queries with `track_total_hits` in the body:
<img width="1370" alt="event_view_before" src="https://user-images.githubusercontent.com/1151048/107594265-bfc92e80-6bce-11eb-8526-8a9aa24e7b3a.png">

after fix where total no longer shows up for queries with `track_total_hits` moved to the params section:
<img width="1309" alt="event_view_after" src="https://user-images.githubusercontent.com/1151048/107594274-c5bf0f80-6bce-11eb-9d8e-698ed430c953.png">

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

x-pack/plugins/security_solution/server/lib/hosts/query.detail_host.dsl.ts

@@ -39,13 +39,13 @@ export const buildHostOverviewQuery = ({
     allowNoIndices: true,
     index: defaultIndex,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       aggregations: {
         ...buildFieldsTermAggregation(esFields.filter((field) => !['@timestamp'].includes(field))),
       },
       query: { bool: { filter } },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/lib/hosts/query.hosts.dsl.ts

@@ -44,6 +44,7 @@ export const buildHostsQuery = ({
     allowNoIndices: true,
     index: defaultIndex,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       ...(!isEmpty(docValueFields) ? { docvalue_fields: docValueFields } : {}),
       aggregations: {
@@ -72,7 +73,6 @@ export const buildHostsQuery = ({
       },
       query: { bool: { filter } },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/lib/hosts/query.last_first_seen_host.dsl.ts

@@ -19,6 +19,7 @@ export const buildLastFirstSeenHostQuery = ({
     allowNoIndices: true,
     index: defaultIndex,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       ...(!isEmpty(docValueFields) ? { docvalue_fields: docValueFields } : {}),
       aggregations: {
@@ -27,7 +28,6 @@ export const buildLastFirstSeenHostQuery = ({
       },
       query: { bool: { filter } },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/all/__mocks__/index.ts

@@ -621,6 +621,7 @@ export const formattedSearchStrategyResponse = {
             'winlogbeat-*',
           ],
           ignoreUnavailable: true,
+          track_total_hits: false,
           body: {
             docvalue_fields: mockOptions.docValueFields,
             aggregations: {
@@ -656,7 +657,6 @@ export const formattedSearchStrategyResponse = {
               },
             },
             size: 0,
-            track_total_hits: false,
           },
         },
         null,
@@ -782,6 +782,7 @@ export const mockBuckets: HostAggEsItem = {
 
 export const expectedDsl = {
   allowNoIndices: true,
+  track_total_hits: false,
   body: {
     aggregations: {
       host_count: { cardinality: { field: 'host.name' } },
@@ -817,7 +818,6 @@ export const expectedDsl = {
     },
     docvalue_fields: mockOptions.docValueFields,
     size: 0,
-    track_total_hits: false,
   },
   ignoreUnavailable: true,
   index: [

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/all/query.all_hosts.dsl.ts

@@ -43,6 +43,7 @@ export const buildHostsQuery = ({
     allowNoIndices: true,
     index: defaultIndex,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       ...(!isEmpty(docValueFields) ? { docvalue_fields: docValueFields } : {}),
       aggregations: {
@@ -71,7 +72,6 @@ export const buildHostsQuery = ({
       },
       query: { bool: { filter } },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/details/__mocks__/index.ts

@@ -1311,6 +1311,7 @@ export const formattedSearchStrategyResponse = {
             'winlogbeat-*',
           ],
           ignoreUnavailable: true,
+          track_total_hits: false,
           body: {
             aggregations: {
               host_architecture: {
@@ -1387,7 +1388,6 @@ export const formattedSearchStrategyResponse = {
               },
             },
             size: 0,
-            track_total_hits: false,
           },
         },
         null,
@@ -1410,6 +1410,7 @@ export const expectedDsl = {
     'winlogbeat-*',
   ],
   ignoreUnavailable: true,
+  track_total_hits: false,
   body: {
     aggregations: {
       host_architecture: {
@@ -1645,6 +1646,5 @@ export const expectedDsl = {
       },
     },
     size: 0,
-    track_total_hits: false,
   },
 };

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/details/query.host_details.dsl.ts

@@ -36,13 +36,13 @@ export const buildHostDetailsQuery = ({
     allowNoIndices: true,
     index: defaultIndex,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       aggregations: {
         ...buildFieldsTermAggregation(esFields.filter((field) => !['@timestamp'].includes(field))),
       },
       query: { bool: { filter } },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/kpi/authentications/query.hosts_kpi_authentications.dsl.ts

@@ -41,6 +41,7 @@ export const buildHostsKpiAuthenticationsQuery = ({
     index: defaultIndex,
     allowNoIndices: true,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       aggs: {
         authentication_success: {
@@ -94,7 +95,6 @@ export const buildHostsKpiAuthenticationsQuery = ({
         },
       },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/kpi/hosts/query.hosts_kpi_hosts.dsl.ts

@@ -30,6 +30,7 @@ export const buildHostsKpiHostsQuery = ({
     index: defaultIndex,
     allowNoIndices: true,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       aggregations: {
         hosts: {
@@ -57,7 +58,6 @@ export const buildHostsKpiHostsQuery = ({
         },
       },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/kpi/unique_ips/query.hosts_kpi_unique_ips.dsl.ts

@@ -30,6 +30,7 @@ export const buildHostsKpiUniqueIpsQuery = ({
     index: defaultIndex,
     allowNoIndices: true,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       aggregations: {
         unique_source_ips: {
@@ -75,7 +76,6 @@ export const buildHostsKpiUniqueIpsQuery = ({
         },
       },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/last_first_seen/__mocks__/index.ts

@@ -69,14 +69,14 @@ export const formattedSearchStrategyResponse = {
             'winlogbeat-*',
           ],
           ignoreUnavailable: true,
+          track_total_hits: false,
           body: {
             aggregations: {
               firstSeen: { min: { field: '@timestamp' } },
               lastSeen: { max: { field: '@timestamp' } },
             },
             query: { bool: { filter: [{ term: { 'host.name': 'siem-kibana' } }] } },
             size: 0,
-            track_total_hits: false,
           },
         },
         null,
@@ -100,13 +100,13 @@ export const expectedDsl = {
     'winlogbeat-*',
   ],
   ignoreUnavailable: true,
+  track_total_hits: false,
   body: {
     aggregations: {
       firstSeen: { min: { field: '@timestamp' } },
       lastSeen: { max: { field: '@timestamp' } },
     },
     query: { bool: { filter: [{ term: { 'host.name': 'siem-kibana' } }] } },
     size: 0,
-    track_total_hits: false,
   },
 };

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/last_first_seen/query.last_first_seen_host.dsl.ts

@@ -20,6 +20,7 @@ export const buildFirstLastSeenHostQuery = ({
     allowNoIndices: true,
     index: defaultIndex,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       ...(!isEmpty(docValueFields) ? { docvalue_fields: docValueFields } : {}),
       aggregations: {
@@ -28,7 +29,6 @@ export const buildFirstLastSeenHostQuery = ({
       },
       query: { bool: { filter } },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/overview/__mocks__/index.ts

@@ -127,6 +127,7 @@ export const formattedSearchStrategyResponse = {
             'winlogbeat-*',
           ],
           ignoreUnavailable: true,
+          track_total_hits: false,
           body: {
             aggregations: {
               auditd_count: { filter: { term: { 'event.module': 'auditd' } } },
@@ -299,7 +300,6 @@ export const formattedSearchStrategyResponse = {
               },
             },
             size: 0,
-            track_total_hits: false,
           },
         },
         null,
@@ -339,6 +339,7 @@ export const expectedDsl = {
     'winlogbeat-*',
   ],
   ignoreUnavailable: true,
+  track_total_hits: false,
   body: {
     aggregations: {
       auditd_count: { filter: { term: { 'event.module': 'auditd' } } },
@@ -511,6 +512,5 @@ export const expectedDsl = {
       },
     },
     size: 0,
-    track_total_hits: false,
   },
 };

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/hosts/overview/query.overview_host.dsl.ts

@@ -31,6 +31,7 @@ export const buildOverviewHostQuery = ({
     allowNoIndices: true,
     index: defaultIndex,
     ignoreUnavailable: true,
+    track_total_hits: false,
     body: {
       aggregations: {
         auditd_count: {
@@ -289,7 +290,6 @@ export const buildOverviewHostQuery = ({
         },
       },
       size: 0,
-      track_total_hits: false,
     },
   };
 

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/matrix_histogram/__mocks__/index.ts

@@ -42,6 +42,7 @@ export const formattedAlertsSearchStrategyResponse: MatrixHistogramStrategyRespo
           ],
           allowNoIndices: true,
           ignoreUnavailable: true,
+          track_total_hits: true,
           body: {
             aggregations: {
               alertsGroup: {
@@ -113,7 +114,6 @@ export const formattedAlertsSearchStrategyResponse: MatrixHistogramStrategyRespo
               },
             },
             size: 0,
-            track_total_hits: true,
           },
         },
         null,
@@ -127,6 +127,7 @@ export const formattedAlertsSearchStrategyResponse: MatrixHistogramStrategyRespo
 
 export const expectedDsl = {
   allowNoIndices: true,
+  track_total_hits: false,
   body: {
     aggregations: {
       host_count: { cardinality: { field: 'host.name' } },
@@ -161,7 +162,6 @@ export const expectedDsl = {
       },
     },
     size: 0,
-    track_total_hits: false,
   },
   ignoreUnavailable: true,
   index: [
@@ -208,6 +208,7 @@ export const formattedAnomaliesSearchStrategyResponse: MatrixHistogramStrategyRe
           ],
           allowNoIndices: true,
           ignoreUnavailable: true,
+          track_total_hits: true,
           body: {
             aggs: {
               anomalyActionGroup: {
@@ -258,7 +259,6 @@ export const formattedAnomaliesSearchStrategyResponse: MatrixHistogramStrategyRe
               },
             },
             size: 0,
-            track_total_hits: true,
           },
         },
         null,
@@ -390,6 +390,7 @@ export const formattedAuthenticationsSearchStrategyResponse: MatrixHistogramStra
           ],
           allowNoIndices: true,
           ignoreUnavailable: true,
+          track_total_hits: true,
           body: {
             aggregations: {
               eventActionGroup: {
@@ -429,7 +430,6 @@ export const formattedAuthenticationsSearchStrategyResponse: MatrixHistogramStra
               },
             },
             size: 0,
-            track_total_hits: true,
           },
         },
         null,
@@ -956,6 +956,7 @@ export const formattedEventsSearchStrategyResponse: MatrixHistogramStrategyRespo
           ],
           allowNoIndices: true,
           ignoreUnavailable: true,
+          track_total_hits: true,
           body: {
             aggregations: {
               eventActionGroup: {
@@ -994,7 +995,6 @@ export const formattedEventsSearchStrategyResponse: MatrixHistogramStrategyRespo
               },
             },
             size: 0,
-            track_total_hits: true,
           },
         },
         null,

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/matrix_histogram/alerts/__mocks__/index.ts

@@ -36,6 +36,7 @@ export const expectedDsl = {
   ],
   allowNoIndices: true,
   ignoreUnavailable: true,
+  track_total_hits: true,
   body: {
     aggregations: {
       alertsGroup: {
@@ -104,6 +105,5 @@ export const expectedDsl = {
       },
     },
     size: 0,
-    track_total_hits: true,
   },
 };

x-pack/plugins/security_solution/server/search_strategy/security_solution/factory/matrix_histogram/alerts/query.alerts_histogram.dsl.ts

@@ -85,6 +85,7 @@ export const buildAlertsHistogramQuery = ({
     index: defaultIndex,
     allowNoIndices: true,
     ignoreUnavailable: true,
+    track_total_hits: true,
     body: {
       aggregations: getHistogramAggregation(),
       query: {
@@ -93,7 +94,6 @@ export const buildAlertsHistogramQuery = ({
         },
       },
       size: 0,
-      track_total_hits: true,
     },
   };