[Search] Add telemetry for data plugin search service

[lukasolson]

Summary

Relies on #69333.
Partially addresses #62964.

Adds telemetry for data plugin search service. It currently tracks the following events:

• When a query times out
• When a group of queries is explicitly cancelled by the user
• When the popup for running queries beyond timeout is shown
• When the popup for long running queries is dismissed
• When a user runs a group of queries beyond timeout

To do

• Add unit tests

Resolved

[lizozom]

LGTM

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 5dc7b16

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
data	204	-1	205

miscellaneous assets size

id	value	diff	baseline
data	397.3KB	+2.2KB	395.2KB
dataEnhanced	78.4KB	+171.0B	78.2KB
upgradeAssistant	22.5KB	-24.0B	22.6KB
total	-	+2.3KB	-

page load bundle size

id	value	diff	baseline
data	1.4MB	+8.7KB	1.4MB
dataEnhanced	177.7KB	+420.0B	177.3KB
total	-	+9.1KB	-

History

• 💔 Build #61772 failed c6cd551
• 💔 Build #61756 failed b874ddc
• 💔 Build #61746 failed d53c86f
• 💔 Build #61719 failed b5c1262
• 💔 Build #61498 failed a734ae2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[chrisronline]

Hey folks,

Looking at this code, it seems the response will also include a total field, even though it does not show up in the interface. I think this is happening because we are seeing a structure like this in the monitoring parity tests:

"search": {
        "successCount": 0,
        "total": 0,
        "errorCount": 0,
        "averageDuration": null
      },

Does this seem right?

[TinaHeiligers]

@lukasolson Are you planning to monitor anything via the telemetry service?

[lizozom]

@chrisonline, is that a problem?
@TinaHeiligers yes, that's the intention of this PR. Why?

[TinaHeiligers]

@lizozom I ask because I don't see an issue in the Telemetry repo asking to update the mapping for this issue. ui-metrics don't have a schema and we need issues requesting an update to the mapping with the shape of the data we are shipping from Kibana.

[chrisronline]

@lizozom It's causing an issue with monitoring parity tests between legacy and metricbeat collection. I don't think it is related to the different collection methods, but rather an issue of the data being available sometimes and not always. Ideally, if the total field is returned and indexed when search-telemetry data is available, it should also be present when it's not (but default to 0).

[lizozom]

@TinaHeiligers @lukasolson we'll open an issue for that.
@chrisronline not sure what we should be doing: Help us help you! :-)

[chrisronline]

@lizozom Sorry, I did some more digging and I'm not sure this has anything to do with this code. I think it's more of a matter of keys colliding when merging telemetry data, which could be a bigger issue, but I'm still looking into it

[afharo]

This as Usage cast is not true.
_source is returning something like:

         {
            "references": [],
            "updated_at": "2020-08-17T10:53:00.318Z",
            "search-telemetry": {
              "successCount": 2,
              "errorCount": 0,
              "averageDuration": 110
            },
            "type": "search-telemetry"
          }