[APM] Instrumentation for ES queries

[dgieselaar]

Currently, when we use the ES client to search for apm events, the name of the ES span is the ES endpoint plus the indices that are being searched, e.g. Elasticsearch: POST /apm-*/_search . This is not very helpful when looking at the trace waterfall, because all ES spans created by APM searches look the same.

What we can do is name our queries, e.g. require a query name when calling apmEventClient.search, and then create a span before calling esClient.search. This will create a parent-child relationship between our span and the spans created by the ES client, and the waterfall will look like this:

Currently it looks like a parent-child relationship is only created when the esClient.search call is not in the same frame as the one where the span is created. One thing that seems to be working is adding an await Promise.resolve() statement before. (see elastic/apm-agent-nodejs#1889).

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[dgieselaar]

Btw, maybe this should be handled in the Kibana ES client itself. I figure it might be useful for everyone. @elastic/kibana-core

[TinaHeiligers]

@dgieselaar we can add this request to the list of improvements in #78869.
cc @joshdover

[trentm]

not in the same frame

A slight terminology correction. It has to be in a separate async operation (i.e. a separate task on the libuv event loop), which is different than a callstack frame (a separate function call).

[dgieselaar]

Slightly gross, but we can also use the stack trace to get a name so we don't have to name our searches:

That requires some knowledge about which specific function in the stack trace would be a good name for the search, which we can probably do for APM, but likely not in a way that it works for all plugins.

We can use the parsed stack trace from the Span (again, maybe gross), or use a library to parse it.

Sample set from browsing around (transaction name followed by span name):

GET /api/apm/ui_filters/environments: getEnvironments 
 GET /api/apm/ui_filters/local_filters/services: /lib/ui_filters/local_ui_filters/index 
 GET /api/apm/services: getAgentNames 
 GET /api/apm/services: getTransactionDurationAverages 
 GET /api/apm/services: getEnvironments 
 GET /api/apm/services: getLegacyDataStatus 
 GET /api/apm/services: getTransactionRates 
 GET /api/apm/services: getTransactionErrorRates 
 GET /api/apm/ui_filters/local_filters/traces: /lib/ui_filters/local_ui_filters/index 
 GET /api/apm/traces: getAverages 
 GET /api/apm/traces: getSums 
 GET /api/apm/traces: getCounts 
 GET /api/apm/service-map: getServicesData 
 GET /api/apm/service-map: getTraceSampleIds 
 GET /api/apm/service-map: fetchServicePathsFromTraceIds 
 GET /api/apm/service-map/service/?: getCpuStats 
 GET /api/apm/service-map/service/?: getAvgMemoryUsage 
 GET /api/apm/service-map/service/?: getTransactionStats 
 GET /api/apm/service-map/service/?: getErrorRate 
 GET /api/apm/services/?/metadata/icons: getServiceMetadataIcons 
 GET /api/apm/services/?/transactions/charts/error_rate: getErrorRate 
 GET /api/apm/services/?/dependencies: getMetrics 
 GET /api/apm/services/?/dependencies: getDestinationMap 
 GET /api/apm/services/?/agent_name: getServiceAgentName 
 GET /api/apm/services/?/annotation/search: getDerivedServiceAnnotations 
 GET /api/apm/services/?/transaction_types: getServiceTransactionTypes 
 GET /api/apm/services/?/transactions/charts/latency: searchLatency 
 GET /api/apm/services/?/transactions/groups/overview: getTransactionGroupsForPage 
 GET /api/apm/services/?/throughput: fetcher 
 GET /api/apm/services/?/error_groups: getServiceErrorGroups 
 GET /api/apm/services/?/transactions/groups/overview: getTimeseriesDataForTransactionGroups 
 GET /api/apm/services/?/service_overview_instances: getServiceInstanceTransactionStats 
 GET /api/apm/services/?/service_overview_instances: getServiceInstanceSystemMetricStats 
 GET /api/apm/services/?/transaction/charts/breakdown: getTransactionBreakdown 
 GET /api/apm/services/?/transactions/groups: getSums 
 GET /api/apm/services/?/transactions/groups: getAverages 
 GET /api/apm/services/?/transactions/groups: getCounts 
 GET /api/apm/services/?/transactions/groups: getPercentiles 
 GET /api/apm/ui_filters/local_filters/transactionGroups: /lib/ui_filters/local_ui_filters/index 
 GET /api/apm/services/?/transactions/charts/throughput: searchThroughput 
 GET /api/apm/services/?/errors/distribution: getBuckets 
 GET /api/apm/services/?/errors: getErrorGroups 
 GET /api/apm/ui_filters/local_filters/errorGroups: /lib/ui_filters/local_ui_filters/index 
 GET /api/apm/services/?/errors/?: getErrorGroupSample 
 GET /api/apm/services/?/errors/?: getTransaction

[alex-fedotyev]

For my understanding, will those values be assigned to span.name?

@dgieselaar - do you think this improvement is generic enough to work well for ES node.js client?
Or is it specific to how Kibana queries ES?

[dgieselaar]

@alex-fedotyev I've been thinking about a slightly different approach that might be more generally useful: using the stack traces of all the items in the waterfall to get the most unique call site in a stack trace for a specific span and use that to display as a name. If I have some time I'll see if I can get a POC up and running just to see what the results are like, I don't think it would be that hard.