[APM] Incorrect error count in trace header

[sorenlouv]

Introduced in #44274

The number of errors in the trace header doesn't match the number of errors in the actual trace. In the screenshot below there are 4 errors in the trace header but only 2 in the trace.

It may happen that an agent sends up errors that occured in a transaction but the transaction itself is never sent. Likelyhood of this happening is rather small though.

Repro url: https://apm.elstc.co/ovk/app/apm#/services/client/transactions/view?rangeFrom=2019-08-27T21:11:00.000Z&rangeTo=2019-08-27T21:37:30.000Z&refreshPaused=true&refreshInterval=0&traceId=17a2c6b6ed58694d38c7c58f80df51d5&transactionId=d8cfb4f38bed3dce&transactionName=customers&transactionType=page-load&detailTab=timeline&_g=()&flyoutDetailTab=&waterfallItemId=

[elasticmachine]

Pinging @elastic/apm-ui

[formgeist]

@sqren Perhaps unrelated to this issue, but was it discussed to add a link to the Errors overview page with the trace.id value filtered to show all the related errors on that summary badge, 4 errors? I was expecting that I could click it like I can the numbered error counts in the Timeline.

[sorenlouv]

Perhaps unrelated to this issue, but was it discussed to add a link to the Errors overview page with the trace.id value filtered to show all the related errors on that summary badge, 4 errors? I was expecting that I could click it like I can the numbered error counts in the Timeline.

Not unrelated at all. That would actually mitigate this problem but since we are now displaying the error count for the full trace, there could be errors for multiple services. We currently don't have an error overview page that shows errors across multiple services (they are always scoped by a service). This reminds me a little of the talk we had about making service.name an optional filter. Then we could use the existing error overview page without any changes.

[formgeist]

Not unrelated at all. That would actually mitigate this problem but since we are now displaying the error count for the full trace, there could be errors for multiple services. We currently don't have an error overview page that shows errors across multiple services (they are always scoped by a service).

Right, I see how that's a challenge.

This reminds me a little of the talk we had about making service.name an optional filter. Then we could use the existing error overview page without any changes.

But obviously that opens up a whole slurry of navigation changes that we'd need to make. I understand that could potentially change it and add the Service as a filter, but that's not really the ideal experience. I imagine we'd want to move the Errors page out of the Service area, and into its own like Services and Traces. Not sure if that's anything we want to act upon right now.

[sorenlouv]

But obviously that opens up a whole slurry of navigation changes that we'd need to make. I understand that could potentially change it and add the Service as a filter, but that's not really the ideal experience. I imagine we'd want to move the Errors page out of the Service area, and into its own like Services and Traces. Not sure if that's anything we want to act upon right now.

Agree. Let's punt on it for now.