QueuedRetry removal, metrics around spans and services

[ibawt]

with the queued retry removal / deprecation I lost metrics around the spans accepted and the service it was.

I like to use this as an overall indicator of who is producing what and to what cluster.
Not sure if it should be post / pre sampling, I think it would be fine either as a processor or integrated into exporterhelper

eg

for each resource span bundle {
   inc( span_count, service_name )
}

[bogdandrutu]

I would suggest having a "TelemetryProcessor" to expose this internal metrics. The main reason:

• Avoid current duplication - few metrics like this are duplicated.
• Allow users to install this anywhere in the pipeline - before sampling, after sampling, etc.
• Allow customization.

[bogdandrutu]

@ibawt are you willing to help with this issue?

[ibawt]

Either me or a new team member, I guess you can assign to me.

[bogdandrutu]

Suggest to start from the current queued_retry processor and copy only the telemetry recording part.

[flands]

Hey folks -- any updates on this? I would like to argue this is a significant regression and something that needs to be addressed ASAP. From what I can tell, if you have an exporter destination that returns 500 or even 429 status code you basically have no visibility. Metrics look good, zpages look good, the only clue you get is a log message which does not align with metric and zpages data:

2020-11-12T02:54:40.402Z	ERROR	exporterhelper/queued_retry.go:149	Dropping data because sending_queue is full. Try increasing queue_size.	{"component_kind": "exporter", "component_type": "sapm", "component_name": "sapm", "dropped_items": 1

[dmitryax]

Hi @ibawt , do you still plan to resume your work on this? Otherwise can take it over

[ibawt]

not doing anything related to this anymore cheers. My old company has a version of it running production for a while now but it's so trivial there's no point to code sharing.

[dmitryax]

Thanks @ibawt! I'll work on this issue then

[dmitryax]

Metrics lost due to removed queued_retry processor are restored in these PRs:

• Report metric about current size of the exporter retry queue #2858
• Add telemetry for dropped data due to exporter sending queue overflow #3328

The following metrics added:

• exporter/queue_size
• exporter/enqueue_failed_metric_points
• exporter/enqueue_failed_spans
• exporter/enqueue_failed_log_records