Add subpolicy metrics for And, Composite policies

[cancub]

Component(s)

processor/tailsampling

Is your feature request related to a problem? Please describe.

I had to comb through the debug logs to determine that my Composite subpolicies weren't being followed as expected. This would be a lot easier if there was an extra level of metrics below
otelcol_processor_tail_sampling_*
which would provide the same statistics about the constituent subpolicies of And and Composite policies.

This also seems like an intuitive next step, since the same logic about wanting metrics about the individual policies also applies to subpolicies.

Describe the solution you'd like

In the same vein that we have a suite of metrics which include the label of policy and sampled, e.g.,

otelcol_processor_tail_sampling_count_traces_sampled{policy="composite-policy",sampled="false",service_instance_id="a63244e2-2969-4010-b1b8-f59c6859012c",service_name="otelcol",service_version=""} 156
otelcol_processor_tail_sampling_count_traces_sampled{policy="composite-policy",sampled="true",service_instance_id="a63244e2-2969-4010-b1b8-f59c6859012c",service_name="otelcol",service_version=""} 2972
otelcol_processor_tail_sampling_sampling_decision_latency_bucket{policy="composite-policy",service_instance_id="a63244e2-2969-4010-b1b8-f59c6859012c",service_name="otelcol",service_version="",le="1"} 2567
otelcol_processor_tail_sampling_sampling_decision_latency_bucket{policy="composite-policy",service_instance_id="a63244e2-2969-4010-b1b8-f59c6859012c",service_name="otelcol",service_version="",le="2"} 2619
otelcol_processor_tail_sampling_sampling_decision_latency_bucket{policy="composite-policy",service_instance_id="a63244e2-2969-4010-b1b8-f59c6859012c",service_name="otelcol",service_version="",le="5"} 2843
...
otelcol_processor_tail_sampling_sampling_decision_latency_bucket{policy="composite-policy",service_instance_id="a63244e2-2969-4010-b1b8-f59c6859012c",service_name="otelcol",service_version="",le="50000"} 3128
otelcol_processor_tail_sampling_sampling_decision_latency_bucket{policy="composite-policy",service_instance_id="a63244e2-2969-4010-b1b8-f59c6859012c",service_name="otelcol",service_version="",le="+Inf"} 3128
otelcol_processor_tail_sampling_sampling_decision_latency_sum{policy="composite-policy",service_instance_id="a63244e2-2969-4010-b1b8-f59c6859012c",service_name="otelcol",service_version=""} 5663
otelcol_processor_tail_sampling_sampling_decision_latency_count{policy="composite-policy",service_instance_id="a63244e2-2969-4010-b1b8-f59c6859012c",service_name="otelcol",service_version=""} 3128

I would like to see extra label of subpolicy added. This could also be a brand new metric, which would allow us to add more details, like in the case of Composite where sometimes the subpolicy actually returned Sampled, but it was over spans-per-second allocation and thus the result was NotSampled.

A new metric feels like the most straight-forward option, but it might take a bit of work, since it would require updating both the And and Composite policies with a new telemetry field like the one we have in the top-level tailSamplingProcessor

In the case of adding an extra label or labels to the existing metrics, this would just be an empty string for the policies without subpolicies.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• processor/tailsampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jamesmoessis]

@cancub would a metric label solve this generally? Because can't you have arbitrarily nested sub-policies i.e. an and inside an and? Let me know if I'm misunderstanding your proposal.

[jamesmoessis]

Would also be interested in hearing @jpkrohling's opinion

[cancub]

@cancub would a metric label solve this generally

Yup, that works.