Need a detailed description of the rate_sampling policy under the tailSampling processor

[DraegoG]

Component(s)

processor/tailsampling

Describe the issue you're reporting

While trying to go through the README of the Tail Sampling Processor, I was not clear that what is the use case of the rate_sampling policy and how it works internally.
I found a discussion started here: #1797, mentioning the same problem but does not see a conclusion there.

Can someone please explain the use of the rate_sampling policy(we can update the same in the README as well)?

Also I feel there is a need to update the README as well with explaining the use of the policies in more detail and what are the values supported for other parameters for e.g. in the ottl_condition policy there is a parameter called error_mode but nowhere its explained what are the supported values for that parameter.

[github-actions]

Pinging code owners:

• processor/tailsampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/tailsampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

@jpkrohling please review?

[jpkrohling]

rate_limiting serves to limit the number of spans that should be sampled. It takes only the number of spans that were sent at the current time frame (each second at the moment) and drops everything once that limit has been reached. You can use this together with the AND sampler, so that you can define that you want 10k spans/second for a specific service, while retaining all other spans for less noisy services.

    policies: # this acts as an "or": if at least one policy has a SAMPLE decision, we sample
      [
        {
          name: rate-limited-tenant,
          type: and,
          and:
            {
              and_sub_policy:
                [
                  {
                    name: tenant-noisy,
                    type: string_attribute,
                    string_attribute: { key: tenant, values: ["noisy"] },
                  },
                  {
                    name: rate-limit-noisy,
                    type: rate_limiting,
                    rate_limiting: { spans_per_second: 100 },
                  },
                ],
            },
        },
      ]

I'm adding a summary of this to the readme as well.

[mugli]

@jpkrohling One question regarding rate_limiting that wasn't clear to me from the doc (or reading the unit tests). Is there any unintended effect from using this policy at top level (without and policy)?

I assume it'll limit the maximum rate of spans from after evaluating all other top level policies. But will it also do the reverse, like take the decision to sample a (potentially uninteresting) trace, because current limit is lower when all the other policies decide not to sample a trace?