Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

client_sample_rate for auto instrumented Queue spans is always 1 in UI even if sampled at a lower rate. #946

Open
Fwang36 opened this issue Oct 9, 2024 · 9 comments
Assignees

Comments

@Fwang36
Copy link
Member

Fwang36 commented Oct 9, 2024

How do you use Sentry?

Sentry SaaS (sentry.io)

SDK version

4.8.0

Laravel version

11.9

Steps to reproduce

  1. Set up tracesSampler to conditionally sample an auto instrumented Queue span.
      $transactionName = \Illuminate\Support\Str::of($context->getTransactionContext()?->getName())->value();


      if (str_contains($transactionName, LogTestMessage::class)) {

          return 0.1;
      }
      
      return 0.99;
  1. Enable the logger and check the logs to see the sampling occuring.
  1. Search for the event Id in the logs and see that the samping rate is set to 1.

Link to the transaction

Expected result

client_sample_rate should show as .1 like I have set in the tracesSampler

Actual result

client_sample_rate shows as 1 even though I do not have 1 set anywhere in my tracesSampler

@getsantry getsantry bot moved this to Waiting for: Product Owner in GitHub Issues with 👀 3 Oct 9, 2024
@cleptric
Copy link
Member

cleptric commented Oct 9, 2024

Hmm, this works fine for me: https://sentry-sdks.sentry.io/performance/trace/4169ef053f87462c813a181ede76c051

'traces_sampler' => function() {
    return 0.1;
},

Image

@stayallive
Copy link
Collaborator

Can you show a bit more about how exactly the tracesSampler is configured?

I also cannot reproduce and the sample rate is shown correctly as 0.99.

@Fwang36
Copy link
Member Author

Fwang36 commented Oct 9, 2024

Hey sorry, I should of clarified more. For me It only happens on the queue spans. The normal network request spans work fine when I sample them.

I edited the tracesSampler to include a network request span, and the client sample rate is shown properly. Here is the entire tracesSampler -

    public static function tracesSampler(SamplingContext $context): float
    {

        $transactionName = \Illuminate\Support\Str::of($context->getTransactionContext()?->getName())->value();


        if (str_contains($transactionName, LogTestMessage::class)) {

            return 0.1;
        }

        if (str_contains($transactionName, "store-cache")) {
            return 0.98;
        }
        
        return 0.99;
    }

This is the new Queue span that shows a client_sample_rate of 1 instead of 0.1

And this is the new http.server /store-cache network span that correctly shows 0.98

@getsantry getsantry bot moved this to Waiting for: Product Owner in GitHub Issues with 👀 3 Oct 9, 2024
@stayallive
Copy link
Collaborator

Okay, I've setup a database queue like you did and fired a job but still:

Image

I did simplify the traces_sampler to just the following in my config/sentry.php:

    'traces_sampler' => static function() {
        return 0.99;
    },

Wondering what variable we are missing here that causes this.

I am on Laravel 11.16 and using the 4.9.0 of the Laravel SDK, but that has no relevant changes as far as I can see.

@Fwang36
Copy link
Member Author

Fwang36 commented Oct 10, 2024

I changed the traces_sampler to yours and also updated laravel and Sentry. I still see the same thing. Maybe it's because of how I am triggering the transaction? I tried to replicate a users set up who also ran into this issue. I'm running php artisan queue:work to start the queue worker and then php artisan app:dispatch-log-test-message to trigger the transaction.

I made a screen recording of exactly how I'm doing it if this helps at all

Screen.Recording.2024-10-10.at.12.39.13.AM.mov

@cleptric
Copy link
Member

There might be something off when we transform the data into an envelope. I have no other ideas tbh. cc @stayallive

@stayallive
Copy link
Collaborator

I am so puzzled right now. I really appreciate all the info you are giving is. And seeing this I think I'm replicating it correctly on my end but with a different result so trying to figure out where the missing link is.

Are you able to show us how you dispatch the job and maybe even a little of what the job is doing (is it doing anything "special" like a HTTP call or calling a route in the application etc.).

Again, appreciate the help with figuring this out because I think there is a bug indeed. It's just very specific 🥲

@cleptric
Copy link
Member

One thought I just had is, that it looks like we apply the parent sampling decision we propagate to the queue job with higher priority than the traces sampler, hence it's always 1.

@stayallive
Copy link
Collaborator

This is a good one to test indeed, will need to just try some stuff. Def something not right here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

5 participants