Add child_spans for Sidekiq Queue instrumentation #2403

frederikspang · 2024-09-16T13:48:29Z

Implements #2322 for Sidekiq.

Description

Adds support for Sentry Queues page (With predefined span op names)

https://docs.sentry.io/platforms/ruby/guides/sidekiq/tracing/instrumentation/custom-instrumentation/queues-module/

sentry-sidekiq/lib/sentry/sidekiq/sentry_context_middleware.rb

rwojsznis · 2024-10-02T10:59:54Z

Oh wow, exactly what I was looking for 😅 🙌

can we somehow help to get this shipped? 💪

solnic

Thanks for working on this! 🙇🏻 Would you be able to add some basic specs for this too?

frederikspang · 2024-10-17T11:55:51Z

@solnic I've added two basic specs.
Latency is a bit difficult, without introducing some timecop kind of tool. Which we do have in sentry-ruby, so we could just introduce it here if we wanted to.

frederikspang · 2024-10-17T18:00:17Z

@solnic After looking at these, I think queue.process may actually replace op: queue.sidekiq instead of being a child span.
They will always be redundant - But we should set origin or some other field to sidekiq, description or name maybe, to differentiate the processor. (ie. using another processor alongside Sidekiq - or doing queries in Sentry UI across applications)

nateberkopec · 2024-10-21T01:39:22Z

Looks like it needs a changelog too

sentry-sidekiq/lib/sentry/sidekiq/sentry_context_middleware.rb

codecov · 2024-10-21T20:53:02Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.16%. Comparing base (27d7384) to head (5e000c6).
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2403   +/-   ##
=======================================
  Coverage   98.15%   98.16%           
=======================================
  Files         126      126           
  Lines        4728     4741   +13     
=======================================
+ Hits         4641     4654   +13     
  Misses         87       87

Components	Coverage Δ
sentry-ruby	`98.51% <ø> (ø)`
sentry-rails	`97.05% <ø> (ø)`
sentry-sidekiq	`97.63% <100.00%> (+0.15%)`	⬆️
sentry-resque	`92.85% <ø> (ø)`
sentry-delayed_job	`95.65% <ø> (ø)`
sentry-opentelemetry	`99.31% <ø> (ø)`

Files with missing lines	Coverage Δ
...iq/lib/sentry/sidekiq/sentry_context_middleware.rb	`98.21% <100.00%> (+0.53%)`	⬆️

solnic · 2024-10-25T13:14:06Z

@solnic After looking at these, I think queue.process may actually replace op: queue.sidekiq instead of being a child span.
They will always be redundant

But we should set origin or some other field to sidekiq, description or name maybe, to differentiate the processor. (ie. using another processor alongside Sidekiq - or doing queries in Sentry UI across applications)

I've been testing this out and I don't see queue.sidekiq/queue.process. This is what I see instead:

solnic · 2024-10-25T13:15:45Z

Latency is a bit difficult, without introducing some timecop kind of tool. Which we do have in sentry-ruby, so we could just introduce it here if we wanted to.

Thanks for adding the specs. I think it would be good to use timecop there actually.

sentry-sidekiq/lib/sentry/sidekiq/sentry_context_middleware.rb

frederikspang · 2024-10-25T13:22:43Z

@solnic After looking at these, I think queue.process may actually replace op: queue.sidekiq instead of being a child span.
They will always be redundant

But we should set origin or some other field to sidekiq, description or name maybe, to differentiate the processor. (ie. using another processor alongside Sidekiq - or doing queries in Sentry UI across applications)

I've been testing this out and I don't see queue.sidekiq/queue.process. This is what I see instead:

You're right! I changed that after my comment, before your review here:

cd3a1df

I'll have a look at the other comments as soon as possible.

frederikspang · 2024-10-27T12:50:05Z

@solnic I have updated PR with feedback adjusted!

Edit: I see specs are broken. I'll look into that as well!

frederikspang · 2024-10-28T09:57:59Z

@solnic The specs failing seems related to sidekiq/sidekiq#6430 - Was able to reproduce locally, after clearing Lock file and bundle install again.

Somewhat unrelated to this PR, although it's noisy.

solnic

Great job thank you!

solnic · 2024-10-28T16:48:04Z

@solnic The specs failing seems related to sidekiq/sidekiq#6430 - Was able to reproduce locally, after clearing Lock file and bundle install again.

Somewhat unrelated to this PR, although it's noisy.

Please update to the latest master. I just checked locally and it passes. Once CI is green I'll merge it in.

frederikspang · 2024-10-28T22:40:09Z

@solnic Done!

nateberkopec · 2024-10-31T00:13:36Z

🙌 Thanks everyone! This is such a great feature.

Linuus · 2025-01-24T08:53:14Z

sentry-sidekiq/spec/sentry/sidekiq_spec.rb

@@ -223,7 +223,7 @@ def retry_last_failed_job
      expect(transaction.contexts.dig(:trace, :trace_id)).to be_a(String)
      expect(transaction.contexts.dig(:trace, :span_id)).to be_a(String)
      expect(transaction.contexts.dig(:trace, :status)).to eq("ok")
-      expect(transaction.contexts.dig(:trace, :op)).to eq("queue.sidekiq")


@frederikspang Did this break sampling? In the docs you show that you can do

case op when /sidekiq/ 0.01 [...]

But this is always set to queue.process now so we are sampling a lot more than we should which is costing us money...

Was this intentional or am I misunderstanding something?

That is very much not intentional. I will submit a PR for docs as well. I was unaware of this example being there.

Yeah we've been running out of performance traces more often recently and were confused what had happened. And then I found that our sampler was not running correctly :-/

Also, this is not all about the docs. It's like a public API and can't just be changed like this without proper deprecations etc. It must be costing your customers a lot of money in both extra Sentry charges as well as debugging time.

I get what you're saying - Except it's not my customers. I've done this as OSS contribution to help the Sentry Ruby SDK conform to the guidelines and "predefined operations" as defined by the Sentry core system - ie making the Queues page work properly for Sidekiq jobs.

Deprecating and semantic versioning of what goes into what release version is very not much my job.

Aha I thought you worked for Sentry :)
Sorry if my tone was harsh, I didn’t mean to offend. I know mistakes can be made.
My message was more directed to Sentry the business.

Also the feature is great and many thanks for implementing it! 🙏

Hey @Linuus I'm one of the maintainers and I approved this PR. It didn't occur to me that transaction's op should be treated as a public interface here. Thanks for bringing this up and I'm very sorry that it caused such issues for you. I updated both the CHANGELOG and the release page with a warning that explains this change.

frederikspang force-pushed the feature/sidekiq-queues-childspan branch 2 times, most recently from cd35fcf to 787436b Compare September 16, 2024 16:02

nateberkopec suggested changes Sep 16, 2024

View reviewed changes

sentry-sidekiq/lib/sentry/sidekiq/sentry_context_middleware.rb Outdated Show resolved Hide resolved

sentry-sidekiq/lib/sentry/sidekiq/sentry_context_middleware.rb Outdated Show resolved Hide resolved

frederikspang force-pushed the feature/sidekiq-queues-childspan branch 3 times, most recently from ac46c1b to 4e505f6 Compare September 17, 2024 09:07

nateberkopec approved these changes Sep 23, 2024

View reviewed changes

solnic reviewed Oct 17, 2024

View reviewed changes

frederikspang mentioned this pull request Oct 17, 2024

Jobs created by Sidekiq-cron shouldn’t propagate trace header #2391

Closed

nateberkopec reviewed Oct 21, 2024

View reviewed changes

sentry-sidekiq/lib/sentry/sidekiq/sentry_context_middleware.rb Outdated Show resolved Hide resolved

frederikspang force-pushed the feature/sidekiq-queues-childspan branch from 04750fa to af385ba Compare October 21, 2024 20:51

solnic requested changes Oct 25, 2024

View reviewed changes

frederikspang force-pushed the feature/sidekiq-queues-childspan branch 2 times, most recently from 9aaf80b to 671dd65 Compare October 28, 2024 09:52

frederikspang mentioned this pull request Oct 28, 2024

Reset trace and transaction for sidekiq-cron #2446

Merged

solnic approved these changes Oct 28, 2024

View reviewed changes

frederikspang added 3 commits October 28, 2024 23:29

Add childspan for Sidekiq Queue instrumentation

1f432da

Add basic specs

1db7b37

Remove unneeded exception alloc

d0e1c32

frederikspang added 5 commits October 28, 2024 23:29

Add CHANGELOG entry

4770b3d

Set root op instead of nested spans

930627e

Adjust feedback

ee20ff5

Remove timecop_delay from options hash

09429a9

Fix 1 day

265307a

frederikspang force-pushed the feature/sidekiq-queues-childspan branch from d3a7219 to 265307a Compare October 28, 2024 22:29

frederikspang added 2 commits October 28, 2024 23:31

Fix 1.day

ea1fa45

Dont use .day helper

5e000c6

solnic merged commit 9446a30 into getsentry:master Oct 29, 2024
140 of 141 checks passed

frederikspang deleted the feature/sidekiq-queues-childspan branch October 29, 2024 13:35

Linuus reviewed Jan 24, 2025

View reviewed changes

frederikspang mentioned this pull request Jan 24, 2025

Adjust for sentry-ruby changes getsentry/sentry-docs#12455

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add child_spans for Sidekiq Queue instrumentation #2403

Add child_spans for Sidekiq Queue instrumentation #2403

frederikspang commented Sep 16, 2024 •

edited

Loading

rwojsznis commented Oct 2, 2024

solnic left a comment

frederikspang commented Oct 17, 2024

frederikspang commented Oct 17, 2024

nateberkopec commented Oct 21, 2024

codecov bot commented Oct 21, 2024 •

edited

Loading

solnic commented Oct 25, 2024

solnic commented Oct 25, 2024

frederikspang commented Oct 25, 2024

frederikspang commented Oct 27, 2024 •

edited

Loading

frederikspang commented Oct 28, 2024

solnic left a comment

solnic commented Oct 28, 2024

frederikspang commented Oct 28, 2024

nateberkopec commented Oct 31, 2024

Linuus Jan 24, 2025

frederikspang Jan 24, 2025

Linuus Jan 24, 2025

Linuus Jan 24, 2025

frederikspang Jan 24, 2025

Linuus Jan 24, 2025

solnic Jan 27, 2025

Add child_spans for Sidekiq Queue instrumentation #2403

Add child_spans for Sidekiq Queue instrumentation #2403

Conversation

frederikspang commented Sep 16, 2024 • edited Loading

Description

rwojsznis commented Oct 2, 2024

solnic left a comment

Choose a reason for hiding this comment

frederikspang commented Oct 17, 2024

frederikspang commented Oct 17, 2024

nateberkopec commented Oct 21, 2024

codecov bot commented Oct 21, 2024 • edited Loading

Codecov Report

solnic commented Oct 25, 2024

solnic commented Oct 25, 2024

frederikspang commented Oct 25, 2024

frederikspang commented Oct 27, 2024 • edited Loading

frederikspang commented Oct 28, 2024

solnic left a comment

Choose a reason for hiding this comment

solnic commented Oct 28, 2024

frederikspang commented Oct 28, 2024

nateberkopec commented Oct 31, 2024

Linuus Jan 24, 2025

Choose a reason for hiding this comment

frederikspang Jan 24, 2025

Choose a reason for hiding this comment

Linuus Jan 24, 2025

Choose a reason for hiding this comment

Linuus Jan 24, 2025

Choose a reason for hiding this comment

frederikspang Jan 24, 2025

Choose a reason for hiding this comment

Linuus Jan 24, 2025

Choose a reason for hiding this comment

solnic Jan 27, 2025

Choose a reason for hiding this comment

frederikspang commented Sep 16, 2024 •

edited

Loading

codecov bot commented Oct 21, 2024 •

edited

Loading

frederikspang commented Oct 27, 2024 •

edited

Loading