Tag queue name and jid on sidekiq events

[st0012]

[ivanovv]

Hey @st0012

Can you clarify one thing – why did you guys decide to add jid to tags and not to the context?

From Sentry docs:

Tags:

We’ll automatically index all tags for an event, as well as the frequency and the last time that Sentry has seen a tag. We also keep track of the number of distinct tags and can assist you in determining hotspots for various issues.

Contexts:

Custom contexts allow you to attach arbitrary data to an event. Often, this context is shared among any issue captured in its lifecycle. You cannot search these, but they are viewable on the issue page

We have 128,741,632 jobs processed by Sidekiq, that means 128,741,632 unique tag values. And I'm sure there are much much bigger numbers out there for systems processing millions of jobs per day.

Is there even a difference between Custom contexts and Tags for Sentry from the perf POV? Or in billing from the customer POV?

[st0012]

@ivanovv using tags doesn't affect billing. and the purpose is to help search events associated with particular jobs easier. we've been tagging request_id (also unique per event) for web applications for a while, which I found useful for searching events and I haven't seen any issues related to that.

the job id is still present in the Additional Data section like before so you don't really lose anything from this change.

[ivanovv]

@st0012 thanks a lot for the clarification.

It looks like Sentry own documentation is a bit confusing – the docs imply that tags are somewhat expensive from the perf point of view and for random (mostly uniq) values that we don't need search on it is better to use custom contexts. But then in your own integrations you put things like jid and request_id inside tags.

That's just my 2c, maybe I'm overthinking it too much

[st0012]

you're welcome 🙂
I do think jid needs search support though. with jid being searchable, you can easily get a jid from sidekiq web and use it to search related exceptions in Sentry. when if it's not searchable, mapping a job with its exceptions directly was much harder I think?

[ivanovv]

I do think jid needs search support though. with jid being searchable, you can easily get a jid from sidekiq web and use it to search related exceptions in Sentry. when if it's not searchable, mapping a job with its exceptions directly was much harder I think?

This is the usual Sidekiq log

2021-02-15T14:13:33.745Z 15733 TID-22b5j5 PusherWorker JID-dce5864533171bbec94491f8 INFO: start
2021-02-15T14:13:33.947Z 15733 TID-22b5j5 PusherWorker JID-dce5864533171bbec94491f8 INFO: done: 0.202 sec

Knowing the jid won't provide you much info unless you add more logging, and when you do so the jid is no longer useful.

At least that's my personal perspective from the last 2-3 years of using Sidekiq.
When we were just starting, (7 years ago) I used to care about jids more and sometimes use the pristine Sidekiq log to connect things together, but additional logging (and in case of Sentry – using tags and contexts) is the correct answer here, IMO.

[ivanovv]

Also, one thing I was wrong about – we only send jid in case the job fails, so it won't be millions of jids.

Finally, for retries, the jid being searchable is a great thing because it will allow to see the stack trace easily.

[sandstrom]

Just gotten started with Sentry. Overall very happy!

But JID and request ID as tags doesn't make sense to me, at least not in some views (two examples below).

Perhaps you should introduce a third category, or a property on tags, to distinguish high cardinality properties?

In short, for any high cardinality tab, skip them in these two views (examples below) and probably a few more places.

These two can be marked high-cardinality in your SDK, and you could also allow custom-added tags to be marked as high_cardinality = true for similar treatment.