Longer transactions get dropped since upgrading to Business 1 week ago

[danielwillemsen]

Environment

SaaS (https://sentry.io/)

What are you trying to accomplish?

We're using sentry for performance monitoring of some python jobs. Some of them can be pretty long, and thus a single transaction can be long as well. We'd like to receive these transactions in sentry.

How are you getting stuck?

Previously (before March 25th, which is when we upgraded from the team to business plan), any transactions that contained too many spans would be cut short and upon inspecting the event details we'd see the message: "The next spans are unavailable. You may have exceeded the span limit or need to address missing instrumentation" This was acceptable.

After March 25th, these long transactions no longer show up alltogether. Instead, it looks like they are getting dropped upon ingestion. (See attached image from the "stats" window).

Before March 25th, nothing got dropped. Now some transactions do get dropped.

I have already turned off all retention policy toggles in the "performance" tab of the project settings, but this didn't affect it.

When running the python task locally & sending the transaction to another (non-prod) environment, the transaction was not filtered out.

What could cause the filtering in this case?

Where in the product are you?

Performance

Link

No response

DSN

No response

Version

No response

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-performance for triage ⏲️

[leeandher]

Routing back to support with a request from Revenue team to look into the specifics for this customer and determine why the events are being dropped. Once that root cause is identified and determined to be unexpected behavior; then an engineering team can take a look from there.

[dalnoki]

Hi @danielwillemsen thanks for your patience here! To make sure that we can take a proper look at this issue, would you mind opening a Support Request here? Could you please share your current SDK package name, version and your Sentry.init()? Thanks a lot in advance!

[getsantry]

Routing to @getsentry/product-owners-ingestion-and-filtering for triage ⏲️

[Dav1dde]

It is possible that a very large transaction close to the 1mb limit accepted by a Relay goes over the 1mb threshold when Relay enriches/adds information to the transaction, e.g. add tags or metadata.

The forwarded, now bigger than 1mb, transaction will be dropped by the next upstream Relay because it is too big.

The AM2 plan now enabled more performance features which makes Relay add more information pushing the transaction over that 1mb limit.

We'll take a look and plan to trim the transaction so it does stay within the 1mb limit and get back to a similar behaviour you're describing:

Previously[...] any transactions that contained too many spans would be cut short and upon inspecting the event details we'd see the message: "The next spans are unavailable. [...]" This was acceptable.

[danielwillemsen]

Thanks for the support (through email) and the follow up comments here!

[jjbayer]

@danielwillemsen we enabled trimming for spans now, did the situation improve for you?

[jjbayer]

@danielwillemsen I'm considering this closed from our side, feel free to reopen if the problem persists.