[EPIC] Investigation Bias

[matejminar]

Problem

Our current biases work well generically in order to collect a baseline of samples, because they prioritise retaining data that is valuable for any customer at any given time. But there are (1) needs specific to the individual customer we simply can’t know and (2) temporary scenarios, where very specific data becomes more important (which we again can’t predict). Today a customer has no way to collect this data when needed.

Proposed solution

When troubleshooting a particular issue, a customer can specify certain criteria and for a limited amount of time (e.g. 1h) or until a certain amount of samples have been gathered (e.g. 100) and we keep all events that match that criteria.

Note: we either only allow users to set criteria that can guarantee full traces, or make them aware that we will only collect transactions and this can break traces.

Phase 1: Proof of Concept

Planned Completion: 2023-09-25

Tasks

Preview Give feedback

1. Create a feature flag #55685
   +0
   
2. Endpoint - Create rule from query #55679
   +0
   
3. Investigation rules : Rule exists endpoint #55680
   +0
   
4. Create rules in project config #55682
   +0
   
5. Design: UI when there are few/no results #55683
   +0
   
6. Have a way to programmatically trigger investigation bias without calling endpoint #57264
   +0
   
7. UI for creating investigation rules #56864
   +0
   
8. Bug bash session #57265
   +0
   

Phase 2: Internal

Planned Completion: 2023-10-23

Tasks

Preview Give feedback

1. [Ingest Investigation] Reservoir of samples #54449
   3 of 3
   +0
   
2. Change the rule to use reservoir #57263
   +0
   
3. Amplitude instrumentation #57755
   +0
   
4. notify people that expressed interest in this feature #58986
   +0
   
5. Create Unit Tests for InvestigationRuleCreation component #57543
   +0
   
6. Do not show investigation bias if you don't have enough samples on page 2+ #57752
   +0
   
7. Set rules to a fixed period of 2 days #57756
   +0
   

Phase 3: EA

Preview Give feedback

1. Send email once we collect enough samples #56840
   +0
   
2. Update docs
3. Decide if the CreateInvestigationRule is public #55914
   +0
   
4. Cleanup expired rules #56841
   +0
   
5. Consider alternative designs #57276
   +0
6. Update link to docs in UI

[aellett]

We’ve got a use case that’s sort of adjacent to this. We occasionally receive a report that “Executive E saw odd behavior in the app” and the request is to understand why.

We’ve got an assortment of manually instrumented performance transactions that should capture some or all of the user’s journey. However, we find that we’re not able to reliably find any transactions associated with that user even though we have 1) a unique user identifier in the form of a tag set on the scope of that session and 2) the sampling rate set at 100%.

It doesn’t sound to me like this approach would address our issue, as you’d have to know that something was about to happen before specifying the criteria. Is there a better way to achieve what we’re looking for? If we sample at 100%, is there a way to guarantee that we’ll be able to find a user’s performance transactions in recent history?

[hubertdeng123]

@aellett Thanks for writing in. Could you file a separate issue for this where we can address your concern? This way we can properly respond and keep track of your use case, since this issue is an epic used for project tracking purposes.

[aellett]

Thanks, filed #55706 for this.

[Jesse-Box]

@matejminar @RaduW

Do we have any understanding as to how people typically investigate performance problems on Sentry? If so, can you elaborate on the most common workflows?