Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ref(seer grouping): Use frame tallies for excess frame check #82414

Merged

Conversation

lobsterkatie
Copy link
Member

@lobsterkatie lobsterkatie commented Dec 19, 2024

This is the first step in simplifying the way we apply the stacktrace length filter to events sent to Seer. It puts in place the new machinery, but does not yet remove the old machinery, which will happen in a follow-up PR.

In the meantime, though, even though the old code is still there, the new code should supplant it, since its check comes before the existing one - meaning anything which should get filtered should be flagged by the new check and shouldn't make it through to the existing check. I used different names for the metrics so that we can make sure that's the case before we remove the existing code.

Key changes:

  • A new has_too_many_contributing_frames to be used in all the places we send events to Seer (ingest, backfill, similar issues tab), which collects a grouping.similarity.frame_count_filter metric, regardless of (and tagged with) the referrer.
  • A new ingest-specific _has_too_many_contributing_frames wrapper for the above, which also records a did_call_seer metric.
  • In both the backfill and similar issues tab endpoint code, getting grouping_info is now split into two steps, first getting the variants and then getting grouping info from the variants, so that the variants can be fed to has_too_many_contributing_frames.
  • All get_stacktrace_string tests whose names included some variation of "too many fames" have been renamed, so as not to confuse them with the tests of the new helper, which are added in a separate class at the bottom of the module.

Note that one of the tests is marked as a skip, because in the process of doing this I found what might be a bug in our grouping logic. (TL;DR, frames which are marked -app +group by a stacktrace rule might not be behaving as expected... depending on what you think is expected.) In any case, for now the test is skipped, and once we resolve the stacktrace rule question, we can ether turn it back on or change it.

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Dec 19, 2024
Copy link

codecov bot commented Dec 19, 2024

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
23363 3 23360 214
View the top 3 failed tests by shortest run time
tests.sentry.grouping.ingest.test_seer.TestMaybeCheckSeerForMatchingGroupHash::test_too_many_only_system_frames_maybe_check_seer_for_matching_group_hash
Stack Traces | 2.34s run time
#x1B[1m#x1B[.../grouping/ingest/test_seer.py#x1B[0m:555: in test_too_many_only_system_frames_maybe_check_seer_for_matching_group_hash
    mock_metrics.incr.assert_any_call(
#x1B[1m#x1B[.../hostedtoolcache/Python/3.12.6.../x64/lib/python3.12/unittest/mock.py#x1B[0m:1018: in assert_any_call
    raise AssertionError(
#x1B[1m#x1B[31mE   AssertionError: incr('grouping.similarity.over_threshold_only_system_frames', sample_rate=1.0, tags={'platform': 'java', 'referrer': 'ingest'}) call not found#x1B[0m
tests.sentry.grouping.ingest.test_seer.ShouldCallSeerTest::test_obeys_excessive_frame_check
Stack Traces | 3.09s run time
#x1B[1m#x1B[.../grouping/ingest/test_seer.py#x1B[0m:189: in test_obeys_excessive_frame_check
    assert should_call_seer_for_grouping(self.event, self.variants) is expected_result
#x1B[1m#x1B[31mE   AssertionError: assert False is True#x1B[0m
#x1B[1m#x1B[31mE    +  where False = should_call_seer_for_grouping(<sentry.eventstore.models.Event at 0x7f2296b75fa0: event_id=11212012123120120415201309082013>, {'app': <ComponentVariant None (component)> contributes=False (in-app), 'system': <ComponentVariant 'dd6a093101475be1dc30ab7540439eba' (component)> contributes=True (exception stack-trace)})#x1B[0m
#x1B[1m#x1B[31mE    +    where <sentry.eventstore.models.Event at 0x7f2296b75fa0: event_id=11212012123120120415201309082013> = <test_seer.ShouldCallSeerTest testMethod=test_obeys_excessive_frame_check>.event#x1B[0m
#x1B[1m#x1B[31mE    +    and   {'app': <ComponentVariant None (component)> contributes=False (in-app), 'system': <ComponentVariant 'dd6a093101475be1dc30ab7540439eba' (component)> contributes=True (exception stack-trace)} = <test_seer.ShouldCallSeerTest testMethod=test_obeys_excessive_frame_check>.variants#x1B[0m
tests.sentry.tasks.test_backfill_seer_grouping_records.TestBackfillSeerGroupingRecords::test_lookup_group_data_stacktrace_bulk_invalid_stacktrace_exception
Stack Traces | 7.47s run time
#x1B[1m#x1B[.../sentry/tasks/test_backfill_seer_grouping_records.py#x1B[0m:416: in test_lookup_group_data_stacktrace_bulk_invalid_stacktrace_exception
    mock_metrics.incr.assert_called_with(
#x1B[1m#x1B[.../hostedtoolcache/Python/3.12.6.../x64/lib/python3.12/unittest/mock.py#x1B[0m:947: in assert_called_with
    raise AssertionError(_error_message()) from cause
#x1B[1m#x1B[31mE   AssertionError: expected call not found.#x1B[0m
#x1B[1m#x1B[31mE   Expected: incr('grouping.similarity.over_threshold_only_system_frames', sample_rate=1.0, tags={'platform': 'java', 'referrer': 'backfill'})#x1B[0m
#x1B[1m#x1B[31mE     Actual: incr('grouping.similarity.frame_count_filter', sample_rate=1.0, tags={'referrer': 'backfill', 'platform': 'java', 'stacktrace_type': 'system', 'outcome': 'block'})#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

@lobsterkatie lobsterkatie merged commit ecbfbcf into master Dec 20, 2024
50 checks passed
@lobsterkatie lobsterkatie deleted the kmclb-use-tallies-for-excessive-frame-seer-filter branch December 20, 2024 17:44
andrewshie-sentry pushed a commit that referenced this pull request Jan 2, 2025
This is the first step in simplifying the way we apply the stacktrace length filter to events sent to Seer. It puts in place the new machinery, but does not yet remove the old machinery, which will happen in a follow-up PR. 

In the meantime, though, even though the old code is still there, the new code should supplant it, since its check comes before the existing one - meaning anything which should get filtered should be flagged by the new check and shouldn't make it through to the existing check. I used different names for the metrics so that we can make sure that's the case before we remove the existing code.

Key changes:

- A new `has_too_many_contributing_frames` to be used in all the places we send events to Seer (ingest, backfill, similar issues tab), which collects a `grouping.similarity.frame_count_filter` metric, regardless of (and tagged with) the referrer.
- A new ingest-specific `_has_too_many_contributing_frames` wrapper for the above, which also records a `did_call_seer` metric.
- In both the backfill and similar issues tab endpoint code, getting `grouping_info` is now split into two steps, first getting the variants and then getting grouping info from the variants, so that the variants can be fed to `has_too_many_contributing_frames`.
- All `get_stacktrace_string` tests whose names included some variation of "too many fames" have been renamed, so as not to confuse them with the tests of the new helper, which are added in a separate class at the bottom of the module.

Note that one of the tests is marked as a skip, because in the process of doing this I found what might be a bug in our grouping logic. (TL;DR, frames which are marked `-app +group` by a stacktrace rule might not be behaving as expected... depending on what you think is expected.) In any case, for now the test is skipped, and once we resolve the stacktrace rule question, we can ether turn it back on or change it.
@github-actions github-actions bot locked and limited conversation to collaborators Jan 5, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants