Get routing and triage above 90%

[hubertdeng123]

Goal

We want to drive up the engagement between Sentry employees and the open source community. Today, this impact is measured in our OSPO dashboard. As a company we should strive to have >90% of issues triaged/routed on time.

Why are numbers low? What are opportunities to fix this?

1. At Sentry, we use Slack as a primary source of communication, which isn't being utilized enough to address issues in GitHub. Our current process involves support going in to Github to find the issues needing to be triaged/routed, and then going to a team's channel to ping them to address the issues. Currently, much of the routing/triaging process is done through Github actions, and migrating to eng-pipes could utilize our Sentaur bot to help out.
2. Support is experiencing pain pinging teams for issues, and it's often difficult for them to find the appropriate channel/team to post in.
3. Employees are not receiving notifications on Untriaged issues after an issue has been routed.
4. Slackbot integration for reminding support on issues is lackluster, it would be better to ping their channel whenever an issue that comes in with the Untriaged or Unrouted labels, and periodically remind their on-call to address.
5. A bug on Github related to ghost labeling has been affecting our dashboard. A support ticket has been filed but their eng team is pretty slow and we shouldn't rely on them.

How will we measure engagement?

NUMBERS!! Engagement on the OSPO dashboard should increase

What are the improvements to be made?

• Migrate naive weekend logic to eng-pipes #71
• Compute business day notifications with more granularity in routing/triage logic #62
• Convert GitHub Actions for routing issues -> Eng-pipes #58 (Medium)
• More regular reminders for issues that need triaging #41 (Medium)
• Standardize on labels for main Sentry Repos #59 (Medium)
• Send Slack notifications for routing. eng-pipes#245 (Medium)

Punchlist

• Use a more readable datetime for routing/triaging eng-pipes#375 (stretch)
• Remove some migrated logic from GHA -> eng-pipes .github#107
• Improve wording of due by dates eng-pipes#382
• Revise wording on "Issue pending triage" notification eng-pipes#384
• Exclude reminding teams of triage if tickets came in less than 4 hours ago eng-pipes#388
• Sort triage notifications by time to due date #87
• Vanguard post!

Follow-up

• Cleanup teams and labels and docs #60
• Track follow ups #89
• Tighten up layout on batch triage message eng-pipes#385

[hubertdeng123]

Weekly update Nov 14:

Last week:

• /notify-for-triage with office arg has been merged
• brainstorming of how notifications will be set up mostly ironed out (will use BigQuery to store the timeToTriageBy and timeToRouteBy timestamps, weekend/business day logic will be set up in eng pipes entirely)
• draft PR up for moving weekend logic to eng-pipes

This week:

• Add timeToTriage and timeToRoute in BigQuery
• Find a way to migrate the queries used in Looker to the new system

[hubertdeng123]

Weekly update Nov 21:

Last week:

• Added timeToTriage and timeToRoute in BigQuery
• Routing notifications for support are in prod

This week:

• use timeToTriage and timeToRoute in OSPO dashboards as a source of truth for SLO violations

[hubertdeng123]

Weekly Update Nov 28:

Last week:

• merged timeToTriage and timeToRoute in OSPO dashboards as a source of truth for SLO violations

This week:

• business hour calculations for timeToTriage and timeToRoute
• allow channels to support multiple offices as timezones
• work on supporting holidays in eng-pipes

[hubertdeng123]

Weekly Update Dec 5:

Last week:

• allow channels to support multiple offices as timezones
• work on supporting holidays in eng-pipes

This week:

• support business hours in timeToTriage and timeToRoute
• merge supporting holidays in eng-pipes

[hubertdeng123]

Weekly Update Dec 12:

Last week:

• support business hours in timeToTriage and timeToRoute
• merge supporting holidays in eng-pipes
• fixed a bug with holiday logic

This week:

• merge business hour support for timeToTriage and timeToRoute
• double check looker to ensure logic is as expected

[hubertdeng123]

Oof late update, slipped my mind this monday

Weekly Update Dec 19:

Last week:

• support business hours in timeToTriage and timeToRoute
• merge supporting holidays in eng-pipes
• fixed a bug with holiday logic

This week:

• merged lookml PR for using business hours in timeToTriage and timeToRoute
• put up PR to standardize github actions for issue routing to eng-pipes

[hubertdeng123]

Weekly Update Dec 28:

Last week:

• merged lookml PR for using business hours in timeToTriage and timeToRoute
• put up Move routing logic from github to eng-pipes eng-pipes#370 to standardize github actions for issue routing to eng-pipes

This week:

• merged Move routing logic from github to eng-pipes eng-pipes#370 and subsequent pr Add routing logic to sentry repo eng-pipes#373 to add routing to sentry repo
• start working on more regular notifications for triaging

[hubertdeng123]

Weekly Update Jan 4:

Last week:

• merged Move routing logic from github to eng-pipes eng-pipes#370 and subsequent pr Add routing logic to sentry repo eng-pipes#373 to add routing to sentry repo
• start working on more regular notifications for triaging

This week:

• merge Rework scheduler triage notifications eng-pipes#376
• merge https://github.com/getsentry/ops/pull/5857 for moving cloud scheduler triage job to terraform for easier edits, should just be able to edit the schedule of the job to get it to run hourly in the future
• put up PR to support hourly notifications

[chadwhitacre]

Are we ready to remove the old yml actions yet? Thinking about:

https://github.com/getsentry/.github/blob/main/.github/workflows/unroute-new-issue.yml

[hubertdeng123]

Yes, we can remove some of those.

[hubertdeng123]

Weekly Update Jan 10:

This week:

• merge Add hourly notification support eng-pipes#377
• merge https://github.com/getsentry/ops/pull/5863
• merge Send slack message to OSPO channel if a channel is not subscribed to any offices eng-pipes#381
• merge Improve wording of due by dates eng-pipes#382
• merge Use channel name instead of channel id to notify ospo channel eng-pipes#383

[hubertdeng123]

Weekly Update Jan 17:

This week:

• Draft vanguard post
• Draft Q1 plan
• Revise wording on "Issue pending triage" notification eng-pipes#384
• Use a more readable datetime for routing/triaging eng-pipes#375

[hubertdeng123]

Weekly Update Jan 24:

This week:

• Exclude reminding teams of triage if tickets came in less than 4 hours ago eng-pipes#388
• Assist on Enable routing for GTM teams #86
• Look into Tighten up layout on batch triage message eng-pipes#385

[hubertdeng123]

Weekly Update Jan 24:

This week:

• merge Sort triage issue list by due date eng-pipes#397

[chadwhitacre]

Duuuuuuuuude ...

[chadwhitacre]

@hubertdeng123 Quick! Close the ticket while we're above 90%! 😅

[hubertdeng123]

Last punchlist item in 😁

Closing this...