Upgrading Sentry's Slack Experience

[Dhrumil-Sentry]

A LOT of developers triage their Sentry issues directly from Slack. Sentry’s Slack integration makes developer workflows better and our users have called out several enhancements they’d like to see in our integration such as #19271, #28731, #57368, #58921, #57370, #58433, #50944, #48852, #44491, #44340

We’re working on addressing these requests and making this integration better over the coming few months 🎉

Here’s what we’re planning to build:

More informative alerts

We plan to start showing more relevant information in issue alerts as a default:

• Error counts
• Count of users impacted
• Suspect Commits
• Issue Level- info/warning/error
• Issue State (New/Regressed/Escalating etc)
• Issue Category- Error/Performance/Cron etc

Support for Mentions and extra Issue Actions

We plan to allow users to specify “@” mentions they’d like to add to their alerts allowing them to tag Slack users directly.
Additionally, we will support more actions in Slack - E.g. being able to archive issues forever or for your custom conditions like >100 errors in 1 day.

Threads

Developers want their Slack alerts to be updated with the latest statuses after any issue update is made in the Sentry UX - For example, if you resolve an issue in the Sentry app or Slack itself we’ll update the Slack alert for the issue via a new threaded message so your team can easily see the latest state of the issue.

Mockup

Here’s a very preliminary mockup(and very likely to change!) of how the issue alert could look like:

Your feedback is much appreciated!

There’s a lot that Sentry can do to improve your Slack experience. We’d love your thoughts on the following:

• What additional information would you like to see in your Slack Alerts? Is there anything you’d like us to not show?

• How can we help you triage more effectively in Slack?

  • Would you like to see suggested assignees for an issue in Slack itself?

• What are other useful Slack features for Sentry to build?

  • GitHub does a digest of PR statuses for a team- We think a periodic digest for a team’s issues and performance problems would be useful for a lot of developers- What do you think?

[mwarkentin]

When I see these alerts in Slack I click into them and then am just acting as a human pattern matcher to pick out the 1/5 errors that actually looks like it correlates to the spike triggering the errors alert above.

It feels like we should be able to confidently say that it looks like the following error that triggered the alert (and most of the time there is 1 error that definitively matches the pattern) - wouldn't it be cool if our alert highlighted that? Or sent a follow up in the thread saying "hey, this looks like this "IntegrationError: Error Communicating with GitHub (HTTP 403)" error?

[Dhrumil-Sentry]

@mwarkentin This is a great idea! We should highlight that in the UX of the Metric Alert too. cc @rachrwang

[wdhdev]

This looks really nice. I'd think it would be good to see similar integration with Discord 👀

[Dhrumil-Sentry]

@WilliamDavidHarrison Thanks for the feedback! We're starting with slack but we'd love to do this for other messaging integrations in the future too

[realkosty]

One customer pointed out it's not obvious which alert triggered notification

[jboulter11]

One thing I'd really like to see is the ability to just include some text with the alert. Instead of only being able to include tags, I'd like a text box where I can explain why the alert is probably firing (unless Sentry can provide that automatically) and what to do, probably linking to some documentation.

Maybe it'd make sense for that to live in the alert as a description and then any integration could include it?

[Dhrumil-Sentry]

@jboulter11 Thanks a lot for this feedback. This has been asked by other larger customers too to help developers know what to do when the alert fires by linking a rulebook

@rachrwang @leedongwei something to consider as we improve alert rules

[jboulter11]

No problem at all. I think the same sentiment I also expressed here: #62565 (comment) applies to this. I really need to know "What, Why, How Bad, and What do I do about it?" when an alert or integration triggers. This is very similar to updates to the Jira ticket mentioned there.

[realkosty]

That an awesome idea
Back in the day at AWS we used to have a long wiki page runbook with headings for each error signature/message and details how to mitigate it. You would basically Cmd+F through that page looking for the closest match 😂

[mwarkentin]

I had an idea about using Sentry Functions a while back to enrich issues / alerts with stuff like links to runbooks and other docs. It was really just doing keyword matching in the issue title and then the plan was to add a comment to the issue with links (but I never really got around to finishing it up).

https://github.com/getsentry/function-enrich-issue-runbooks/blob/main/handler.js

[adamgsg]

My wishlist for Slack alerts:

• Ability to include more than just tags. Any details / contexts visible on the issue page should be available.
• If there is a JIRA ticket associated with the issue, include the ticket name, and make it link to JIRA
• Button to create a new JIRA ticket from the alert (if no ticket already associated)
• Ability to temporarily silence/archive noisy alerts
• Include description of error rate (n,nnn errors in last nn minutes)
• Include helpful links (we'd likely link to runbooks, escalation lists, and other helpful pages)

[Dhrumil-Sentry]

@adamgsg Thanks for the feedback on this discussion as well 🙌 .

We'll be tackling some of these items immediately and will add your other suggestions to our backlog

[JessJNielsen]

Is there an option to use the old style alerts? The red quote-like box made them easier to pick apart when you have many alerts in slack.

[Dhrumil-Sentry]

Hi @JessJNielsen , We released the first set of improvements to Early Access customers - so thank you for this feedback 🙏 . The red quote like box was a feature of Slack's legacy composition format which is not supported by their latest Block Kit formatting. Which is why we could not continue to keep them. We're using various icons to denote the level of the issue instead now such as ❗ for errors, ⚠️ for warning and ℹ️ for info

We are moving all our alerts to the latest block kit format which is why it won't be possible to keep the old style. But we're considering adding some configurability as I mentioned in this issue #64984 (comment)

[dachakra]

Hi @Dhrumil-Sentry, are there any plans on adding notes to Metric Alerts? A team has described the notes feature as a life-saver for them if it was available on Metric Alerts.

[Dhrumil-Sentry]

@dachakra We do plan on including Notes for Metric Alerts too, but we don't expect taking it on in this quarter, we'll update here if anything changes

[Dhrumil-Sentry]

Update

We made some Slack improvements live for customers who had opted in for Early Access for a couple weeks, We got a ton of feedback and we’ve made certain changes to address this feedback

1. No tags are added to the alerts as a default: Every org cares about different tags so we will not be adding default tags to the alert, this reduced the size of the alerts too
2. Colored quote Blocks had to be replaced with colored circles: We had to move over to Slack’s latest Block kit which unfortunately don’t support quote blocks - So instead we’re using 🔴 🟡 🔵 to denote error, warning and info levels
3. Event and User Counts only showed when needed: Alerts for new issues will have always have event and user counts as 1, so we’re showing this information only when they’re helpful
4. Support for Notes: You can add notes to your alerts by modifying your alert rule action, If you want to mention a specific user in the alert you can add them by adding “<@user_Id>” to your notes

[edhgoose]

Hey @Dhrumil-Sentry - I wondered if there was any updated on the new experience being available for Self Hosted? We're on 24.5 and would love to try it out.

[ykamo001]

Hey @edhgoose , you can turn this on for self-hosted by enabling the feature flag (by setting it to true) here: https://github.com/getsentry/sentry/blob/24.5.0/src/sentry/conf/server.py#L1862

[edhgoose]

Nice, thanks @ykamo001. Stupid question, but where is that config file on self hosted?

Or what can I set to change it?

[ykamo001]

You should just change the False value there to True in that exact file and code line location

# Send Slack notifications to threads for Issue Alerts
    "organizations:slack-thread-issue-alert": False,

should get updated to

# Send Slack notifications to threads for Issue Alerts
    "organizations:slack-thread-issue-alert": True,

[edhgoose]

Thanks @ykamo001 - I've found it now. The problem was that self hosted doesn't have a server.py file.

It's sentry.conf.py, hence my confusion.

For anyone else looking, it's this set: https://github.com/getsentry/self-hosted/blob/master/sentry/sentry.conf.example.py#L266-L295

[Dhrumil-Sentry]

We've often heard complains about how our metric alerts can spam a channel as the alert changes from Critical/Warning/Resolved.

So we're excited to announce another improvement that we've made available for Early Adopters- We're going to start using threads in Slack to inform users when a metric alert changes state.

If the alert changes to Critical status- We will send the notification to the main channel. Stay tuned for more thread based improvements 😄

[Dhrumil-Sentry]

We will be making the Slack Formatting improvements generally available to all customers over the next 3 days. There is no action that customers need to take. Your Slack Alerts will automatically start using the new Block Kit Format. Please join the discussion here if you have any feedback 🙏

[Dhrumil-Sentry]

Yes, we did @adamgsg - We had some problems with our feature flagging. Please let us know if you have any feedback!

[adamgsg]

Do you know how to include in the note some text that is a link? I tried with the regular slack markdown format, but it was sent to slack as-is, with the full url visible instead of linking the text.

[Dhrumil-Sentry]

@adamgsg I was able to get this working by using the following format

<http://example.com|Click Here>

In my alert rule's note section - I added Please refer <https://develop.sentry.dev/docs/|here> and here's how it shows up for me:

[dmitry-textmagic]

Hello! We are happy with the new notification format, except one small detail: these notifications are now being trimmed to 254 symbols.

It is not convenient for us because not all developers nor support agents, for security reasons, have access to Sentry. Before this change, they were able to read the whole message in Slack.

Is there any chance of keeping the whole User Feedback message when sending it to Slack? Thanks.

[Dhrumil-Sentry]

Hi @dmitry-textmagic , This is definitely a fair ask to make - I've created an issue to track this request: #68433

[Dhrumil-Sentry]

We are continuing to iterate on improving the Slack experience- We've shipped a new daily Slack digest to a few early-access customers. This Slack digest will show the top issues, regressions, and escalations observed that day.

Here's how it looks:

We'd love to get your feedback on the following:

• What other content would you like to see in a Daily digest?
• Are there any performance metrics that you care about seeing daily?

Your feedback helps us make this better for many developers 🙏

[codeinthehole]

Is it possible to get a digest for a Sentry team, and posted to a team Slack channel?

[Dhrumil-Sentry]

@codeinthehole It's not possible to do this today but we'll be working on this in the next few months - Could you please provide some feedback about the following:

• What information do you want to see in this digest?
• A team could own multiple projects
  • Do you want the above information for each project or do you want to control the projects for which you receive this digest?
  • Do you prefer getting a combined digest for all the team's projects?

[codeinthehole]

What information do you want to see in this digest?

Something similar to what you have screenshotted above, but for each project that a team is linked to - and only reporting on issues assigned to that team.

Am interested in a two types of metric:

• metrics from things that happened in the reporting period (e.g. last 14 days). E.g.

  • num new issues
  • most frequent issues
  • num resolved issues

• snapshot metrics from end of reporting period. E.g.

  • num assigned, open issues
  • MTTR of resolved issues

do you want to control the projects for which you receive this digest?

Being able to choose which projects are included would be useful.

Do you prefer getting a combined digest for all the team's projects?

I imagine both options would be useful. But personally, I would prefer separate reports per project.

[rodolfoBee]

Created a feature request regarding the threads feature: #70316

[ido-green]

Another enhacement request - post Slack messages in the related thread, whenever there are new Activity events (such as Jira was associated).

[Dhrumil-Sentry]

@ido-green Thanks for this feedback - We currently have a feature for Early Access customers that posts new activity in the threads of the Slack Issue Alerts, We'll make sure this included ticket linking activity too

[Dhrumil-Sentry]

@ido-green We just delivered this feature. Please take a look at this changelog