PR comment emojis for user feedback

[codecovdesign]

Problem to Solve

To better understand the accuracy of our PR comments reports by leveraging direct user feedback. Currently, there is a feedback loop in footer that links to an issue (related issue), but we need a better way to check the perceived accuracy of our reports from the users' perspective.

Proposed Solution

By enabling ability for user to submit thumbs up/down emoji reactions on PR comments, we can collect and analyze feedback systematically. This data will help us identify reports that are perceived as inaccurate by our users, allowing us to review and troubleshoot them effectively. Example currently in use by Sentry:

Next, set up a dashboard, potentially in Redash, to aggregate and visualize the feedback data (Sentry uses). This dashboard will help us monitor the frequency of reports marked as inaccurate and facilitate a review process.

Next Steps

1. Research the technical feasibility of leveraging GitHub webhook for emoji reactions (w/ platform team).
2. Implement the feedback mechanism on PR comments.
3. Set up the data analysis dashboard in Redash (or other?).
4. Develop a process for reviewing and troubleshooting flagged reports.

Questions

1. How much to release this, in terms of % / visibility?
2. What is the text and where does it live in the PR comment structure (footer?)