From e88b7ac499f306070c4caff584b7c7c5415ef2df Mon Sep 17 00:00:00 2001 From: Daniel Szoke Date: Thu, 15 Jan 2026 14:05:50 +0100 Subject: [PATCH 1/4] feat(seer-review): Add AI code review skill using Sentry CLI [PoC] Add new seer-review skill that invokes `sentry-cli review` to get AI-powered code review feedback on local changes before creating a PR. Features: - Invokes sentry-cli review to analyze HEAD vs HEAD~1 - Supports SENTRY_CLI_PATH env var override for testing - Provides instructions for processing predictions and auto-fixing Co-Authored-By: Claude --- .claude/settings.json | 3 +- README.md | 1 + .../skills/claude-settings-audit/SKILL.md | 3 +- .../sentry-skills/skills/seer-review/SKILL.md | 81 +++++++++++++++++++ .../seer-review/scripts/get-sentry-cli.sh | 4 + 5 files changed, 90 insertions(+), 2 deletions(-) create mode 100644 plugins/sentry-skills/skills/seer-review/SKILL.md create mode 100755 plugins/sentry-skills/skills/seer-review/scripts/get-sentry-cli.sh diff --git a/.claude/settings.json b/.claude/settings.json index 18606b9..611e8c6 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -48,7 +48,8 @@ "Skill(sentry-skills:find-bugs)", "Skill(sentry-skills:iterate-pr)", "Skill(sentry-skills:claude-settings-audit)", - "Skill(sentry-skills:agents-md)" + "Skill(sentry-skills:agents-md)", + "Skill(sentry-skills:seer-review)" ], "deny": [] }, diff --git a/README.md b/README.md index afc9fd1..c172d99 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,7 @@ Copy the `skills/` directory to your agent's skills location, or reference the S | [iterate-pr](plugins/sentry-skills/skills/iterate-pr/SKILL.md) | Iterate on a PR until CI passes and feedback is addressed | | [claude-settings-audit](plugins/sentry-skills/skills/claude-settings-audit/SKILL.md) | Analyze repo and generate recommended Claude Code settings.json permissions | | [agents-md](plugins/sentry-skills/skills/agents-md/SKILL.md) | Maintain AGENTS.md with concise agent instructions | +| [seer-review](plugins/sentry-skills/skills/seer-review/SKILL.md) | AI-powered code review using Sentry CLI | ## Available Subagents diff --git a/plugins/sentry-skills/skills/claude-settings-audit/SKILL.md b/plugins/sentry-skills/skills/claude-settings-audit/SKILL.md index 274432e..f9b54c4 100644 --- a/plugins/sentry-skills/skills/claude-settings-audit/SKILL.md +++ b/plugins/sentry-skills/skills/claude-settings-audit/SKILL.md @@ -147,7 +147,8 @@ If this is a Sentry project (or sentry-skills plugin is installed), include: "Skill(sentry-skills:find-bugs)", "Skill(sentry-skills:iterate-pr)", "Skill(sentry-skills:claude-settings-audit)", - "Skill(sentry-skills:agents-md)" + "Skill(sentry-skills:agents-md)", + "Skill(sentry-skills:seer-review)" ] ``` diff --git a/plugins/sentry-skills/skills/seer-review/SKILL.md b/plugins/sentry-skills/skills/seer-review/SKILL.md new file mode 100644 index 0000000..dba59e6 --- /dev/null +++ b/plugins/sentry-skills/skills/seer-review/SKILL.md @@ -0,0 +1,81 @@ +--- +name: seer-review +description: Get AI-powered code review on local changes using Sentry CLI. Use when reviewing commits before creating a PR, finding bugs in recent changes, or getting automated code feedback. Triggers on: review my code, find bugs in commit, sentry review, ai code review, seer. +--- + +# Seer Code Review + +Review local code changes using Sentry's AI-powered bug prediction service. + +## Overview + +This skill invokes `sentry-cli review` to analyze the most recent commit (HEAD vs HEAD~1) and get AI feedback on potential bugs before creating a pull request. + +## Step 1: Resolve CLI Path + +Get the sentry-cli path by running the helper script: + +```bash +./plugins/sentry-skills/skills/seer-review/scripts/get-sentry-cli.sh +``` + +Store the output as `SENTRY_CLI` for subsequent commands. + +## Step 2: Run the Review + +Execute the review command: + +```bash +$SENTRY_CLI review +``` + +**Expected outputs:** + +1. **Success with no issues**: "No issues found in this commit." +2. **Success with issues**: Lists predictions with severity (HIGH/MEDIUM/LOW), file path, line number, description, and suggested fixes +3. **Error cases**: + - "HEAD is a merge commit" - cannot review merge commits + - "HEAD has no parent commit" - cannot review initial commit + - "Diff is too large" - diff exceeds 500KB limit + - "Authentication required" - user needs to run `sentry-cli login` + - "No remote URL found" - repo needs an origin or upstream remote + +## Step 3: Process Results + +### If no issues found +Report success to the user. + +### If issues found +For each prediction in the output: + +1. **Present the issue** to the user with: + - Severity level + - File and line number + - Description of the problem + - Suggested fix (if provided) + +2. **Offer to fix**: Ask the user if they want you to apply the suggested fixes. + +3. **Apply fixes** (if user agrees): + - Read the file at the specified path + - Apply the suggested fix at the indicated line + - Show the change to the user + +4. **Verify fixes**: After applying all fixes, offer to re-run the review to confirm the issues are resolved. + +## Step 4: Handle Errors + +| Error | Resolution | +|-------|------------| +| Authentication required | Tell user to run `sentry-cli login` | +| HEAD is a merge commit | Tell user to review from a non-merge commit | +| HEAD has no parent | Tell user this is the initial commit, cannot review | +| Diff too large | Suggest breaking into smaller commits | +| No remote URL | Tell user to add a git remote | + +## Notes + +- The review analyzes HEAD vs HEAD~1 only (most recent commit) +- Binary files are automatically skipped +- The API has a 10-minute timeout for long-running analysis +- The base commit must be pushed to the remote repository diff --git a/plugins/sentry-skills/skills/seer-review/scripts/get-sentry-cli.sh b/plugins/sentry-skills/skills/seer-review/scripts/get-sentry-cli.sh new file mode 100755 index 0000000..dae1d26 --- /dev/null +++ b/plugins/sentry-skills/skills/seer-review/scripts/get-sentry-cli.sh @@ -0,0 +1,4 @@ +#!/bin/bash +# Returns the path to sentry-cli. +# Uses SENTRY_CLI_PATH environment variable if set, otherwise falls back to "sentry-cli" (resolved via PATH). +echo "${SENTRY_CLI_PATH:-sentry-cli}" From c1cf1b9e466ec7074dca61f7820a6723a703c329 Mon Sep 17 00:00:00 2001 From: Daniel Szoke Date: Thu, 15 Jan 2026 14:57:22 +0100 Subject: [PATCH 2/4] ref(seer-review): Improve skill with context-aware evaluation workflow - Rename get-sentry-cli.sh to run-review.sh and execute review directly - Add allowed-tools for full workflow support (Task, AskUserQuestion, etc.) - Add skeptical evaluation workflow for Seer comments: - Spawn Plan sub-agents to evaluate each comment against user context - Present results to user with AskUserQuestion - Plan and implement fixes for accepted issues - Re-run review in a loop until done - Update error handling (auth token URL, CLI upgrade instructions) - Emphasize that Seer's confidence/severity ratings should not be blindly trusted when agent context suggests otherwise Co-Authored-By: Claude --- .../sentry-skills/skills/seer-review/SKILL.md | 146 +++++++++++++----- .../{get-sentry-cli.sh => run-review.sh} | 5 +- 2 files changed, 107 insertions(+), 44 deletions(-) rename plugins/sentry-skills/skills/seer-review/scripts/{get-sentry-cli.sh => run-review.sh} (52%) diff --git a/plugins/sentry-skills/skills/seer-review/SKILL.md b/plugins/sentry-skills/skills/seer-review/SKILL.md index dba59e6..c082805 100644 --- a/plugins/sentry-skills/skills/seer-review/SKILL.md +++ b/plugins/sentry-skills/skills/seer-review/SKILL.md @@ -1,6 +1,20 @@ --- name: seer-review -description: Get AI-powered code review on local changes using Sentry CLI. Use when reviewing commits before creating a PR, finding bugs in recent changes, or getting automated code feedback. Triggers on: review my code, find bugs in commit, sentry review, ai code review, seer. +description: >- + Get AI-powered code review on local changes using Sentry CLI. + Use when reviewing commits before creating a PR, finding bugs in recent changes, + or getting automated code feedback. + Triggers on: review my code, find bugs in commit, sentry review, ai code review, seer. +allowed-tools: + - Bash + - Task + - Read + - Edit + - Write + - Glob + - Grep + - AskUserQuestion + - EnterPlanMode --- # Seer Code Review @@ -11,67 +25,114 @@ Review local code changes using Sentry's AI-powered bug prediction service. This skill invokes `sentry-cli review` to analyze the most recent commit (HEAD vs HEAD~1) and get AI feedback on potential bugs before creating a pull request. -## Step 1: Resolve CLI Path +**Important context:** Seer analyzes code without knowing **why** changes were made. You have context from the user's original request that Seer lacks. Treat Seer's suggestions with healthy skepticism and evaluate them against your existing context. -Get the sentry-cli path by running the helper script: +## Step 1: Run Review + +Execute the review script: ```bash -./plugins/sentry-skills/skills/seer-review/scripts/get-sentry-cli.sh +./plugins/sentry-skills/skills/seer-review/scripts/run-review.sh ``` -Store the output as `SENTRY_CLI` for subsequent commands. +## Step 2: Handle Errors + +If the review command fails, handle these common errors: + +| Error | Resolution | +|-------|------------| +| Authentication required | Tell user to create a token at https://sentry.io/settings/account/api/auth-tokens/ with the correct permissions | +| Command not found / unavailable | Tell user to upgrade Sentry CLI to the latest version | +| HEAD is a merge commit | Tell user to review from a non-merge commit | +| HEAD has no parent | Tell user this is the initial commit, cannot review | +| Diff too large | Suggest breaking into smaller commits | +| No remote URL | Tell user to add a git remote | + +If an error occurs, stop here and report the error to the user with the resolution. -## Step 2: Run the Review +## Step 3: Evaluate Each Comment -Execute the review command: +If the review returns issues, you must evaluate each one against your existing context. -```bash -$SENTRY_CLI review +**Critical:** Seer analyzes code in isolation without access to the conversation history or the user's intent. This means Seer may identify issues that seem valid from its perspective but are incorrect given what you know. For example: +- Seer might flag a behavioral change as a bug when the user explicitly requested that change +- Seer might suggest reverting code that the user specifically asked you to write +- Seer might identify a "missing" check that was intentionally removed + +**Do not blindly trust Seer's confidence levels or severity ratings.** Seer may label an issue as "critical" or express high confidence, but this confidence is based solely on its limited context. When your context contradicts Seer's assessment, your context takes precedence. + +**For each Seer comment, spawn a Plan sub-agent** (using the Task tool) to evaluate validity: + +``` +Evaluate this Seer code review comment against the context of the user's original request. + +SEER COMMENT: +[paste the comment here] + +USER'S ORIGINAL CONTEXT: +[summarize what the user asked you to do and why] + +IMPORTANT: Seer does not have access to the conversation history or the user's intent. +Its confidence levels and severity ratings are based only on its limited context. +If Seer's suggestion contradicts what the user asked for, the suggestion is likely invalid +regardless of how confident or critical Seer's assessment appears. + +Determine if this issue is: +- **Fully valid**: The issue is a real problem that should be fixed +- **Not valid**: The issue is incorrect because it conflicts with the user's intent +- **Partially valid**: Part of the issue is valid (specify which part) + +Do NOT plan fixes yet. Only evaluate validity. + +OUTPUT FORMAT: +VERDICT: [fully valid | not valid | partially valid (specify which part)] + +OPTIONS CONSIDERED: +1. Fully valid + - Implication: [what this means if true] + - Reasoning: [why you accepted/rejected this - one sentence] +2. Not valid + - Implication: [what this means if true] + - Reasoning: [why you accepted/rejected this - one sentence] +3. Partially valid (if applicable) + - Implication: [what this means if true] + - Reasoning: [why you accepted/rejected this - one sentence] ``` -**Expected outputs:** +**Spawn these sub-agents in parallel** for efficiency when there are multiple comments. -1. **Success with no issues**: "No issues found in this commit." -2. **Success with issues**: Lists predictions with severity (HIGH/MEDIUM/LOW), file path, line number, description, and suggested fixes -3. **Error cases**: - - "HEAD is a merge commit" - cannot review merge commits - - "HEAD has no parent commit" - cannot review initial commit - - "Diff is too large" - diff exceeds 500KB limit - - "Authentication required" - user needs to run `sentry-cli login` - - "No remote URL found" - repo needs an origin or upstream remote +## Step 4: Present Results to User -## Step 3: Process Results +After all sub-agents complete, aggregate their findings and use the AskUserQuestion tool to present each issue to the user. -### If no issues found -Report success to the user. +For each issue, present: +- The original Seer comment (severity, file, line, description) +- The sub-agent's verdict and reasoning +- Options: Accept (fix it), Reject (ignore it), or Modify (partially accept) -### If issues found -For each prediction in the output: +The sub-agent's verdict should be the recommended option, but the user has ultimate authority. -1. **Present the issue** to the user with: - - Severity level - - File and line number - - Description of the problem - - Suggested fix (if provided) +## Step 5: Plan and Implement Fixes -2. **Offer to fix**: Ask the user if they want you to apply the suggested fixes. +If the user accepts any issues as valid: -3. **Apply fixes** (if user agrees): - - Read the file at the specified path - - Apply the suggested fix at the indicated line - - Show the change to the user +1. Use EnterPlanMode to plan fixes for the accepted issues only +2. Get user approval on the plan +3. Implement the approved fixes +4. Commit the changes if appropriate -4. **Verify fixes**: After applying all fixes, offer to re-run the review to confirm the issues are resolved. +## Step 6: Re-run Review (Loop) -## Step 4: Handle Errors +After implementing fixes, re-run the review: -| Error | Resolution | -|-------|------------| -| Authentication required | Tell user to run `sentry-cli login` | -| HEAD is a merge commit | Tell user to review from a non-merge commit | -| HEAD has no parent | Tell user this is the initial commit, cannot review | -| Diff too large | Suggest breaking into smaller commits | -| No remote URL | Tell user to add a git remote | +```bash +./plugins/sentry-skills/skills/seer-review/scripts/run-review.sh +``` + +**Repeat this entire workflow** (Steps 1-6) until one of these conditions is met: +- Seer returns an error +- Seer finds no issues +- User rejects all remaining comments ## Notes @@ -79,3 +140,4 @@ For each prediction in the output: - Binary files are automatically skipped - The API has a 10-minute timeout for long-running analysis - The base commit must be pushed to the remote repository +- Maximum diff size: 500 KB diff --git a/plugins/sentry-skills/skills/seer-review/scripts/get-sentry-cli.sh b/plugins/sentry-skills/skills/seer-review/scripts/run-review.sh similarity index 52% rename from plugins/sentry-skills/skills/seer-review/scripts/get-sentry-cli.sh rename to plugins/sentry-skills/skills/seer-review/scripts/run-review.sh index dae1d26..92fc346 100755 --- a/plugins/sentry-skills/skills/seer-review/scripts/get-sentry-cli.sh +++ b/plugins/sentry-skills/skills/seer-review/scripts/run-review.sh @@ -1,4 +1,5 @@ #!/bin/bash -# Returns the path to sentry-cli. +# Runs sentry-cli review command. # Uses SENTRY_CLI_PATH environment variable if set, otherwise falls back to "sentry-cli" (resolved via PATH). -echo "${SENTRY_CLI_PATH:-sentry-cli}" +SENTRY_CLI="${SENTRY_CLI_PATH:-sentry-cli}" +exec "$SENTRY_CLI" review "$@" From 6240d14c129e6fa56d58de2a842deeeb4bd8adfb Mon Sep 17 00:00:00 2001 From: Daniel Szoke Date: Thu, 15 Jan 2026 15:34:28 +0100 Subject: [PATCH 3/4] ref(seer-review): Improve UX for presenting Seer feedback to user - Add SUMMARY field to Plan sub-agent output format for concise issue descriptions - Update Step 4 with two-phase presentation: 1. Print raw Seer output with reference numbers (emphasizing it's uninterpreted) 2. Use AskUserQuestion with sub-agent summaries citing issue numbers - Summaries must be standalone so users can decide without reading full output Co-Authored-By: Claude --- .../sentry-skills/skills/seer-review/SKILL.md | 40 ++++++++++++++++--- 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/plugins/sentry-skills/skills/seer-review/SKILL.md b/plugins/sentry-skills/skills/seer-review/SKILL.md index c082805..ed55ee1 100644 --- a/plugins/sentry-skills/skills/seer-review/SKILL.md +++ b/plugins/sentry-skills/skills/seer-review/SKILL.md @@ -87,6 +87,10 @@ Do NOT plan fixes yet. Only evaluate validity. OUTPUT FORMAT: VERDICT: [fully valid | not valid | partially valid (specify which part)] +SUMMARY: [1-2 sentence summary of the issue that would make sense to someone who has NOT +read the full Seer output. This summary will be shown in a dialog, so it must be +standalone and clearly explain what Seer is concerned about and your assessment.] + OPTIONS CONSIDERED: 1. Fully valid - Implication: [what this means if true] @@ -103,14 +107,38 @@ OPTIONS CONSIDERED: ## Step 4: Present Results to User -After all sub-agents complete, aggregate their findings and use the AskUserQuestion tool to present each issue to the user. +After all sub-agents complete, present results in two phases: + +### Phase 1: Print Raw Seer Output + +First, print the complete Seer output with reference numbers. Emphasize this is Seer's uninterpreted output: + +``` +## Raw Seer Output + +The following issues were identified by Seer. Note: This is Seer's uninterpreted +output—Seer does not have access to our conversation context. + +### Issue #1: [file:line] [SEVERITY] +[Full description from Seer] +Suggested fix: [Seer's suggested fix] + +### Issue #2: [file:line] [SEVERITY] +[Full description from Seer] +Suggested fix: [Seer's suggested fix] +``` + +### Phase 2: Ask User with Summaries + +Then use AskUserQuestion for each issue, using the sub-agent's SUMMARY in the question. The summary must be clear enough that users can make decisions without reading the raw output above. -For each issue, present: -- The original Seer comment (severity, file, line, description) -- The sub-agent's verdict and reasoning -- Options: Accept (fix it), Reject (ignore it), or Modify (partially accept) +For each issue: +- Reference the issue number (e.g., "Issue #1") +- Include the sub-agent's summary (1-2 sentences) +- Offer options: Accept, Reject, or Modify +- Mark the sub-agent's verdict as the recommended option -The sub-agent's verdict should be the recommended option, but the user has ultimate authority. +The user has ultimate authority over all decisions. ## Step 5: Plan and Implement Fixes From e298467b5685dbc43ba832120d39ab801b40bd98 Mon Sep 17 00:00:00 2001 From: Daniel Szoke Date: Thu, 15 Jan 2026 15:41:52 +0100 Subject: [PATCH 4/4] ref(seer-review): Clarify SUMMARY field instructions Make the SUMMARY output format instructions more explicit about distinguishing Seer's concern from the agent's assessment. Co-Authored-By: Claude --- plugins/sentry-skills/skills/seer-review/SKILL.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/plugins/sentry-skills/skills/seer-review/SKILL.md b/plugins/sentry-skills/skills/seer-review/SKILL.md index ed55ee1..5b0726e 100644 --- a/plugins/sentry-skills/skills/seer-review/SKILL.md +++ b/plugins/sentry-skills/skills/seer-review/SKILL.md @@ -87,9 +87,10 @@ Do NOT plan fixes yet. Only evaluate validity. OUTPUT FORMAT: VERDICT: [fully valid | not valid | partially valid (specify which part)] -SUMMARY: [1-2 sentence summary of the issue that would make sense to someone who has NOT -read the full Seer output. This summary will be shown in a dialog, so it must be -standalone and clearly explain what Seer is concerned about and your assessment.] +SUMMARY: [1-2 sentence summary for someone who has NOT read the full Seer output. +First, briefly state what Seer flagged. Then, state your assessment of whether it's valid. +This will be shown in a dialog, so it must be standalone and clearly distinguish +Seer's concern from your evaluation.] OPTIONS CONSIDERED: 1. Fully valid