Add PostHog thumbs up/down feedback links to PR review comments by xingyaoww · Pull Request #1714 · OpenHands/software-agent-sdk

xingyaoww · 2026-01-13T17:23:26Z

Summary

Adds PostHog-backed thumbs up/down feedback links to every OpenHands PR review comment.

Centralizes PR review posting in agent_script.py so feedback links are always appended.
Updates PR review prompt to require JSON-only output (agent no longer posts via gh/curl).
Adds a GitHub Pages feedback page (/feedback/index.html) that captures pr_review_feedback in PostHog.
Adds a GitHub Actions workflow to deploy the feedback page to GitHub Pages.

Checklist

If the PR is changing/adding functionality, are there tests to reflect this?
If there is an example, have you run the example to make sure that it works?
If there are instructions on how to run the code, have you followed the instructions and made sure that it works?
If the feature is significant enough to require documentation, is there a PR open on the OpenHands/docs repository with the same branch name?
Is the github CI passing?

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	`eclipse-temurin:17-jdk`	Link
python	amd64, arm64	`nikolaik/python-nodejs:python3.12-nodejs22`	Link
golang	amd64, arm64	`golang:1.21-bookworm`	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:0c888f2-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-0c888f2-python \
  ghcr.io/openhands/agent-server:0c888f2-python

All tags pushed for this build

ghcr.io/openhands/agent-server:0c888f2-golang-amd64
ghcr.io/openhands/agent-server:0c888f2-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:0c888f2-golang-arm64
ghcr.io/openhands/agent-server:0c888f2-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:0c888f2-java-amd64
ghcr.io/openhands/agent-server:0c888f2-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:0c888f2-java-arm64
ghcr.io/openhands/agent-server:0c888f2-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:0c888f2-python-amd64
ghcr.io/openhands/agent-server:0c888f2-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-amd64
ghcr.io/openhands/agent-server:0c888f2-python-arm64
ghcr.io/openhands/agent-server:0c888f2-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-arm64
ghcr.io/openhands/agent-server:0c888f2-golang
ghcr.io/openhands/agent-server:0c888f2-java
ghcr.io/openhands/agent-server:0c888f2-python

About Multi-Architecture Support

Each variant tag (e.g., 0c888f2-python) is a multi-arch manifest supporting both amd64 and arm64
Docker automatically pulls the correct architecture for your platform
Individual architecture tags (e.g., 0c888f2-python-amd64) are also available if needed

Co-authored-by: openhands <openhands@all-hands.dev>

all-hands-bot

Good implementation overall, but there are some important error handling and robustness issues to address before merging. The core functionality looks solid, but needs better exception handling and test coverage.

all-hands-bot · 2026-01-13T17:27:16Z

examples/03_github_workflows/02_pr_review/agent_script.py

+    try:
+        json.loads(text)
+        return text
+    except Exception:


🟠 Important: Using bare except Exception is too broad and can hide bugs. Be more specific about what exceptions you're catching.

Suggested change

except Exception:

except json.JSONDecodeError:

This makes it clear you're only catching JSON parsing errors, not other unexpected exceptions like KeyboardInterrupt or SystemExit.

all-hands-bot · 2026-01-13T17:27:16Z

examples/03_github_workflows/02_pr_review/agent_script.py

+            ReviewComment(
+                path=path,
+                side=side,
+                line=int(line),


🟠 Important: Converting line to int without validation could raise ValueError if the agent returns invalid data (e.g., a float or string that isn't a number).

Consider adding try-except or validation:

Suggested change

line=int(line),

line=int(line) if isinstance(line, (int, str)) else line,

all-hands-bot · 2026-01-13T17:27:16Z

examples/03_github_workflows/02_pr_review/agent_script.py

+        except Exception as e:
+            logger.error(f"Failed to parse agent JSON output: {e}")
+            event = "COMMENT"
+            raw = review_content.strip()
+            if len(raw) > 20000:
+                raw = raw[:20000] + "\n\n...[truncated]..."
+
+            review_body = append_feedback_links(
+                "Automated review output could not be parsed into structured comments. "
+                "Posting raw output instead.\n\n" + raw,
+                base_url=feedback_base_url,
+                repo=repo,
+                pr_number=pr_number,
+                run_id=run_id,
+                comment_key="summary",
+            )
+            gh_comments = []


🟡 Suggestion: The fallback behavior truncates at 20,000 characters, which could cut off in the middle of important information. Consider truncating at a newline boundary or providing a better summary.

Also, the broad except Exception here could hide unexpected errors. Consider catching more specific exceptions or at least logging the full traceback for debugging.

all-hands-bot · 2026-01-13T17:27:16Z

.github/pages/feedback/index.html

+      (function(t,e){var o,n,p,r;e.__SV=1;window.posthog=e;e._i=[];e.init=function(i,s,a){function g(t,e){var o=e.split(".");2==o.length&&(t=t[o[0]],e=o[1]);t[e]=function(){t.push([e].concat(Array.prototype.slice.call(arguments,0)))}}(p=t.createElement("script")).type="text/javascript";p.async=!0;p.src=s.api_host+"/static/array.js";(r=t.getElementsByTagName("script")[0]).parentNode.insertBefore(p,r);var u=e;for(void 0!==a?u=e[a]=[]:a="posthog",u.people=u.people||[],u.toString=function(t){var e="posthog";return"posthog"!==a&&(e+="."+a),t||(e+=" (stub)"),e},u.people.toString=function(){return u.toString(1)+".people (stub)"},o="capture identify alias people.set people.set_once set_config register register_once unregister opt_out_capturing opt_in_capturing has_opted_out_capturing reset get_distinct_id debug on off".split(" "),n=0;n<o.length;n++)g(u,o[n]);e._i.push([i,s,a])};
+      })(document,window.posthog||[]);
+
+      posthog.init('phc_QkAtbXVsh3Ja0Pw4IK696cxYEmr20Bx1kbnI7QtOCqg', {


🟢 Nit: While PostHog project API keys (starting with phc_) are designed to be public and safe to use client-side, it would be helpful to add a comment explaining this to prevent future confusion.

Suggested change

posthog.init('phc_QkAtbXVsh3Ja0Pw4IK696cxYEmr20Bx1kbnI7QtOCqg', {

// Public PostHog project key - safe to expose client-side

posthog.init('phc_QkAtbXVsh3Ja0Pw4IK696cxYEmr20Bx1kbnI7QtOCqg', {

all-hands-bot · 2026-01-13T17:27:16Z

.github/pages/feedback/index.html

+      function ensureDistinctId() {
+        const key = 'oh_review_feedback_distinct_id';
+        let id = localStorage.getItem(key);
+        if (!id) {
+          id = (crypto && crypto.randomUUID) ? crypto.randomUUID() : String(Date.now()) + '-' + Math.random().toString(16).slice(2);
+          localStorage.setItem(key, id);
+        }
+        return id;


🟠 Important: localStorage can fail in browsers with strict privacy settings or in incognito/private mode. This will cause the entire tracking to fail.

Add try-catch error handling:

Suggested change

function ensureDistinctId() {

const key = 'oh_review_feedback_distinct_id';

let id = localStorage.getItem(key);

if (!id) {

id = (crypto && crypto.randomUUID) ? crypto.randomUUID() : String(Date.now()) + '-' + Math.random().toString(16).slice(2);

localStorage.setItem(key, id);

}

return id;

function ensureDistinctId() {

try {

const key = 'oh_review_feedback_distinct_id';

let id = localStorage.getItem(key);

if (!id) {

id = (crypto && crypto.randomUUID) ? crypto.randomUUID() : String(Date.now()) + '-' + Math.random().toString(16).slice(2);

localStorage.setItem(key, id);

}

return id;

} catch (e) {

// Fallback if localStorage is unavailable

return (crypto && crypto.randomUUID) ? crypto.randomUUID() : String(Date.now()) + '-' + Math.random().toString(16).slice(2);

}

all-hands-bot · 2026-01-13T17:27:16Z

.github/pages/feedback/index.html

+        document.getElementById('line').textContent = line || '(n/a)';
+
+        const distinctId = ensureDistinctId();
+        posthog.identify(distinctId);


🟠 Important: If PostHog fails to load or initialize (network issues, ad blockers, etc.), this will throw an error. Add error handling:

Suggested change

posthog.identify(distinctId);

try {

const distinctId = ensureDistinctId();

posthog.identify(distinctId);

} catch (e) {

console.warn('PostHog identification failed:', e);

}

all-hands-bot · 2026-01-13T17:27:16Z

examples/03_github_workflows/02_pr_review/agent_script.py


-    github_token = os.getenv("GITHUB_TOKEN")
+    github_token = _get_required_env("GITHUB_TOKEN")
+    run_id = os.getenv("GITHUB_RUN_ID") or uuid.uuid4().hex


🟡 Suggestion: If GITHUB_RUN_ID is not set (e.g., local testing), using uuid.uuid4().hex is good, but consider logging this for debugging purposes so it's clear when feedback is from a local run vs CI.

all-hands-bot · 2026-01-13T17:27:16Z

examples/03_github_workflows/02_pr_review/agent_script.py

+@dataclass(frozen=True)
+class ReviewComment:
+    path: str
+    side: str
+    line: int
+    body: str
+    start_line: int | None = None
+    start_side: str | None = None


🔴 Critical: This PR adds significant new functionality but includes no tests. At minimum, add tests for:

_extract_json_object() with various input formats

parse_review_response() with valid and invalid JSON

build_feedback_url() with different parameters

append_feedback_links() output format

The lack of tests makes it difficult to verify correctness and prevents regressions.

all-hands-bot · 2026-01-13T17:27:16Z

.github/pages/feedback/index.html

+        if (repo && pr && runId && commentKey && (rating === 'up' || rating === 'down') && !alreadySent(dedupeKey)) {
+          posthog.capture('pr_review_feedback', {
+            repo,
+            pr_number: pr,
+            run_id: runId,
+            comment_key: commentKey,
+            rating,
+            path,
+            line,
+            return_to: returnTo
+          });
+          markSent(dedupeKey);
+        }


🟠 Important: If PostHog capture fails, it should be caught to prevent breaking the page functionality:

Suggested change

if (repo && pr && runId && commentKey && (rating === 'up' || rating === 'down') && !alreadySent(dedupeKey)) {

posthog.capture('pr_review_feedback', {

repo,

pr_number: pr,

run_id: runId,

comment_key: commentKey,

rating,

path,

line,

return_to: returnTo

});

markSent(dedupeKey);

}

const dedupeKey = [repo, pr, runId, commentKey, rating].join('|');

if (repo && pr && runId && commentKey && (rating === 'up' || rating === 'down') && !alreadySent(dedupeKey)) {

try {

posthog.capture('pr_review_feedback', {

repo,

pr_number: pr,

run_id: runId,

comment_key: commentKey,

rating,

path,

line,

return_to: returnTo

});

markSent(dedupeKey);

} catch (e) {

console.warn('Failed to send feedback to PostHog:', e);

}

}

xingyaoww · 2026-01-15T22:09:13Z

@OpenHands review diff against main. Do you think if it is possible to get the feedback mechanism to work WITHOUT deploying a page?

openhands-ai · 2026-01-15T22:09:23Z

I'm on it! xingyaoww can track my progress at all-hands.dev

openhands-ai · 2026-01-15T22:11:02Z

Summary

I reviewed the diff for PR #1714 which adds PostHog thumbs up/down feedback links to PR review comments.

Answer to your question: Can the feedback mechanism work WITHOUT deploying a page?

Short answer: Technically yes, but the GitHub Pages approach is the best solution.

Why a deployed page is needed:

PostHog requires an API key to send events
You can't expose the API key directly in PR comment links (security risk)
GitHub Markdown links can't execute code or make authenticated API calls
The HTML page acts as a secure proxy that safely uses PostHog's client-side JS library

Alternatives without GitHub Pages:

✅ Serverless function (Vercel/Cloudflare Workers) - Still "deploying" but no GitHub Pages setup
⚠️ URL shortener with analytics - Less real-time, more complex
⚠️ Third-party forms (Tally.so, Google Forms) - Less control, data in third-party service
❌ Direct PostHog API - Not possible (security + technical limitations)

Why the GitHub Pages approach in this PR is ideal:

✅ Free and automatic (no external services)
✅ One-time setup with GitHub Actions
✅ Secure (API key protected by client-side JS on GitHub's domain)
✅ Fast static HTML serving
✅ Built-in deduplication via localStorage
✅ No ongoing maintenance costs

Recommendation: Keep the GitHub Pages implementation unless you have a specific requirement to avoid it. If you must avoid it, use a serverless function (Vercel/Cloudflare), but that's more complex and requires maintaining a separate service.

No changes were made to the codebase - I only answered your question.

View full conversation

- Relocate feedback/index.html to .github/pages/feedback/index.html - Update GitHub Actions workflow to deploy from new location - Keeps feedback page hidden in .github directory while maintaining functionality Co-authored-by: openhands <openhands@all-hands.dev>

openhands-ai · 2026-01-15T22:22:57Z

Looks like there are a few issues preventing this PR from being merged!

GitHub Actions are failing:
- Pre-commit checks

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #1714 at branch `pr-review-feedback-links-posthog`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

all-hands-bot · 2026-01-30T12:22:56Z

[Automatic Post]: It has been a while since there was any activity on this PR. @xingyaoww, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

xingyaoww · 2026-02-10T11:04:59Z

the lamniar evaluation seems like a better idea so close this

Add PostHog feedback links to PR review comments

89793c0

Co-authored-by: openhands <openhands@all-hands.dev>

all-hands-bot reviewed Jan 13, 2026

View reviewed changes

xingyaoww marked this pull request as draft January 13, 2026 17:28

Merge branch 'main' into pr-review-feedback-links-posthog

c1ba6b8

xingyaoww closed this Feb 10, 2026

	line=int(line),
	line=int(line) if isinstance(line, (int, str)) else line,

	posthog.init('phc_QkAtbXVsh3Ja0Pw4IK696cxYEmr20Bx1kbnI7QtOCqg', {
	// Public PostHog project key - safe to expose client-side
	posthog.init('phc_QkAtbXVsh3Ja0Pw4IK696cxYEmr20Bx1kbnI7QtOCqg', {

Conversation

xingyaoww commented Jan 13, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

all-hands-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xingyaoww commented Jan 15, 2026

Uh oh!

openhands-ai bot commented Jan 15, 2026

Uh oh!

openhands-ai bot commented Jan 15, 2026

Summary

Answer to your question: Can the feedback mechanism work WITHOUT deploying a page?

Uh oh!

openhands-ai bot commented Jan 15, 2026

Uh oh!

all-hands-bot commented Jan 30, 2026

Uh oh!

xingyaoww commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xingyaoww commented Jan 13, 2026 •

edited by github-actions bot

Loading