ci: add workflow to listen to comments for benchmark requests

[westonpace]

I am working on a historical database of benchmark results at https://github.com/lancedb/lance-bench

I would like to be able to evaluate a PR to see if it improves performance or significantly regresses performance. To do this I am following a pattern used in the Arrow project (and others). By commenting @bench-bot benchmark a user (only users with write permission or PR authors) can trigger the lance-bench repo to run the lance benchmarks on the PR. It will then compare the results with its historical record and decide if the PR's results are an improvement or regression. It will then post that results diff back to the PR as a comment.

This PR adds the hook to listen for comments requesting benchmarks. These comments are then forwarded to the lance-bench repository for further analysis. For more details see https://github.com/lancedb/lance-bench/blob/main/.github/workflows/comment-monitor.yml

[github-actions]

Code Review

P0 - Security Issue: Hardcoded repository should be lancedb/lance-bench

The workflow forwards to westonpace/lance-bench (line 41) but the PR description says the repository is at lancedb/lance-bench. This appears to be a personal fork rather than the organization repository:

repository: westonpace/lance-bench  # Should this be lancedb/lance-bench?

Please confirm the intended target repository. If this is intentional for testing purposes, it should be updated to the organization repo before merge.

---

No other issues identified. The workflow logic is straightforward and follows common patterns for cross-repo dispatch.

[github-actions]

Code Review

This PR adds a GitHub Actions workflow to forward benchmark requests from PR comments to the lance-bench repository. The implementation is straightforward and follows the described pattern.

P1: Security Concern - Missing Permission Check

The workflow currently forwards any comment matching @bench-bot + benchmark to the lance-bench repository. According to the PR description, only users with write permission or PR authors should be able to trigger benchmarks. However, this authorization check is deferred to the lance-bench repository.

Recommendation: Consider adding a permission check in this workflow before dispatching. This would:

1. Reduce unnecessary workflow dispatches from unauthorized users
2. Provide faster feedback to unauthorized users (they would see the workflow fail or skip immediately)

Example check:

- name: Check permissions
  uses: actions/github-script@v7
  with:
    script: |
      const { data: permission } = await github.rest.repos.getCollaboratorPermissionLevel({
        owner: context.repo.owner,
        repo: context.repo.repo,
        username: context.payload.comment.user.login
      });
      const isPrAuthor = context.payload.issue.user.login === context.payload.comment.user.login;
      const hasWriteAccess = ['admin', 'write'].includes(permission.permission);
      if (\!isPrAuthor && \!hasWriteAccess) {
        core.setFailed('User is not authorized to trigger benchmarks');
      }

If the permission check is intentionally handled in lance-bench for centralized control, that's acceptable but should be documented in this workflow's comments.

Minor Notes (not blocking)

• The peter-evans/repository-dispatch@v2 action is stable and appropriate for this use case
• The LANCE_BENCH_DISPATCH_TOKEN secret will need to be configured with appropriate permissions for cross-repo dispatch

Overall, the implementation is clean and serves its purpose. The only significant concern is the missing authorization check.

[westonpace]

The workflow currently forwards any comment matching @bench-bot + benchmark to the lance-bench repository. According to the PR description, only users with write permission or PR authors should be able to trigger benchmarks. However, this authorization check is deferred to the lance-bench repository.

Yes, currently this check is done in the lance-bench repository. I have a script there to post changes on the PR and want to re-use that script to post a "you are not authorized to request this" message. Also, I think the ownership of authorization should belong to lance-bench. This isn't a security / abuse concern for lance as it is not the repository that is doing the bulk of the work.

[westonpace]

Example: #5532