[ENHANCEMENT] Global FIFO queue for Evals runs (1 at a time)

[hannesrudolph]

Type

Enhancement

Problem

When launching multiple eval runs from the Web UI, there’s no way to queue them. This forces me to wait for a run to finish or manually time the next one.

Context

• Affects users running evals from the Web UI.
• Desired behavior: a simple global FIFO queue so only one run executes at a time; additional runs are queued automatically.
• The Runs list should show a clear “Queued” status with the run’s position in the queue.
• Allow canceling queued runs.
• No reordering needed.
• Task-level parallelism inside a single run should remain as configured.

Constraints/Preferences

Keep the flow simple and predictable, with minimal UI changes: show status and position, and provide a cancel action for queued entries.

[roomote]

I'm exploring a potential fix for implementing a global FIFO queue for Evals runs and will open a PR to share progress.

[roomote]

I've opened PR #7967 with my attempt at implementing the global FIFO queue for Evals runs: #7967

The implementation includes:

• Redis-based FIFO queue management
• Queue processor ensuring only one run executes at a time
• UI updates showing queued status with position numbers
• Ability to cancel queued runs
• Database schema updates with proper migration

All requirements from the issue have been addressed. The CI checks are currently running. Feedback and suggestions are welcome!

[hannesrudolph]

Hey @roomote-agent,

Summary of findings

• Today, the web app spawns a controller immediately for every run in createRun(). There’s no global run-level queue—only per-run task concurrency via PQueue inside runEvals().
• A run’s “running” state can be inferred from the Redis heartbeat set by startHeartbeat() and cleared by stopHeartbeat(). The UI’s run details page renders this via RunStatus.
• The schema for runs doesn’t include a status field, and there’s no queueing logic in either the web action or the CLI. This matches the docs (‘Not used for queuing’) in ARCHITECTURE.md.

Hypothesis validated

• Because createRun() unconditionally spawns the controller, multiple runs can overlap. Intra-run limits are respected by PQueue creation in runEvals(), but inter-run FIFO doesn’t exist. The Runs list Runs/Run has no ‘Queued’ status, and there’s no cancel-queued affordance. This matches the enhancement request.

Suggested solution (lightweight global FIFO)

• Goal: Only one run executes at a time. Additional runs are queued automatically. Preserve per-run task concurrency. Minimal UI changes (status + position + cancel).
• Use Redis for queuing/locks (no DB migration required initially):
  • Keys
    • evals:run-queue (LIST) – FIFO of run IDs.
    • evals:active-run (STRING) – set to the currently executing runId; use SET NX with EX for crash safety.
    • evals:dispatcher:lock (STRING) – short-lived lock to serialize dispatch operations (SET NX EX).
• Enqueue on creation (web)
  • In createRun():
    • Push run.id to evals:run-queue and skip immediate spawn if evals:active-run exists.
    • Call a new dispatchNextRun() that:
      • Acquires evals:dispatcher:lock.
      • If no evals:active-run exists, LPOP evals:run-queue → nextId.
      • If nextId found: set evals:active-run = nextId (NX, generous EX), then spawn the controller using the same command currently built at createRun().
• Auto-advance when a run completes
  • At the end of runEvals() after stopHeartbeat():
    • DEL evals:active-run.
    • Immediately dispatch next:
      • Option A (simplest): implement the same dispatchNextRun() in the CLI so it can LPOP and spawn the next controller directly (use the same docker invocation pattern as processTaskInContainer() but with runId: ‘pnpm --filter @roo-code/evals cli --runId X’).
      • Option B: publish a ‘queue:dispatch’ event; have the web service subscribe and run dispatchNextRun(). This keeps all spawning centralized in the web app if preferred.
• UI updates (minimal)
  • Runs list Runs / row Run:
    • Add a Status column:
      • Running if getHeartbeat returns a value for the run.
      • Queued (#N) if LPOS(evals:run-queue, runId) = i, display N = i + 1.
      • Completed when run.taskMetricsId is set.
    • Add ‘Cancel’ for queued entries: server action cancelQueuedRun(runId) that LREM evals:run-queue 1 runId and then calls deleteRun().

Notes

• This preserves intra-run concurrency (PQueue in runEvals()).
• Crash safety: evals:active-run uses EX TTL and is independent of the heartbeat TTL from startHeartbeat(). If a controller dies, both TTLs expire and the dispatcher advances.
• Optionally, a runs.status column can be added later in schema for analytics (‘queued|running|finished|canceled’), but not required for the initial feature since status can be derived from Redis + existing fields.

[roomote]

Thanks @hannesrudolph for the detailed analysis and suggested solution! I've reviewed your findings and the lightweight global FIFO approach using Redis. I'll implement this solution following your specifications:

• Redis-based queue management with evals:run-queue, evals:active-run, and evals:dispatcher:lock
• Modified createRun() to enqueue runs instead of immediate spawning
• Auto-advance mechanism when runs complete
• UI updates for queue status and cancel functionality

I'll start exploring the codebase and implementing the changes now.