add effort as an input + support parsing from payload #21

ssalbdivad · 2026-01-10T18:00:26Z

Effort Input & Environment-Based API Keys

This PR introduces effort-based model selection and switches API key handling from action inputs to environment variables.

🚨 BREAKING CHANGE - Action Required:

This PR changes API keys from action inputs to environment variables. All pullfrog.yml workflow files in each repo must be updated to move API keys from with: to env: for agents to work.

This migration should be done in parallel with merging this PR to avoid breaking existing workflows.

See Migration Guide below for the required changes.

⚠️ Note: Effort is currently a no-op for OpenCode and Cursor agents. These agents use fixed models regardless of the effort level. Full effort support for these agents is planned for a future release.

Summary of Changes

1. New `effort` Input Parameter

Added a new effort input to control model selection and thinking levels:

Effort	Description	Claude	Codex	Gemini	Cursor	OpenCode
`nothink`	Fast, lower cost	haiku-4-5	gpt-4o-mini	flash-2.5	(no-op)	(no-op)
`think`	Balanced (default)	sonnet-4-5	gpt-5.2-instant	pro-2.5	(no-op)	(no-op)
`max`	Most capable	opus-4-5	gpt-5.2-thinking	deep-think-2.5	(no-op)	(no-op)

Cursor and OpenCode currently ignore the effort parameter and use their default models.

action.yml:

inputs:
  prompt:
    description: "Prompt to send to the agent"
    required: true
  effort:
    description: "Effort level: nothink (fast), think (default), max (most capable)"
    required: false
    default: "think"

2. API Keys via Environment Variables (Breaking Change)

Before: API keys were passed as action inputs (snake_case):

with:
  prompt: "..."
  anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
  openai_api_key: ${{ secrets.OPENAI_API_KEY }}

After: API keys are passed via environment variables (SCREAMING_CASE):

with:
  prompt: "..."
env:
  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

This change:

Aligns with standard conventions for secrets
Simplifies the action interface (only prompt and effort inputs)
Allows agents to read keys directly from process.env

3. Effort Parsing from Payload

For internal Pullfrog invocations (via webhooks), effort can be specified in the JSON payload:

// internal invocation: use effort from payload, fallback to input
effort: parsedPrompt.effort ?? inputs.effort

// external invocation: use effort from input, default to "think"
effort: inputs.effort ?? "think"

The Payload interface now includes:

readonly effort?: Effort; // "nothink" | "think" | "max"

4. Agent Model Configuration

Each agent now receives effort in its config and selects models accordingly:

Claude:

const claudeModels: Record<Effort, { model: string; thinking: boolean }> = {
  nothink: { model: "claude-haiku-4-5", thinking: false },
  think: { model: "claude-sonnet-4-5", thinking: true },
  max: { model: "claude-opus-4-5", thinking: true },
};

Similar configurations exist for Codex and Gemini.

⚠️ Cursor and OpenCode do not yet support effort-based model selection—they use fixed default models regardless of the effort parameter.

5. Updated Runtime to Node 24

runs:
  using: "node24"  # was node20
  main: "entry"

Files Changed

File	Changes
`action.yml`	Removed API key inputs, added `effort` input, updated to node24
`external.ts`	Added `Effort` type, updated `apiKeyNames` to SCREAMING_CASE, added `effort` to `Payload`
`main.ts`	Simplified `Inputs` type, API keys read from `process.env`, effort parsing logic
`entry.ts`	Simplified input handling, validates effort via ArkType
`agents/claude.ts`	Added model selection based on effort, thinking token control
`agents/codex.ts`	Added model selection based on effort
`agents/gemini.ts`	Added model selection based on effort
`agents/cursor.ts`	Receives effort param (no-op, uses fixed model)
`agents/opencode.ts`	Receives effort param (no-op, uses fixed model)
`agents/shared.ts`	Added `effort: Effort` to `AgentConfig` interface
`README.md`	Updated workflow example with env-based API keys

Migration Guide

Update your workflow file:

# Before
- name: Run agent
  uses: pullfrog/action@v0
  with:
    prompt: ${{ inputs.prompt }}
    anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

# After
- name: Run agent
  uses: pullfrog/action@v0
  with:
    prompt: ${{ inputs.prompt }}
    effort: think  # optional: nothink, think (default), max
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Required Workflow Updates

Every repository using Pullfrog needs its .github/workflows/pullfrog.yml updated:

Change	Before	After
API key location	`with: anthropic_api_key:`	`env: ANTHROPIC_API_KEY:`
Key naming	`snake_case`	`SCREAMING_CASE`
Secrets inheritance	Not required	Add `secrets: inherit` to workflow dispatch
Prompt reference	`${{ github.event.inputs.prompt }}`	`${{ inputs.prompt }}`

Full example of updated workflow:

on:
  workflow_dispatch:
    inputs:
      prompt:
        description: 'Agent prompt'
        type: string
    secrets: inherit  # NEW: required for env secrets

jobs:
  agent:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run agent
        uses: pullfrog/action@v0
        with:
          prompt: ${{ inputs.prompt }}
          effort: think
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          # Add other API keys as needed

Testing

All 15 agent/effort combinations tested successfully:

Agent	nothink	think	max	Effort Support
claude	✅	✅	✅	✅ Full
codex	✅	✅	✅	✅ Full
gemini	✅	✅	✅	✅ Full
cursor	✅	✅	✅	⚠️ No-op
opencode	✅	✅	✅	⚠️ No-op

Cursor and OpenCode tests pass but effort does not change model selection.

pullfrog

Well-structured PR with clear documentation. The effort-based model selection is clean and the API key migration improves security. Found one critical issue with model names and several implementation concerns that need addressing.

^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ pullfrog.com ｜ 𝕏}

pullfrog · 2026-01-10T18:01:26Z

agents/codex.ts

+// uses model family aliases that auto-resolve to latest version
+const codexModels: Record<Effort, string> = {
+  nothink: "gpt-4o-mini",
+  think: "gpt-5.2-instant",


Critical: These OpenAI model names appear to be fictional. OpenAI's actual models are gpt-4o-mini, gpt-4o, o1-mini, o1, etc. There is no gpt-5.2-instant or gpt-5.2-thinking. This will cause runtime failures when users select think or max effort levels. Replace with actual OpenAI model names.

Update to use model_reasoning_effort

model_reasoning_effort minimal | low | medium | high | xhigh Adjust reasoning effort for supported models (Responses API only; xhigh is model-dependent).

"low" for "nothink"
"xhigh" for "max"

Updated to use model_reasoning_effort parameter

^{｜ Triggered by Pullfrog ｜ Using Claude Code ｜ View workflow run ｜ pullfrog.com ｜ 𝕏}

pullfrog · 2026-01-10T18:01:26Z

agents/gemini.ts

+// uses model family aliases that auto-resolve to latest version
+const geminiModels = {
+  nothink: "gemini-2.5-pro",
+  think: "gemini-3-pro",


Critical: These Gemini model names don't match Google's naming conventions. Current Gemini models are gemini-2.0-flash-exp, gemini-1.5-pro, gemini-1.5-flash, etc. There is no gemini-2.5-pro or gemini-3-pro. Verify and use actual Gemini API model names or this will fail at runtime.

Deafult to auto (no specification) for "think"

Then force "flash" for nothink and "pro" for "max"

Description Models

Auto (Gemini 3) Let the system choose the best Gemini 3 model for your task.

Auto (Gemini 2.5) Let the system choose the best Gemini 2.5 model for your task.

Manual Select a specific model.

Updated to use flash for nothink, pro for think/max

^{｜ Triggered by Pullfrog ｜ Using Claude Code ｜ View workflow run ｜ pullfrog.com ｜ 𝕏}

pullfrog · 2026-01-10T18:01:26Z

agents/claude.ts

+// model configuration based on effort level
+// uses model family aliases that auto-resolve to latest version
+const claudeModels: Record<Effort, { model: string; thinking: boolean }> = {
+  nothink: { model: "claude-haiku-4-5", thinking: false },


These model names use version suffixes like claude-haiku-4-5 that don't match Anthropic's actual model naming. Current Claude models use formats like claude-3-5-sonnet-20241022, claude-3-5-haiku-20241022, etc. Verify these are valid model identifiers or this will fail when invoking the Anthropic API.

For Claude Code do not update model, the best model is already selected automatically. Use "effort" only

https://platform.claude.com/docs/en/build-with-claude/effort

Updated to use effort parameter only, removed model selection

^{｜ Triggered by Pullfrog ｜ Using Claude Code ｜ View workflow run ｜ pullfrog.com ｜ 𝕏}

pullfrog · 2026-01-10T18:01:26Z

main.ts

+    // internal invocation: use effort from payload, fallback to input
+    return {
+      ...parsedPrompt,
+      effort: parsedPrompt.effort ?? inputs.effort,


The effort fallback logic is inconsistent. For internal invocations, you use parsedPrompt.effort ?? inputs.effort, but for external invocations you use inputs.effort ?? "think". If inputs.effort is undefined in both cases, internal gets undefined while external gets "think". This could cause the agent to receive undefined for effort. Consider: effort: parsedPrompt.effort ?? inputs.effort ?? "think"

Added consistent fallback to "think" for both branches

^{｜ Triggered by Pullfrog ｜ Using Claude Code ｜ View workflow run ｜ pullfrog.com ｜ 𝕏}

pullfrog · 2026-01-10T18:01:26Z

agents/opencode.ts

    for (const [key, value] of Object.entries(apiKeys || {})) {
-      const upperKey = key.toUpperCase();
-      env[upperKey] = value;
+      env[key] = value;


This change removes the toUpperCase() call on API key names. The previous code normalized keys to uppercase (e.g., ANTHROPIC_API_KEY), but now it preserves the original case. Since the keys in apiKeys come from agent.apiKeyNames which are already uppercase in external.ts, this should be fine. However, verify that OpenCode accepts both GEMINI_API_KEY and GOOGLE_GENERATIVE_AI_API_KEY in the exact case provided.

Add this back for safety

Added back toUpperCase() for safety

^{｜ Triggered by Pullfrog ｜ Using Claude Code ｜ View workflow run ｜ pullfrog.com ｜ 𝕏}

colinhacks

cc @pullfrog address these reviews according to my comments and update the PR

colinhacks · 2026-01-12T20:34:39Z

agents/codex.ts

+// uses model family aliases that auto-resolve to latest version
+const codexModels: Record<Effort, string> = {
+  nothink: "gpt-4o-mini",
+  think: "gpt-5.2-instant",


Update to use model_reasoning_effort

model_reasoning_effort minimal | low | medium | high | xhigh Adjust reasoning effort for supported models (Responses API only; xhigh is model-dependent).

"low" for "nothink"
"xhigh" for "max"

colinhacks · 2026-01-12T20:37:22Z

agents/gemini.ts

+// uses model family aliases that auto-resolve to latest version
+const geminiModels = {
+  nothink: "gemini-2.5-pro",
+  think: "gemini-3-pro",


Deafult to auto (no specification) for "think"

Then force "flash" for nothink and "pro" for "max"

Description Models

Auto (Gemini 3) Let the system choose the best Gemini 3 model for your task.

Auto (Gemini 2.5) Let the system choose the best Gemini 2.5 model for your task.

Manual Select a specific model.

colinhacks · 2026-01-12T20:37:41Z

agents/opencode.ts

    for (const [key, value] of Object.entries(apiKeys || {})) {
-      const upperKey = key.toUpperCase();
-      env[upperKey] = value;
+      env[key] = value;


Add this back for safety

colinhacks · 2026-01-12T20:38:54Z

main.ts

+    // internal invocation: use effort from payload, fallback to input
+    return {
+      ...parsedPrompt,
+      effort: parsedPrompt.effort ?? inputs.effort,


colinhacks · 2026-01-12T20:40:10Z

agents/claude.ts

+// model configuration based on effort level
+// uses model family aliases that auto-resolve to latest version
+const claudeModels: Record<Effort, { model: string; thinking: boolean }> = {
+  nothink: { model: "claude-haiku-4-5", thinking: false },


For Claude Code do not update model, the best model is already selected automatically. Use "effort" only

https://platform.claude.com/docs/en/build-with-claude/effort

colinhacks · 2026-01-12T20:48:41Z

cc @pullfrog address all my comments

pullfrog · 2026-01-12T20:48:49Z

Addressed all 5 review comments and pushed changes.

^{｜ Triggered by Pullfrog ｜ Using Claude Code ｜ View workflow run ｜ pullfrog.com ｜ 𝕏}

…ty checks

ssalbdivad requested a review from colinhacks January 10, 2026 18:00

pullfrog bot reviewed Jan 10, 2026

View reviewed changes

colinhacks reviewed Jan 12, 2026

View reviewed changes

ssalbdivad and others added 5 commits January 12, 2026 14:12

init

c335032

fix play

89e93d3

Address review feedback: use effort params, fix model names, add safe…

c6572f0

…ty checks

Fixes

45f837c

Update claude impl

edb7603

colinhacks force-pushed the effort branch from 4ce831b to edb7603 Compare January 12, 2026 22:12

colinhacks merged commit 9903072 into main Jan 12, 2026
1 check passed

Description	Models
Auto (Gemini 3)	Let the system choose the best Gemini 3 model for your task.
Auto (Gemini 2.5)	Let the system choose the best Gemini 2.5 model for your task.
Manual	Select a specific model.

add effort as an input + support parsing from payload #21

add effort as an input + support parsing from payload #21

Conversation

ssalbdivad commented Jan 10, 2026

Effort Input & Environment-Based API Keys

Summary of Changes

1. New effort Input Parameter

2. API Keys via Environment Variables (Breaking Change)

3. Effort Parsing from Payload

4. Agent Model Configuration

5. Updated Runtime to Node 24

Files Changed

Migration Guide

Required Workflow Updates

Testing

Uh oh!

pullfrog bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

colinhacks Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pullfrog bot Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

colinhacks left a comment

Choose a reason for hiding this comment

Uh oh!

colinhacks Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

colinhacks commented Jan 12, 2026

Uh oh!

pullfrog bot commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

1. New `effort` Input Parameter

pullfrog bot left a comment •

edited

Loading

colinhacks Jan 12, 2026 •

edited

Loading

colinhacks Jan 12, 2026 •

edited

Loading

pullfrog bot commented Jan 12, 2026 •

edited

Loading