feat: mock AI provider for run route testing without API keys

[nick-inkeep]

Summary

• Adds a built-in mock AI provider implementing LanguageModelV2 for testing run routes without real API keys
• Registered in ModelFactory as mock/{modelName} alongside anthropic, openai, google, etc.
• 19 unit tests covering interface compliance, streaming, ModelFactory integration, and edge cases
• Docs updated in models.mdx

Motivation

Run domain endpoints (/v1/chat/completions, /api/chat, A2A) require a real Anthropic/OpenAI API key to produce responses. This blocks:

• Local dev pipeline validation after pnpm setup-dev
• CI/CD integration tests for run routes
• Quick health checks on the run API without incurring LLM costs

Approach

Mock provider (packages/agents-core/src/utils/mock-provider.ts):

• Implements LanguageModelV2 from @ai-sdk/provider
• doGenerate: returns deterministic text with model name, message count, last user message, timestamp
• doStream: streams response line-by-line with small delays simulating real streaming
• Synthetic token usage (chars/4)
• No API keys required

ModelFactory integration (packages/agents-core/src/utils/model-factory.ts):

• Added mock to BUILT_IN_PROVIDERS
• mock/default, mock/anything all resolve via createMockModel()

Test plan

19 unit tests in packages/agents-core/src/__tests__/utils/mock-provider.test.ts:

• LanguageModelV2 interface compliance (specificationVersion, provider, modelId)
• Any model name accepted under mock/ prefix
• Structured mock response with all fields
• Message count reflects prompt length
• Synthetic token usage (inputTokens + outputTokens = totalTokens)
• Long messages truncated to 200 chars
• No user message → "(no user message)"
• Custom model names in response
• Streaming: multiple chunks with correct part types
• Streaming: line-by-line deltas with newline separators
• Streaming: conversation history message count
• ModelFactory.parseModelString for mock/
• ModelFactory.createModel produces working mock model
• Mock model works without API keys
• Mock model works with providerOptions present
• Unsupported providers rejected
• validateConfig passes for mock/default

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agents-api	Ready	Preview, Comment	Feb 17, 2026 1:45pm
agents-docs	Ready	Preview, Comment	Feb 17, 2026 1:45pm
agents-manage-ui	Ready	Preview, Comment	Feb 17, 2026 1:45pm

[changeset-bot]

🦋 Changeset detected

Latest commit: b6a707b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 10 packages

Name	Type
@inkeep/agents-core	Patch
@inkeep/agents-api	Patch
@inkeep/agents-manage-ui	Patch
@inkeep/agents-cli	Patch
@inkeep/agents-sdk	Patch
@inkeep/agents-work-apps	Patch
@inkeep/ai-sdk-provider	Patch
@inkeep/create-agents	Patch
@inkeep/agents-manage-mcp	Patch
@inkeep/agents-mcp	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[claude]

PR Review Summary

(2) Total Issues | Risk: Medium

🟠⚠️ Major (1) 🟠⚠️

Inline Comments:

• 🟠 Major: model-factory.ts:206 Echo provider throws "Unsupported provider: echo" when providerOptions are supplied

🟡 Minor (1) 🟡

Inline Comments:

• 🟡 Minor: echo-provider.test.ts:322-339 Production warning test doesn't verify logger was actually called

💭 Consider (5) 💭

💭 1) echo-provider.ts / index.ts Public API surface expansion

Issue: The echo provider is exported via the utils/index.ts barrel, making createEchoModel and EchoLanguageModel part of the public @inkeep/agents-core API.
Why: While internally-facing, this expands the public contract surface. Any breaking changes to the echo provider shape would technically be breaking changes to the core package API.
Fix: Consider either: (1) not exporting from the barrel file and importing directly in model-factory.ts, or (2) adding a JSDoc comment marking it as @internal or @experimental.
Refs: utils/index.ts:6

💭 2) echo-provider.test.ts Test style inconsistency

Issue: The echo-provider tests use it() syntax while the adjacent model-factory.test.ts uses test() syntax.
Why: While functionally equivalent, maintaining consistent test style within the same directory aids readability.
Fix: Consider changing it to test throughout to match model-factory.test.ts conventions.
Refs: model-factory.test.ts:1-2

💭 3) model-factory.ts:206 Type cast through unknown

Issue: The double cast as unknown as LanguageModel bypasses TypeScript's structural type checking entirely.
Why: If the ai package's LanguageModel type adds required properties in a future version, TypeScript won't catch the incompatibility at compile time. Currently low-risk since both types are from the same AI SDK ecosystem.
Fix: The current approach is pragmatic. Consider adding an integration test that explicitly verifies the echo model works with AI SDK's streamText/generateText functions.
Refs: model-factory.ts:53-55 — similar cast for Azure

💭 4) package.json:137 Version pinning inconsistency

Issue: The new @ai-sdk/provider dependency uses exact pinning ("3.0.2") while some siblings like @ai-sdk/azure use caret (^3.0.4).
Why: Exact pinning is safer for reproducibility but creates more maintenance burden for coordinated upgrades.
Fix: Consider whether all @ai-sdk/* packages should use consistent version pinning strategy.
Refs: package.json:137

💭 5) models.mdx:306-308 Tip placement

Issue: The echo provider Tip is placed after the Provider Options SkillRule section (~230 lines after the models table where echo is introduced).
Why: Users scanning the models table may not discover the Tip explaining echo's purpose. The placement after Provider Options implies echo has special options to configure, but it actually has none.
Fix: Consider moving the Tip closer to the models table (after line 79), or adding a brief inline note in the table description column.
Refs: models.mdx:77

---

💡 APPROVE WITH SUGGESTIONS

Summary: The echo provider implementation is well-designed and follows existing patterns nicely. The main issue to address is the missing echo case in createProvider() — when users pass any providerOptions (even unrelated ones), the echo provider will crash with "Unsupported provider: echo". This is a straightforward fix. The test coverage is comprehensive (20 tests), the documentation is accurate, and the dependency addition is justified. Once the createProvider case is added, this is ready to ship! 🚀

Discarded (8)

Location	Issue	Reason Discarded
echo-provider.ts:50-56	Timestamp variability in response	Intentional per spec — "deterministic" refers to structure, not exact content
PR description	Lists integration tests T1-T8 as implemented	Documentation gap in PR body, not code issue; spec clearly defers these
echo-provider.ts	LanguageModelV2 interface implementation	Validated as consistent with InkeepChatLanguageModel pattern
utils/index.ts	Export barrel pattern	Validated as following alphabetical order convention
models.mdx:77	Table row format	Validated as consistent with other provider entries
package.json	Dependency justified	@ai-sdk/provider is already a transitive dep, making explicit is correct
pnpm-lock.yaml	Lockfile change	Proportionate to package.json edit, only 3 lines added
.changeset/	Changeset correctness	Correctly scoped to agents-core with patch bump

Reviewers (7)

Reviewer	Returned	Main Findings	Consider	While You're Here	Inline Comments	Pending Recs	Discarded
pr-review-standards	1	0	0	0	1	0	0
pr-review-consistency	6	0	1	0	0	0	5
pr-review-product	4	0	1	0	0	0	3
pr-review-tests	2	0	0	0	1	0	1
pr-review-types	1	0	1	0	0	0	0
pr-review-devops	5	0	1	0	0	0	4
pr-review-docs	1	0	1	0	0	0	0
Total	20	0	5	0	2	0	13

Note: The MAJOR finding was identified by both pr-review-standards and pr-review-consistency (merged into single inline comment).

[inkeep]

📚 Documentation Review

The documentation updates in models.mdx look good:

• ✅ Echo provider added to the supported models table with "None required" for API key
• ✅ Tip block clearly explains the use case for testing and CI/CD

No additional documentation updates needed. The echo provider is appropriately documented in the model configuration reference where users would naturally look for this information.

[claude]

PR Review Summary

(0) Total Issues | Risk: Low

This is a delta re-review following commit 1dd97408 which addressed the feedback from my prior review.

Prior Issues — Now Resolved

Issue	Status	Resolution
🟠 MAJOR: Echo provider throws when providerOptions supplied	✅ Resolved	Added provider !== 'echo' guard at line 186 to bypass createProvider()
🟡 Minor: Production warning test lacks assertion	✅ Resolved	Added vi.spyOn() on logProductionWarning and expect(logSpy).toHaveBeenCalled()

Delta Changes Reviewed

+ if (provider !== 'echo' && (provider === 'azure' || Object.keys(providerConfig).length > 0)) {

The guard correctly ensures echo models always route to the default switch case (line 205-206), avoiding the createProvider() path that lacks an echo case.

New test coverage:

it('should create echo model even when providerOptions are present', () => {
  const model = ModelFactory.createModel({
    model: 'echo/default',
    providerOptions: { baseURL: 'https://example.com' },
  });
  expect(model).toBeDefined();
});

This validates the fix and guards against regression.

---

✅ APPROVE

Summary: Both issues from the prior review have been cleanly resolved. The echo provider is now production-ready for its intended use case (testing and CI/CD without API keys). Ship it! 🚀

Reviewers (0)

Reviewer	Returned	Main Findings	Consider	While You're Here	Inline Comments	Pending Recs	Discarded
Total	0	0	0	0	0	0	0

Note: Delta re-review with minimal scope — subagent dispatch skipped. Orchestrator reviewed delta directly.

[nick-inkeep]

Review feedback addressed

Resolved (MAJOR + MINOR):
Both inline threads resolved — the provider !== 'echo' guard in model-factory.ts:186 and the logProductionWarning spy in the production warning test. Confirmed by the delta re-review.

Consider items assessment:

1. Public API surface expansion — Declined. The barrel export follows existing patterns; ModelFactory imports from the barrel. Marking @internal adds JSDoc churn with no enforcement benefit.

2. Test style it() vs test() — Declined. Both are standard Vitest aliases used across the repo. Not worth a churn commit.

3. Type cast through unknown — Declined. Same pattern used for Azure (model-factory.ts:53). Pragmatic and acknowledged by reviewer.

4. Version pinning inconsistency — Declined. @ai-sdk/provider is a type-only import; exact pinning is intentional. Aligning all @ai-sdk/* pinning is a separate concern.

5. Tip placement in docs — Accepted. Moved the echo provider tip from after the Provider Options section (line 306) to right after the models table (line 80). Commit ff3f76da0.

[claude]

PR Review Summary

(0) Total Issues | Risk: Low

This is a delta re-review following commits 1cc477b0 and fcb3af8e which are pure housekeeping changes.

Delta Changes Reviewed

Commits:

• 1cc477b0 chore: gitignore .claude/specs/ and ship-state.json
• fcb3af8e chore: remove spec file from tracked files

Changes:

1. Added .claude/specs/ and .claude/ship-state.json to .gitignore
2. Removed .claude/specs/echo-ai-provider.md from version control

Analysis

These are purely administrative changes with no product or code impact:

• The spec file was an internal design document used during implementation
• Gitignoring local spec files and ship state is standard practice for working artifacts
• No code changes, no test changes, no documentation changes

Prior Review Status

Review	Status	Notes
Initial review (2026-02-17)	✅	2 issues raised (1 Major, 1 Minor)
Delta re-review #1	✅ Approved	Both issues resolved
Delta re-review #2	✅ Approved	Docs improvement only
Delta re-review #3	✅ Approved	Removed production warning (acceptable)
Delta re-review #4 (this)	✅	Housekeeping only

---

✅ APPROVE

Summary: Pure housekeeping delta — removing internal spec file from version control and gitignoring local working artifacts. The echo AI provider implementation is complete and ready to ship. All prior issues resolved, comprehensive test coverage (20+ tests), clean LanguageModelV2 implementation, proper ModelFactory integration. 🚀

Reviewers (0)

Reviewer	Returned	Main Findings	Consider	While You're Here	Inline Comments	Pending Recs	Discarded
Total	0	0	0	0	0	0	0

Note: Trivial delta (housekeeping) — subagent dispatch skipped. Orchestrator reviewed delta directly.

[claude]

PR Review Summary

(0) Total Issues | Risk: Low

This is a delta re-review following commits e5ea640a and fcb3af8e which rename the provider from "echo" to "mock".

Delta Changes Reviewed

Commits since last review:

• e5ea640a refactor: rename echo provider to mock provider
• fcb3af8e chore: remove spec file from tracked files
• aaa9b8d7 fix: increase browser screenshot test timeouts (unrelated to this PR)

Rename scope:

Before	After
echo-provider.ts	mock-provider.ts
EchoLanguageModel	MockLanguageModel
createEchoModel()	createMockModel()
echo/default	mock/default
provider !== 'echo'	provider !== 'mock'

Verification:

• ✅ All file renames applied correctly
• ✅ Class/function names updated throughout
• ✅ Model strings updated (echo/ → mock/)
• ✅ ModelFactory guard updated at line 186
• ✅ BUILT_IN_PROVIDERS updated
• ✅ Documentation (models.mdx) updated
• ✅ Changeset message updated
• ✅ Barrel export updated (index.ts)
• ✅ No residual "echo" references in source code (verified via grep)
• ✅ Test file renamed and all assertions updated

Analysis

The rename from "echo" to "mock" is a sensible naming improvement:

• "Mock" is more conventional terminology for test doubles in software engineering
• The provider's purpose (deterministic responses for testing) aligns better with "mock" semantics
• All 20+ tests pass with the updated naming

No functional changes were introduced — this is a pure refactoring commit.

Prior Review Status

Review	Status	Notes
Initial review (2026-02-17)	✅	2 issues raised (1 Major, 1 Minor)
Delta re-review #1	✅ Approved	Both issues resolved
Delta re-review #2	✅ Approved	Docs improvement only
Delta re-review #3	✅ Approved	Removed production warning
Delta re-review #4	✅ Approved	Housekeeping only
Delta re-review #5 (this)	✅	Pure rename — echo → mock

---

✅ APPROVE

Summary: Clean rename from "echo" to "mock" provider — all references updated consistently across source, tests, docs, and changeset. The mock AI provider is ready to ship for its intended use case (testing and CI/CD without API keys). 🎉

Reviewers (0)

Reviewer	Returned	Main Findings	Consider	While You're Here	Inline Comments	Pending Recs	Discarded
Total	0	0	0	0	0	0	0

Note: Trivial delta (pure rename) — subagent dispatch skipped. Orchestrator reviewed delta directly.

[github-actions]

🔎💬 Inkeep AI search and chat service is syncing content for source 'Inkeep Agent Framework Docs'

[claude]

PR Review Summary

(0) Total Issues | Risk: Low

This is a delta re-review following commit b6a707bf which simplifies the mock provider documentation.

Delta Changes Reviewed

Commit: b6a707bf docs: lighten mock provider entry in models table

Changes:

• Simplified the models table row from mock/default
mock/any-name to just mock/default
• Removed the (testing) suffix from the provider name
• Removed the Tip block explaining the mock provider's purpose

Analysis

This is a reasonable documentation cleanup:

• The mock provider accepts any model name (mock/anything works), but documenting just mock/default is sufficient for typical use
• "None required" for the API key column already signals this is a testing utility
• The PR description and changeset provide adequate context about the mock provider's purpose
• Users who need more details can check the source or use mock/default and discover the deterministic response format

The change aligns with keeping documentation concise — the Tip was useful when placed far from the table, but after the prior move (closer to the table) and now with a single clean row, the extra explanation is redundant.

Prior Review Status

Review	Status	Notes
Initial review (2026-02-17)	✅	2 issues raised (1 Major, 1 Minor)
Delta re-review #1	✅ Approved	Both issues resolved
Delta re-review #2	✅ Approved	Docs improvement (tip placement)
Delta re-review #3	✅ Approved	Removed production warning
Delta re-review #4	✅ Approved	Housekeeping (.gitignore)
Delta re-review #5	✅ Approved	echo → mock rename
Delta re-review #6 (this)	✅	Docs simplification

---

✅ APPROVE

Summary: Clean documentation simplification — the mock provider entry is now minimal and consistent with other providers in the table. The mock AI provider implementation is complete with comprehensive test coverage (20+ tests), proper LanguageModelV2 implementation, and ModelFactory integration. Ready to ship! 🚀

Reviewers (0)

Reviewer	Returned	Main Findings	Consider	While You're Here	Inline Comments	Pending Recs	Discarded
Total	0	0	0	0	0	0	0

Note: Trivial delta (docs cleanup) — subagent dispatch skipped. Orchestrator reviewed delta directly.