feat: integrate mocker with dynamo-run and python cli

[PeaBrane]

Overview:

Integration with dynamo-run: Allow out=mocker for dynamo-run and read mocker configs from --extra-engine-args. Update dynamo-run guide docs

Integration with python cli: Add Mocker as an EngineType to entrypoint.rs, and a thin Python wrapper to start the drt and pass in the args in components/backend/mocker

A continuation of #1236, and tested again to be functional with Kv routing

Summary by CodeRabbit

• New Features

  • Added support for a "Mocker" engine, allowing users to simulate token generation for testing and development without requiring GPUs.
  • Users can configure the mocker engine using a JSON file with options such as token generation speed, number of simulated workers, and cache thresholds.

• Documentation

  • Introduced a new guide section explaining how to use the mocker engine, including configuration details and usage examples.

• Enhancements

  • Improved logging to display detailed configuration information when starting the mocker engine.

[coderabbitai]

Walkthrough

This change integrates a mocker engine into the dynamo-run CLI and runtime, enabling configuration via JSON files and supporting a new mocker output type. It updates CLI flags, engine creation logic, and documentation to describe and support the mocker engine for testing and benchmarking without real model inference. No core inference logic is altered.

Changes

File(s)	Change Summary
docs/guides/dynamo_run.md	Added documentation section detailing the mocker engine, its purpose, configuration, and usage examples.
launch/dynamo-run/src/opt.rs	Added Mocker variant to Output enum; updated string conversions and available engines list to include mocker.
launch/dynamo-run/src/flags.rs	Replaced JSON loading method with mocker_config; added logic for loading and validating mocker config; updated validation.
launch/dynamo-run/src/lib.rs	Added support for Output::Mocker in engine creation; updated runtime handling to support distributed runtime for mocker.
lib/llm/src/mocker/engine.rs	Improved logging to include mocker engine configuration details during creation.
lib/llm/src/mocker/protocols.rs	Added Default and from_json_file to MockEngineArgs for safe, validated config loading from JSON files.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant CLI
    participant Flags
    participant EngineFactory
    participant DistributedRuntime
    participant MockerEngine

    User->>CLI: Run with --output mocker and config JSON
    CLI->>Flags: Parse flags and load mocker config
    Flags->>MockEngineArgs: Load and validate config from JSON
    CLI->>EngineFactory: Request engine_for(..., Output::Mocker, ...)
    EngineFactory->>DistributedRuntime: Ensure distributed runtime is available
    EngineFactory->>MockerEngine: Create mocker engine with config
    MockerEngine-->>EngineFactory: Engine instance
    EngineFactory-->>CLI: Return engine

Loading

Possibly related PRs

• feat: vllm mocker enhancement #1236: Implements core enhancements and bug fixes to the mocker engine’s internal scheduling, block management, and runtime integration, directly supporting the integration and exposure of the mocker engine in this PR.

Poem

In Dynamo’s warren, a mocker appears,
Simulating tokens, without GPU fears.
With configs from JSON, it’s easy to test,
For benchmarking, debugging, or load at your behest.
No real models needed, just hop in and play—
The mocker’s here to brighten your day!
🐇✨

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6a1350c and 59b3d6c.

📒 Files selected for processing (6)

• docs/guides/dynamo_run.md (1 hunks)
• launch/dynamo-run/src/flags.rs (3 hunks)
• launch/dynamo-run/src/lib.rs (6 hunks)
• launch/dynamo-run/src/opt.rs (4 hunks)
• lib/llm/src/mocker/engine.rs (1 hunks)
• lib/llm/src/mocker/protocols.rs (2 hunks)

🧰 Additional context used

🧠 Learnings (6)

📓 Common learnings

Learnt from: PeaBrane
PR: ai-dynamo/dynamo#1285
File: lib/llm/src/kv_router/scheduler.rs:260-266
Timestamp: 2025-05-30T06:34:12.785Z
Learning: In the KV router scheduler code, PeaBrane prefers fail-fast behavior over silent failure handling. When accessing worker metrics data that could be out-of-bounds (like dp_rank indexing), explicit panics are preferred over graceful degradation with continue statements to ensure data integrity issues are caught early.

lib/llm/src/mocker/engine.rs (1)

Learnt from: ryanolson
PR: ai-dynamo/dynamo#1919
File: lib/runtime/src/engine.rs:168-168
Timestamp: 2025-07-14T21:25:56.898Z
Learning: The AsyncEngineContextProvider trait in lib/runtime/src/engine.rs was intentionally changed from `Send + Sync + Debug` to `Send + Debug` because the Sync bound was overly constraining. The trait should only require Send + Debug as designed.

launch/dynamo-run/src/opt.rs (1)

Learnt from: ryanolson
PR: ai-dynamo/dynamo#1919
File: lib/runtime/src/engine.rs:168-168
Timestamp: 2025-07-14T21:25:56.898Z
Learning: The AsyncEngineContextProvider trait in lib/runtime/src/engine.rs was intentionally changed from `Send + Sync + Debug` to `Send + Debug` because the Sync bound was overly constraining. The trait should only require Send + Debug as designed.

lib/llm/src/mocker/protocols.rs (3)

Learnt from: PeaBrane
PR: ai-dynamo/dynamo#1236
File: lib/llm/src/mocker/protocols.rs:85-112
Timestamp: 2025-06-16T20:02:54.935Z
Learning: When using derive_builder::Builder macro, the macro generates the builder struct and its methods, but does NOT generate a `builder()` method on the original struct. A manual `impl StructName { pub fn builder() -> StructNameBuilder { StructNameBuilder::default() } }` is required to provide the convenient `StructName::builder()` API pattern.

Learnt from: ryanolson
PR: ai-dynamo/dynamo#1919
File: lib/runtime/src/engine.rs:168-168
Timestamp: 2025-07-14T21:25:56.898Z
Learning: The AsyncEngineContextProvider trait in lib/runtime/src/engine.rs was intentionally changed from `Send + Sync + Debug` to `Send + Debug` because the Sync bound was overly constraining. The trait should only require Send + Debug as designed.

Learnt from: alec-flowers
PR: ai-dynamo/dynamo#1181
File: lib/llm/src/kv_router/publisher.rs:379-425
Timestamp: 2025-05-29T00:02:35.018Z
Learning: In lib/llm/src/kv_router/publisher.rs, the functions `create_stored_blocks` and `create_stored_block_from_parts` are correctly implemented and not problematic duplications of existing functionality elsewhere in the codebase.

launch/dynamo-run/src/flags.rs (2)

Learnt from: PeaBrane
PR: ai-dynamo/dynamo#1392
File: lib/llm/src/kv_router/scoring.rs:35-46
Timestamp: 2025-06-05T01:02:15.318Z
Learning: In lib/llm/src/kv_router/scoring.rs, PeaBrane prefers panic-based early failure over Result-based error handling for the worker_id() method to catch invalid data early during development.

Learnt from: PeaBrane
PR: ai-dynamo/dynamo#1285
File: lib/llm/src/kv_router/scoring.rs:58-63
Timestamp: 2025-05-30T06:38:09.630Z
Learning: In lib/llm/src/kv_router/scoring.rs, the user prefers to keep the panic behavior when calculating load_avg and variance with empty endpoints rather than adding guards for division by zero. They want the code to fail fast on this error condition.

launch/dynamo-run/src/lib.rs (2)

Learnt from: ryanolson
PR: ai-dynamo/dynamo#1919
File: lib/runtime/src/engine.rs:168-168
Timestamp: 2025-07-14T21:25:56.898Z
Learning: The AsyncEngineContextProvider trait in lib/runtime/src/engine.rs was intentionally changed from `Send + Sync + Debug` to `Send + Debug` because the Sync bound was overly constraining. The trait should only require Send + Debug as designed.

Learnt from: nnshah1
PR: ai-dynamo/dynamo#1444
File: tests/fault_tolerance/utils/metrics.py:30-32
Timestamp: 2025-07-01T13:55:03.940Z
Learning: The `@dynamo_worker()` decorator in the dynamo codebase returns a wrapper that automatically injects the `runtime` parameter before calling the wrapped function. This means callers only need to provide the non-runtime parameters, while the decorator handles injecting the runtime argument automatically. For example, a function with signature `async def get_metrics(runtime, log_dir)` decorated with `@dynamo_worker()` can be called as `get_metrics(log_dir)` because the decorator wrapper injects the runtime parameter.

🧬 Code Graph Analysis (1)

launch/dynamo-run/src/flags.rs (1)

lib/llm/src/mocker/protocols.rs (2)

• default (103-107)
• from_json_file (116-206)

🔇 Additional comments (12)

lib/llm/src/mocker/engine.rs (1)

516-516: LGTM: Enhanced logging improves debugging visibility.

The addition of configuration details to the logging output is a valuable improvement for debugging and operational transparency.

launch/dynamo-run/src/opt.rs (1)

35-35: LGTM: Complete and consistent integration of the Mocker output type.

All necessary trait implementations are properly updated to support the new Mocker variant.

Also applies to: 52-52, 85-85, 113-113

lib/llm/src/mocker/protocols.rs (1)

102-108: LGTM: Default implementation follows correct builder pattern.

The implementation properly uses the builder's defaults and builds the instance correctly.

docs/guides/dynamo_run.md (1)

541-566: LGTM: Comprehensive and clear documentation for the mocker engine.

The documentation effectively explains the purpose, configuration options, and usage patterns for the new mocker engine. The examples and notes about current limitations provide valuable user guidance.

launch/dynamo-run/src/flags.rs (3)

22-22: LGTM: Proper import addition for MockEngineArgs.

The import is correctly added to support the new mocker configuration functionality.

---

215-217: LGTM: Appropriate validation handling for mocker engine.

No validation is needed for the mocker engine, so the empty case is correct.

---

234-241: LGTM: Configuration loading aligns with fail-fast preferences.

The method properly:

• Provides default configuration with warning when no path is specified
• Uses the typed MockEngineArgs::from_json_file method for robust loading
• Employs panic on failure, which aligns with the user's preference for fail-fast behavior from the retrieved learnings

launch/dynamo-run/src/lib.rs (5)

12-12: LGTM: Clean import addition.

The addition of DistributedRuntime to the existing import is clean and necessary for the mocker integration.

---

52-58: Well-structured runtime initialization logic.

The conditional creation of DistributedRuntime for endpoint inputs is logical and follows a clear pattern. The Either wrapper provides a clean abstraction for handling both runtime types.

---

73-80: Function call updated correctly.

The engine_for call is properly updated to pass the runtime wrapper, maintaining consistency with the signature changes.

---

105-105: Function signature updated appropriately.

The signature change to accept Either<Runtime, DistributedRuntime> is necessary and well-designed to support both runtime types.

---

171-189: Mocker implementation follows established patterns.

The implementation correctly:

• Uses fail-fast behavior with explicit panic (aligning with your documented preferences)
• Extracts configuration via flags.mocker_config()
• Follows the same EngineConfig::StaticCore pattern as other engines
• Properly handles async operations and error propagation

The panic message clearly communicates the requirement for distributed runtime.

[grahamking]

@PeaBrane dynamo-run is being removed. The new approach is:

• Ingress node: python -m dynamo.ingress --http-port <port> feat: Python frontend / ingress node #1912
• Worker: python -m dynamo.llama_cpp --model-path /data/models/Qwen3-0.6B-Q8_0.gguf feat(backends): Python llama.cpp engine #1925

The means you should put mocker in components/backends/mocker and ensure it can be run like this:

python -m dynamo.mocker [--your-args]

The first step would be making mocker a library and giving it a Python binding.

[PeaBrane]

@grahamking Is there any reason why we are not supporting dynamo-run anymore? I think it had really good ergonomics for dev as we can introduce new components without having Python bindings (e.g. a new router, mocker, recorder etc). It feels a bit awkward if we rely completely on register_llm for the future