feat: support HF_HOME/_ENDPOINT env for Hugging Face models

[hhk7734]

Overview:

This PR adds support for the HF_HOME and HF_ENDPOINT environment variables, allowing the use of already downloaded models or a Hugging Face cache server.

Details:

Replaced hf_hub::api::tokio::ApiBuilder::new() with hf_hub::api::tokio::ApiBuilder::from_env().

Where should the reviewer start?

lib/llm/src/hub.rs

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: [FEATURE]: Dynamo should support HF_ENDPOINT while connects to huggingface. #2631

Test

Before this PR, DecodeWorkers did not work. After this change, the following tests pass.

hf_home_test.yaml

apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: pd
  namespace: dynamo-cloud
spec:
  envs:
    - name: HF_HOME
      value: /models
    - name: HF_HUB_OFFLINE
      value: "1"
  services:
    # ...
    VllmDecodeWorker:
      # ...
      pvc:
        name: models
        mountPoint: /models
      extraPodSpec:
        mainContainer:
          image: 192.168.212.30:1780/moreh/hyeonki/dynamo:v0.4.0.3
          command:
            - python3
            - -m
            - dynamo.vllm
          args:
            - --model
            - meta-llama/Meta-Llama-3-8B
    VllmPrefillWorker:
      # ...
      pvc:
        name: models
        mountPoint: /models
      extraPodSpec:
        mainContainer:
          image: 192.168.212.30:1780/moreh/hyeonki/dynamo:v0.4.0.3
          command:
            - python3
            - -m
            - dynamo.vllm
          args:
            - --model
            - meta-llama/Meta-Llama-3-8B
            - --is-prefill-worker

hf_endpoint_test.yaml

apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: pd
  namespace: dynamo-cloud
spec:
  envs:
    - name: HF_ENDPOINT
      value: http://192.168.212.30:8090
  services:
    # ...
    VllmDecodeWorker:
      # ...
      extraPodSpec:
        mainContainer:
          image: 192.168.212.30:1780/moreh/hyeonki/dynamo:v0.4.0.3
          command:
            - python3
            - -m
            - dynamo.vllm
          args:
            - --model
            - meta-llama/Meta-Llama-3-8B
    VllmPrefillWorker:
      # ...
      extraPodSpec:
        mainContainer:
          image: 192.168.212.30:1780/moreh/hyeonki/dynamo:v0.4.0.3
          command:
            - python3
            - -m
            - dynamo.vllm
          args:
            - --model
            - meta-llama/Meta-Llama-3-8B
            - --is-prefill-worker

Summary by CodeRabbit

• Refactor
  • Switched external API client initialization to automatically read configuration from environment variables.
  • Preserves existing behavior (e.g., progress display and authentication) while aligning with env-based setup.
  • No changes to public interfaces or user-facing commands.
• Chores
  • Internal construction streamlined for consistency with environment-driven configuration.
  • Expected to reduce manual setup friction without impacting current workflows.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi hhk7734! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

Walkthrough

Updated Hugging Face client initialization in lib/llm/src/hub.rs::from_hf by replacing ApiBuilder::new() with ApiBuilder::from_env(), keeping subsequent builder methods unchanged. No public API signatures were altered.

Changes

Cohort / File(s)	Summary
HF API initialization lib/llm/src/hub.rs	Switched client construction from ApiBuilder::new() to ApiBuilder::from_env() while retaining .with_progress(true).with_token(token).high().build()? chain. No exported signatures changed.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Caller
  participant from_hf as from_hf()
  participant Builder as ApiBuilder
  participant HF as HuggingFace API

  Caller->>from_hf: request client
  from_hf->>Builder: from_env()
  Note right of Builder: Preload env defaults<br/>(e.g., base URL, proxies)
  from_hf->>Builder: with_progress(true)<br/>with_token(token)<br/>high()
  from_hf->>Builder: build()
  Builder-->>from_hf: Api client
  from_hf-->>Caller: client
  Caller->>HF: use client

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• fix(dynamo-llm): Use HF_TOKEN env var #1249 — Adjusts from_hf to use ApiBuilder env-driven construction and token handling in the same file, indicating a direct correspondence.

Poem

A hop to the hub, a twitch of the ear,
From env we sip configs, crisp and clear.
Tokens tucked snug, progress in tow,
High we build, then off we go.
Thump-thump—client ready, onward, ho! 🐇✨

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

lib/llm/src/hub.rs (1)

44-49: Guard against clearing tokens discovered by from_env().
Calling .with_token(None) will wipe out any token already loaded by ApiBuilder::from_env() (e.g. from the HuggingFace CLI’s token file). Only override when HF_TOKEN is explicitly set.

Please update lib/llm/src/hub.rs (around lines 44–49) as follows:

@@ -42,11 +42,16 @@ pub async fn from_hf(name: impl AsRef<Path>, ignore_weights: bool) -> anyhow::Result<PathBuf> {
-    let token = env::var(HF_TOKEN_ENV_VAR).ok();
-    let api = ApiBuilder::from_env()
-        .with_progress(true)
-        .with_token(token)
-        .high()
-        .build()?;
+    // Let from_env() load any token from the HF_HOME cache; only override when HF_TOKEN is provided.
+    let mut builder = ApiBuilder::from_env()
+        .with_progress(true)
+        .high();
+    if let Ok(t) = env::var(HF_TOKEN_ENV_VAR) {
+        builder = builder.with_token(Some(t));
+    }
+    let api = builder.build()?;

Optional: you can drop explicit token handling entirely if you prefer to rely solely on from_env().

🧹 Nitpick comments (2)

lib/llm/src/hub.rs (2)

57-61: Clarify error message for offline/endpoint configurations.

Small UX improvement: mention HF_HUB_OFFLINE and HF_ENDPOINT hints to aid troubleshooting.

-            return Err(anyhow::anyhow!(
-                "Failed to fetch model '{}' from HuggingFace: {}. Is this a valid HuggingFace ID?",
-                model_name,
-                e
-            ));
+            return Err(anyhow::anyhow!(
+                "Failed to fetch model '{}' from Hugging Face: {}. \
+If running with HF_HUB_OFFLINE=1, ensure the repo is cached under HF_HOME; \
+otherwise verify HF_ENDPOINT/HF_TOKEN and that the ID is valid.",
+                model_name,
+                e
+            ));

---

120-127: Simplify and make image check case-insensitive.

Current logic duplicates extensions to handle case. Normalize once and compare.

-fn is_image(s: &str) -> bool {
-    s.ends_with(".png")
-        || s.ends_with("PNG")
-        || s.ends_with(".jpg")
-        || s.ends_with("JPG")
-        || s.ends_with(".jpeg")
-        || s.ends_with("JPEG")
-}
+fn is_image(s: &str) -> bool {
+    let s = s.to_ascii_lowercase();
+    s.ends_with(".png") || s.ends_with(".jpg") || s.ends_with(".jpeg")
+}

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled
• Linear integration is disabled

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 6a358f7 and 93aa89a.

📒 Files selected for processing (1)

• lib/llm/src/hub.rs (1 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)

• GitHub Check: Build and Test - dynamo
• GitHub Check: pre-merge-rust (lib/runtime/examples)
• GitHub Check: pre-merge-rust (lib/bindings/python)
• GitHub Check: pre-merge-rust (.)

🔇 Additional comments (1)

lib/llm/src/hub.rs (1)

45-49: Switch to ApiBuilder::from_env() achieves the PR goal.

Using from_env() is the right move to pick up HF_HOME/HF_ENDPOINT/HF_HUB_OFFLINE automatically. The rest of the builder chain remains intact. Looks good.

[grahamking]

Thank you @hhk7734 !