fix(ci): Reduce the free gpu memory fraction

[tanmayv25]

Overview:

With the new cache_transceiver config, this high free_gpu_memory_fraction can lead to OOM issues on A100 machines.
Updating to reasonable defaults.

Summary by CodeRabbit

• Chores
  • Updated default GPU memory allocation for inference, reducing reserved space for the key-value cache from 95% to 85%.
  • Applied consistently across prefill, decode, and aggregation stages for uniform behavior.
  • Adjusts runtime memory distribution and may affect concurrency and throughput depending on hardware. No user action required.

[rmccorm4]

With the new cache_transceiver config, this high free_gpu_memory_fraction can lead to OOM issues

Does this reserve some GPU memory as well? Are there any guides/tips on estimating combination of free_gpu_memory_fraction and cache_transceiver_config?

I wonder if we should consider this a bug to raise with TRTLLM, such that free_gpu_memory_fraction should only be considered AFTER all the other things (model weights, cache transceiver config, etc.) so that it doesn't need to be tuned as much based on the others.

[coderabbitai]

Walkthrough

Updated kv_cache_config.free_gpu_memory_fraction from 0.95 to 0.85 across three TRT-LLM engine configuration YAMLs: agg.yaml, decode.yaml, and prefill.yaml. No other configuration fields were modified.

Changes

Cohort / File(s)	Summary of Changes
TRT-LLM engine configs components/backends/trtllm/engine_configs/agg.yaml, components/backends/trtllm/engine_configs/decode.yaml, components/backends/trtllm/engine_configs/prefill.yaml	Set kv_cache_config.free_gpu_memory_fraction from 0.95 to 0.85 in each file; no other changes.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

• fix: Fix message truncation in disagg flow #1572 — Also changes kv_cache_config.free_gpu_memory_fraction in TRT-LLM-related configs, indicating aligned parameter adjustments.

Poem

A bunny tweaks the cache so neat,
From ninety-five to eighty-five—complete.
Three YAML carrots in a row,
Nibbled values set to go.
GPUs purr, the burrow’s bright,
Configs aligned—hop into night! 🐇✨

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the "Integrations" page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

components/backends/trtllm/engine_configs/prefill.yaml (1)

26-27: Document KV-cache change and validate prefill capacity

Quick summary: I searched the repo — the trtllm prefill engine config now sets free_gpu_memory_fraction: 0.85 and there are no other prefill engine configs still set to 0.95. Please add the inline rationale comment and run the runtime sanity checks below.

Files to inspect / keep consistent:

• components/backends/trtllm/engine_configs/prefill.yaml — update here (primary)
• Notable repo occurrences found (for context):
  • components/backends/trtllm/engine_configs/decode.yaml, agg.yaml — 0.85
  • components/backends/trtllm/engine_configs/multimodal/*/prefill.yaml — 0.30
  • components/backends/trtllm/engine_configs/deepseek_r1/*/prefill.yaml — 0.75 / 0.30
  • components/backends/trtllm/engine_configs/llama4/**/eagle_prefill.yaml — 0.5

Proposed inline comment (apply to components/backends/trtllm/engine_configs/prefill.yaml):

kv_cache_config:
-  free_gpu_memory_fraction: 0.85
+  # Lowered free GPU memory fraction to 0.85 to reduce OOM risk for long prefill sequences.
+  # Verify target concurrency and long-context workloads still fit with this KV allocation.
+  free_gpu_memory_fraction: 0.85

Sanity checks to run (manual / CI):

• Replay representative prefill-heavy workloads (e.g., max_num_tokens=8192) at expected batch sizes and concurrency; confirm no regression in throughput and no OOMs.
• If using MIG, validate MIG profiles remain stable with this fraction.
• If you use the perf_sweeps tools, regenerate/test configs that inject ctx_free_gpu_memory_fraction (scripts in components/backends/trtllm/performance_sweeps/) to ensure automation matches this default.

Repo check performed:

• Searched for free_gpu_memory_fraction across repo; no other prefill engine configs were left at 0.95.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between cebe921 and c862af9.

📒 Files selected for processing (3)

• components/backends/trtllm/engine_configs/agg.yaml (1 hunks)
• components/backends/trtllm/engine_configs/decode.yaml (1 hunks)
• components/backends/trtllm/engine_configs/prefill.yaml (1 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)

• GitHub Check: Build and Test - dynamo
• GitHub Check: pre-merge-rust (lib/bindings/python)
• GitHub Check: pre-merge-rust (.)
• GitHub Check: pre-merge-rust (lib/runtime/examples)