Merge branch 'main' into jacky-ft-complete-final

Author: kthui

docs/dynamo_glossary.md

@@ -0,0 +1,96 @@
+# NVIDIA Dynamo Glossary
+
+## B
+**Block** - A fixed-size chunk of tokens (typically 16 or 64 tokens) used for efficient KV cache management and memory allocation, serving as the fundamental unit for techniques like PagedAttention.
+
+## C
+**Component** - The fundamental deployable unit in Dynamo. A discoverable service entity that can host multiple endpoints and typically maps to a Docker container (such as VllmWorker, Router, Processor).
+
+**Conditional Disaggregation** - Dynamo's intelligent decision-making process within disaggregated serving that determines whether a request is processed locally or sent to a remote prefill engine based on prefill length and queue status.
+
+## D
+**Decode Phase** - The second phase of LLM inference that generates output tokens one at a time.
+
+**depends()** - A Dynamo function that creates dependencies between services, enabling automatic client generation and service discovery.
+
+**Disaggregated Serving** - Dynamo's core architecture that separates prefill and decode phases into specialized engines to maximize GPU throughput and improve performance.
+
+**Distributed Runtime** - Dynamo's Rust-based core system that manages service discovery, communication, and component lifecycle across distributed clusters.
+
+**Dynamo** - NVIDIA's high-performance distributed inference framework for Large Language Models (LLMs) and generative AI models, designed for multinode environments with disaggregated serving and cache-aware routing.
+
+**Dynamo Artifact** - A packaged archive containing an inference graph and its dependencies, created using `dynamo build`. It's the containerized, deployable version of a Graph.
+
+**Dynamo Cloud** - A Kubernetes platform providing managed deployment experience for Dynamo inference graphs.
+
+**dynamo build** - The CLI command to containerize inference graphs or parts of graphs into Docker containers.
+
+**dynamo deploy** - The CLI command to deploy inference graphs to Kubernetes with Helm charts or custom operators.
+
+**dynamo run** - The CLI command to quickly experiment and test models with various LLM engines.
+
+**dynamo serve** - The CLI command to compose and serve inference graphs locally.
+
+## E
+**@endpoint** - A Python decorator used to define service endpoints within a Dynamo component.
+
+**Endpoint** - A specific network-accessible API within a Dynamo component, such as `generate` or `load_metrics`.
+
+## F
+**Frontend** - Dynamo's API server component that receives user requests and provides OpenAI-compatible HTTP endpoints.
+
+## G
+**Graph** - A collection of interconnected Dynamo components that form a complete inference pipeline with request paths (single-in) and response paths (many-out for streaming). A graph can be packaged into a Dynamo Artifact for deployment.
+
+## I
+**Instance** - A running process with a unique `instance_id`. Multiple instances can serve the same namespace, component, and endpoint for load balancing
+
+## K
+**KV Block Manager (KVBM)** - Dynamo's scalable runtime component that handles memory allocation, management, and remote sharing of Key-Value blocks across heterogeneous and distributed environments.
+
+**KV Cache** - Key-Value cache that stores computed attention states from previous tokens to avoid recomputation during inference.
+
+**KV Router** - Dynamo's intelligent routing system that directs requests to workers with the highest cache overlap to maximize KV cache reuse. Determines routing based on KV cache hit rates and worker metrics.
+
+**KVIndexer** - Dynamo component that maintains a global view of cached blocks across all workers using a prefix tree structure to calculate cache hit rates.
+
+**KVPublisher** - Dynamo component that emits KV cache events (stored/removed) from individual workers to the global KVIndexer.
+
+## N
+**Namespace** - Dynamo's logical grouping mechanism for related components. Similar to directories in a file system, they prevent collisions between different deployments.
+
+**NIXL (NVIDIA Inference tranXfer Library)** - High-performance data transfer library optimized for inference workloads, supporting direct GPU-to-GPU transfers and multiple memory hierarchies.
+
+## P
+**PagedAttention** - Memory management technique from vLLM that efficiently manages KV cache by chunking requests into blocks.
+
+**Planner** - Dynamo component that performs dynamic resource scaling based on real-time demand signals and system metrics.
+
+**Prefill Phase** - The first phase of LLM inference that processes the input prompt and generates KV cache.
+
+**Prefix Caching** - Optimization technique that reuses previously computed KV cache for common prompt prefixes.
+
+**Processor** - Dynamo component that handles request preprocessing, tokenization, and routing decisions.
+
+## R
+**RadixAttention** - Technique from SGLang that uses a prefix tree structure for efficient KV cache matching, insertion, and eviction.
+
+**RDMA (Remote Direct Memory Access)** - Technology that allows direct memory access between distributed systems, used for efficient KV cache transfers.
+
+## S
+**@service** - Python decorator used to define a Dynamo service class.
+
+**SGLang** - Fast LLM inference framework with native embedding support and RadixAttention.
+
+## T
+**Tensor Parallelism (TP)** - Model parallelism technique where model weights are distributed across multiple GPUs.
+
+**TensorRT-LLM** - NVIDIA's optimized LLM inference engine with multinode MPI distributed support.
+
+**Time-To-First-Token (TTFT)** - The latency from receiving a request to generating the first output token.
+
+## V
+**vLLM** - High-throughput LLM serving engine with Ray distributed support and PagedAttention.
+
+## X
+**xPyD (x Prefill y Decode)** - Dynamo notation describing disaggregated serving configurations where x prefill workers serve y decode workers. Dynamo supports runtime-reconfigurable xPyD.

docs/index.rst

@@ -135,5 +135,11 @@ Dive in: Examples
    Multinode Examples <examples/multinode.md>
    LLM Deployment Examples using TensorRT-LLM <examples/trtllm.md>
 
+.. toctree::
+   :hidden:
+   :caption: Reference
+
+   Glossary <dynamo_glossary.md>
+   KVBM Reading <architecture/kvbm_reading.md>