feat: sglang examples launch and deploy #2068
Conversation
biswapanda
requested review from
grahamking and
ishandhanani
and removed request for
grahamking
WalkthroughThis update introduces a new Kubernetes deployment YAML for the sglang backend, defining both frontend and decode worker services with detailed resource and probe configurations. Additionally, it corrects shell script syntax in two launch scripts by removing unnecessary trailing backslashes from the last argument lines in multi-line Python command invocations. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Frontend (Pod)
participant SGLangDecodeWorker (Pod)
User->>Frontend (Pod): HTTP request
Frontend (Pod)->>SGLangDecodeWorker (Pod): Model decode request
SGLangDecodeWorker (Pod)-->>Frontend (Pod): Decoded response
Frontend (Pod)-->>User: Response
Estimated code review effort2 (~12 minutes) Possibly related PRs
Poem
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🔭 Outside diff range comments (2)
components/backends/sglang/launch/agg.sh (1)
1-13: Harden the script withset -euo pipefail.The launch script controls backend services; silent failures will be painful to debug. Add the standard safety flags right after the shebang so the script aborts on any error, undefined variable or failed pipe.
#!/bin/bash +# Fail fast and propagate non-zero exit codes +set -euo pipefailcomponents/backends/sglang/launch/agg_router.sh (1)
1-13: Replicate safety flags fromagg.sh.For consistency and robustness, add
set -euo pipefailto this script as well.#!/bin/bash +# Abort on error, undefined var or pipe failure +set -euo pipefail
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
components/backends/sglang/deploy/agg.yaml(1 hunks)components/backends/sglang/launch/agg.sh(1 hunks)components/backends/sglang/launch/agg_router.sh(1 hunks)
🧠 Learnings (2)
📓 Common learnings
Learnt from: fsaady
PR: ai-dynamo/dynamo#1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:230-244
Timestamp: 2025-07-03T10:14:30.570Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd) are intentionally left running even if later processes fail. This design choice allows users to manually connect to nodes and debug issues without having to restart the entire SLURM job from scratch, providing operational flexibility for troubleshooting in cluster environments.
components/backends/sglang/deploy/agg.yaml (3)
Learnt from: julienmancuso
PR: #2012
File: deploy/cloud/helm/crds/templates/nvidia.com_dynamocomponentdeployments.yaml:1178-1180
Timestamp: 2025-07-18T16:05:05.482Z
Learning: The stopSignal field under lifecycle in DynamoComponentDeployment CRDs is autogenerated due to Kubernetes library upgrades (k8s.io/api and k8s.io/apimachinery from v0.32.3 to v0.33.1), not a manual design decision by the user.
Learnt from: julienmancuso
PR: #2012
File: deploy/cloud/helm/crds/templates/nvidia.com_dynamographdeployments.yaml:1233-1235
Timestamp: 2025-07-18T16:04:47.445Z
Learning: The stopSignal field in Kubernetes CRDs like DynamoGraphDeployment and DynamoComponentDeployment is autogenerated by controller-gen when upgrading Kubernetes library versions, and represents expected upstream API changes rather than manual code that needs custom validation.
Learnt from: julienmancuso
PR: #2012
File: deploy/cloud/helm/crds/templates/nvidia.com_dynamocomponentdeployments.yaml:92-98
Timestamp: 2025-07-18T16:04:31.743Z
Learning: CRD schemas in files like deploy/cloud/helm/crds/templates/*.yaml are auto-generated from Kubernetes library upgrades and should not be manually modified as changes would be overwritten during regeneration.
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: fsaady
PR: ai-dynamo/dynamo#1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:230-244
Timestamp: 2025-07-03T10:14:30.570Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd) are intentionally left running even if later processes fail. This design choice allows users to manually connect to nodes and debug issues without having to restart the entire SLURM job from scratch, providing operational flexibility for troubleshooting in cluster environments.
components/backends/sglang/deploy/agg.yaml (3)
Learnt from: julienmancuso
PR: #2012
File: deploy/cloud/helm/crds/templates/nvidia.com_dynamocomponentdeployments.yaml:1178-1180
Timestamp: 2025-07-18T16:05:05.482Z
Learning: The stopSignal field under lifecycle in DynamoComponentDeployment CRDs is autogenerated due to Kubernetes library upgrades (k8s.io/api and k8s.io/apimachinery from v0.32.3 to v0.33.1), not a manual design decision by the user.
Learnt from: julienmancuso
PR: #2012
File: deploy/cloud/helm/crds/templates/nvidia.com_dynamographdeployments.yaml:1233-1235
Timestamp: 2025-07-18T16:04:47.445Z
Learning: The stopSignal field in Kubernetes CRDs like DynamoGraphDeployment and DynamoComponentDeployment is autogenerated by controller-gen when upgrading Kubernetes library versions, and represents expected upstream API changes rather than manual code that needs custom validation.
Learnt from: julienmancuso
PR: #2012
File: deploy/cloud/helm/crds/templates/nvidia.com_dynamocomponentdeployments.yaml:92-98
Timestamp: 2025-07-18T16:04:31.743Z
Learning: CRD schemas in files like deploy/cloud/helm/crds/templates/*.yaml are auto-generated from Kubernetes library upgrades and should not be manually modified as changes would be overwritten during regeneration.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: pre-merge-rust (lib/runtime/examples)
- GitHub Check: pre-merge-rust (.)
- GitHub Check: pre-merge-rust (lib/bindings/python)
- GitHub Check: Build and Test - vllm
🔇 Additional comments (4)
components/backends/sglang/launch/agg.sh (1)
22-29: Trailing back-slash removal is correct and prevents an unterminated command.
Nothing further required for this line.components/backends/sglang/launch/agg_router.sh (1)
22-29: Back-slash removal fixes the shell syntax—LGTM.components/backends/sglang/deploy/agg.yaml (2)
67-74: Namespace mismatch might cause cross-talk.
dynamoNamespaceissglang-agg, but both launch scripts and the frontend container clear the namespacedynamo. Verify this is intentional; otherwise messages from distinct deployments could collide.
71-78: Non-standard GPU resource key.Kubernetes default for NVIDIA GPUs is
nvidia.com/gpu.
If your CRD doesn’t translategpu:automatically, scheduling will fail.
ishandhanani
left a comment
ishandhanani
left a comment
There was a problem hiding this comment.
Do you think you can launch the sgl http server in the background biswa? On maybe the head node?
After I add the engine specific health endpoint we can use it for health and in the future readiness
You can find an example of this in the dsr1 wide ep guide |
biswapanda
requested review from
a team,
alec-flowers,
nnshah1,
ptarasiewiczNV,
richardhuo-nv,
rmccorm4 and
tanmayv25
as code owners
|
Nit: It shouldn't be |
biswapanda
changed the title
biswapanda
force-pushed
the
bis/sgl-examples
branch
from
44256ec to
f706c5d
Compare
Overview:
launchscriptsdeployyamlsTodo
closes:
linear: DYN-750
linear: DYN-751
Where should the reviewer start?
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
New Features
Bug Fixes