The Sglang version in docker image

[AIR-hl]

I pull the docker image and find ths version of sglang is 0.0.0.dev1+gdce8b0606, is this a outdated version? if slime support latest sglang?

root@lshn-quicksilver-hb3n-24:~# pip show sglang
Name: sglang
Version: 0.0.0.dev1+gdce8b0606
Summary: SGLang is a fast serving framework for large language models and vision language models.
Home-page: https://github.com/sgl-project/sglang
Author: 
Author-email: 
License: 
Location: /usr/local/lib/python3.12/dist-packages
Editable project location: /sgl-workspace/sglang/python

[zhuzilin]

The latest image is using sglang 0.5.7.
The reason for this is that sglang now detecting version dynamically with git tag and the sglang in the dev image does not have tag...

[AIR-hl]

The latest image is using sglang 0.5.7. The reason for this is that sglang now detecting version dynamically with git tag and the sglang in the dev image does not have tag...

I see. I meet the following problem under latest image:

(RolloutManager pid=86887)                                   ^^^^^^^^^^^^^^^^^^^                                                                                    
(RolloutManager pid=86887) ray.exceptions.RayTaskError(TypeError): ray::SGLangEngine.init() (pid=4740, ip=10.146.231.107, actor_id=128943a169ed373c599217b105000000,
 repr=<slime.backends.sglang_utils.sglang_engine.SGLangEngine object at 0x7ed555cc84a0>)                                                                            
(RolloutManager pid=86887)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                          
(RolloutManager pid=86887)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                               
(RolloutManager pid=86887)   File "/mnt/tidal-alsh01/dataset/redone/xushiyue/slime/slime/backends/sglang_utils/sglang_engine.py", line 155, in init                 
(RolloutManager pid=86887)     self._init_normal(server_args_dict)                                                                                                  
(RolloutManager pid=86887)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                               
(RolloutManager pid=86887)   File "/mnt/tidal-alsh01/dataset/redone/xushiyue/slime/slime/backends/sglang_utils/sglang_engine.py", line 183, in _init_normal         
(RolloutManager pid=86887)     self.process = launch_server_process(ServerArgs(**server_args_dict))                                                                 
(RolloutManager pid=86887)                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                  
(RolloutManager pid=86887)   File "<string>", line 293, in __init__                                                                                                 
(RolloutManager pid=86887)   File "/sgl-workspace/sglang/python/sglang/srt/server_args.py", line 620, in __post_init__                                              
(RolloutManager pid=86887)     self._handle_gpu_memory_settings(gpu_mem)                                                                                            
(RolloutManager pid=86887)   File "/sgl-workspace/sglang/python/sglang/srt/server_args.py", line 826, in _handle_gpu_memory_settings                                
(RolloutManager pid=86887)     self._generate_piecewise_cuda_graph_tokens()                                                                                         
(RolloutManager pid=86887)   File "/sgl-workspace/sglang/python/sglang/srt/server_args.py", line 919, in _generate_piecewise_cuda_graph_tokens                      
(RolloutManager pid=86887)     + list(range(4352, self.piecewise_cuda_graph_max_tokens + 1, 256))                                                                   
(RolloutManager pid=86887)                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~                                                                          
(RolloutManager pid=86887) TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' 

[zhuzilin]

Could you run with --debug-rollout-only to test if the running sglang standalong works? It seems like a bug or config issue with sglang.

[AIR-hl]

Could you run with --debug-rollout-only to test if the running sglang standalong works? It seems like a bug or config issue with sglang.

I have another question, if i train with deepseek-v3.1-t with fp8 format, what's the format of the output checkpoint? fp8 or others?

[AIR-hl]

Could you run with --debug-rollout-only to test if the running sglang standalong works? It seems like a bug or config issue with sglang.

cant solve error, so I try to use sft again, the trainning seems hang at this point:

main part of my script is as followed, i use 12 x 8H20 GPUs:

# ---------- CKPT ----------
CKPT_ARGS=(
   --hf-checkpoint "${HF_CKPT_DIR}"
   # For safety: ensure model_name contains "deepseekv3" so megatron->hf conversion picks the correct mapping.
   --model-name deepseekv3
   --ref-load "${REF_TORCH_DIST_DIR}"
   --load "${LOAD_DIR}"
   --save "${SAVE_DIR}"
   --save-interval 20
)

# ---------- SFT Data ----------
# Replace prompt-data with your dataset (jsonl/parquet). Default uses the dapo-math-17k in docs.
SFT_ARGS=(
   --rollout-function-path slime.rollout.sft_rollout.generate_rollout
   --prompt-data "${TRAIN_DATA_FILE}"
   --input-key messages
   --rollout-shuffle
   --num-epoch 3
   --micro-batch-size 8
   --rollout-batch-size 96
   --global-batch-size 96
   --rollout-max-context-len 8192
   --rollout-max-prompt-len 2048
   --rollout-max-response-len 6144   

   --loss-type sft_loss
   --calculate-per-token-loss
   --disable-compute-advantages-and-returns
   --debug-train-only
)

# ---------- Parallelism / Perf (Megatron) ----------
# These defaults follow the DeepSeek-R1 128xH100 example; adjust to match your cluster topology.
PERF_ARGS=(
   --tensor-model-parallel-size ${TENSOR_MP_SIZE:-8}
   --sequence-parallel
   --pipeline-model-parallel-size ${PIPELINE_MP_SIZE:-12}
   --context-parallel-size ${CONTEXT_PARALLEL_SIZE:-1}
   --expert-model-parallel-size ${EXPERT_MP_SIZE:-8}
   --expert-tensor-parallel-size ${EXPERT_TP_SIZE:-1}

   # If pipeline stages don't evenly divide layers, override this like the DeepSeek-R1 example.
   --decoder-last-pipeline-num-layers ${DECODER_LAST_PP_NUM_LAYERS:-6}

   --recompute-granularity full
   --recompute-method uniform
   --recompute-num-layers 1

   --use-dynamic-batch-size
   --max-tokens-per-gpu ${MAX_TOKENS_PER_GPU:-16384}
)

# ---------- Optimizer ----------
# For large models, CPU Adam is common to save GPU memory; disable if you have enough HBM.
OPTIMIZER_ARGS=(
   --optimizer adam
   --lr 1e-5
   --lr-decay-style cosine
   --min-lr 1e-6
   --lr-warmup-fraction 0.1
   --weight-decay 0.1
   --adam-beta1 0.9
   --adam-beta2 0.98

   --optimizer-cpu-offload
   --overlap-cpu-optimizer-d2h-h2d
   --use-precision-aware-optimizer
)

WANDB_ARGS=(
   # --use-wandb
   # --wandb-project slime-dev
   # --wandb-group deepseek-v3.1-terminus
   # --wandb-key ${WANDB_KEY}
)

MISC_ARGS=(
   # default dropout in megatron is 0.1
   --attention-dropout 0.0
   --hidden-dropout 0.0
   # should be good for model performance
   --accumulate-allreduce-grads-in-fp32
   --attention-softmax-in-fp32

   # use deepep for megatron
   --moe-enable-deepep
   --moe-token-dispatcher-type flex

   # For DeepSeek MLA models, attention backend may need adjustment
   # --attention-backend flash
)

# Ray cluster lifecycle is managed externally by the user.
# This script only submits a job to an existing Ray cluster.
export MASTER_ADDR=${MASTER_ADDR:-"127.0.0.1"}
export no_proxy="127.0.0.1,${MASTER_ADDR}"
export RAY_DASHBOARD_ADDR=${RAY_DASHBOARD_ADDR:-"http://127.0.0.1:8265"}


RUNTIME_ENV_JSON="{
  \"env_vars\": {
    \"PYTHONPATH\": \"/root/Megatron-LM/\",
    \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\",
    \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\",
    \"no_proxy\": \"${no_proxy}\",
    \"MASTER_ADDR\": \"${MASTER_ADDR}\",
    \"PYTORCH_CUDA_ALLOC_CONF\": \"expandable_segments:True\",
    \"NVTE_FP8_BLOCK_SCALING_FP32_SCALES\": \"1\",
    \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\",
    \"TORCH_CUDA_ARCH_LIST\": \"9.0\",
    \"NCCL_TIMEOUT\": \"1800\",
    \"NCCL_SOCKET_IFNAME\": \"eth0\",
    \"GLOO_SOCKET_IFNAME\": \"eth0\"
  }
}"

ray job submit --address="${RAY_DASHBOARD_ADDR}" \
   --runtime-env-json="${RUNTIME_ENV_JSON}" \
   -- python3 train_async.py \
   --actor-num-nodes "${ACTOR_NUM_NODES}" \
   --actor-num-gpus-per-node "${ACTOR_NUM_GPUS_PER_NODE}" \
   ${MODEL_ARGS[@]} \
   ${CKPT_ARGS[@]} \
   ${SFT_ARGS[@]} \
   ${OPTIMIZER_ARGS[@]} \
   ${WANDB_ARGS[@]} \
   ${PERF_ARGS[@]} \
   ${MISC_ARGS[@]}