[FEATURE]: Request Migration when Decode Worker Failed/Shutdown

[tedzhouhk]

Feature request

Support request migration when a decode engine fails or is shutdown

• vllm: feat: migrate requests when planner shutdown decode engine (vllm) #2280
• trtllm: TODO
• sglang: TODO

Describe the problem you're encountering

Currently, different backends behaves differently when error or shutting down.

Describe alternatives you've tried

No response

[qimcis]

@tedzhouhk if you have yet to do so, i would love to help out by implementing request migration support for sglang! if you've already gotten started on it let me know

[wenchen76]

Hi! I'd love to help out with this for trtllm. Can I jump in?

[rmccorm4]

Hi @wenchen76, go for it!

[zhixiongli2011]

Hello, might I help with this issue?

[vyalamar]

I reviewed the earlier PR that added the --migration-limit flag and CancelledError → GeneratorExit handling in decode_worker and worker/main.py.

That PR covered worker-side fail detection, but it didn’t implement the planner-side requeue, fencing, or telemetry, so migration isn’t yet complete.

I’d like to pick up from here and finish the full Tier-A request-migration path:

Extend planner logic to reassign undecoded/prefill requests on worker drain or missed heartbeat.

Add a light lease/version fence to ensure only one active owner.

Integrate Prometheus counters (request_migrations_total, failed, latency_ms).

Add chaos test: terminate a worker mid-prefill, confirm automatic requeue and zero request loss.

Keep --migration-limit and cancellation behavior from the previous PR.

I can post a short RFC (state-machine + requeue sequence + test plan) before pushing the new PR if maintainers approve.

@tedzhouhk — please confirm if I can take ownership and continue from the last PR.