[FlashRL 3/N] Add example for FP8 training with FlashRL

[SumanthRH]

What does this PR do?

WIP PR to add FP8 training with FlashRL.

Note that we currently only support online FP8 quantization. Support for pre-quantized fp8 and in8 will follow soon - it's a bit more involved given that you need to calibrate scaling

uses a custom vllm wheel. Found this to be the simplest way to manage the custom vllm patches in flashRL. Wheel is pre-packaged build from branch: https://github.com/SumanthRH/vllm/tree/flashrl . Specifying the git url directly led to uv building the cpu-only version of vllm for some reason. We'll use this wheel for now

TODO:

• Verify E2E run on Deepspeed and FSDP
• Verify training on qwen3 14B and 32B
• upload wheel to github releases and use the link from Github
• add docs

[SumanthRH]

This is an important warning. I've already improved weight syncing with the batching impl + fixes for FlashRL's patch_load_weights method, but it is still not good enough. We will revisit the fp8 slowdown, and meanwhile also see if int8 can provide good overall step time improvements

[erictang000]

just to confirm - we can only use the offline engine for flash-rl, so only single turn rollouts?

[erictang000]

maybe worth a clarification in the doc, I didn't realize until i hit this line of code

[SumanthRH]

Yeah let me add a warning

[erictang000]

Suggested change

We pass `quantization=fp8` flag to the vLLM engine at initialization time. This means that the weights are loaded as usual in half precision and then quantized down to fp8. During training, generations are sampled as usual, and in this case, sampled from quantized weights. Since we use online quantization, the scale factor used for quantizing activations are computed on the fly by vLLM internally.

We pass the `quantization=fp8` flag to the vLLM engine at initialization time. This means that the weights are loaded as usual in half precision and then quantized down to FP8. During training, generations are sampled as usual, but now from the quantized weights. Since vLLM uses online quantization, the scale factors used for quantizing activations are computed dynamically during runtime.