[RFC] Run HF processing on GPU

[DarkLight1337]

Motivation.

Now that fast processors have been merged into HF Transformers, we can call HF processors with device="cuda" to run them on GPU. This has been shown to improve the processing speed up to one order of magnitude (though that requires a large batch size which doesn't really happen in practice since we call the HF processor once per prompt).

Benchmarks: huggingface/transformers#39591

Proposed Change.

We can support this via mm_processor_kwargs, e.g.:

# Use CUDA by default
llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
          mm_processor_kwargs={"use_fast": True, "device": "cuda"})

# Use CUDA for one request - technically possible but requires memory profiling
# to assume GPU preprocessing usage even if no requests actually do this,
# so likely not support this niche case to save code complexity
llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
          mm_processor_kwargs={"use_fast": True})
llm.generate({
    "prompt": formatted_prompt,
    "multi_modal_data": {"image": image},
    "mm_processor_kwargs": {"device": "cuda"},
})
# -- or --
llm.chat(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": text},
                {"type": "image_url", "image_url": {"url": image_url}},
            ],
        }
    ],
    mm_processor_kwargs={"device": "cuda"},
)

Issues to solve:

• We need to accurately determine whether the HF processor accepts device argument (or any other argument in general) ([Misc] Automatically resolve HF processor init kwargs #22005)
• How to accommodate this in memory profiling so that users cannot cause OOM during inference? ([Core] Enable HF processing on GPU #22070)
• Release benchmarks on vLLM

Follow-ups:

• After processing, some (but not all!) of the inputs will be on GPU but still need to be transferred between API and EngineCore processes. Can we avoid serializing the inputs which wastefully moves them back to CPU?
• To increase the batch size, we can accumulate multiple prompts before running the HF processor on all of the inputs together. This is expected to improve throughput at the cost of latency.

Feedback Period.

No response

CC List.

@hmellor @Isotr0py @ywang96 @njhill

Alternatives

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Isotr0py]

How to accommodate this in memory profiling so we don't end up with OOM during inference?

IIRC, the GPU memory used by processor could also be profiled as 2. memory used by torch in the current vLLM instance. (kind of peak activation tensors) from get_decoder_dummy_data during profile run generally.

Can we simply assume that mm inputs resulting longest prompts can also result maximum peak memory during preprocessing?

vllm/vllm/v1/worker/gpu_worker.py

Lines 238 to 244 in 9484641

	# Execute a forward pass with dummy inputs to profile the memory usage
	# of the model.
	with memory_profiling(
	self.init_snapshot,
	weights_memory=int(
	self.model_runner.model_memory_usage)) as profile_result:
	self.model_runner.profile_run()

vllm/vllm/v1/worker/gpu_model_runner.py

Lines 2475 to 2482 in 9484641

	# Create dummy batch of multimodal inputs.
	dummy_mm_kwargs = self.mm_registry.get_decoder_dummy_data(
	model_config=self.model_config,
	seq_len=max_tokens_per_mm_item,
	mm_counts={
	dummy_data_modality: 1
	},
	).multi_modal_data

[DarkLight1337]

Can we simply assume that mm inputs resulting longest prompts can also result maximum peak memory during preprocessing?

This is a good direction as we don't have to modify the code much to support this.

However, we should use the maximum number of multi-modal data items that can fill the max_model_len, not just a single item. (i.e. replace mm_counts with min(limit_mm_per_prompt, max_model_len / mm_max_contiguous_tokens)) to construct the data.

[DarkLight1337]

Added more points to think about:

• We should disable the multi-modal processor cache automatically if GPU pre-processing is applied because its effectiveness truly shines when the batch size is large.
• After processing, some (but not all!) of the inputs will be on GPU but still need to be transferred between API and EngineCore processes. Can we avoid serializing the inputs which wastefully moves them back to CPU?
• To increase the batch size, we can accumulate multiple prompts before running the HF processor on all of the inputs together. This is expected to improve throughput at the cost of latency.

[njhill]

Another advantage of this is that we don't need to block the CPU / event loop and so other API / request processing can happen in parallel.

• To increase the batch size, we can accumulate multiple prompts before running the HF processor on all of the inputs together. This is expected to improve throughput at the cost of latency.

I'd hope we could do this in a way that has negligible impact on latency... a dedicated thread where the next batch can be assembled while the current one is processing. Similar to what we've done for tokenization: #19334

After processing, some (but not all!) of the inputs will be on GPU but still need to be transferred between API and EngineCore processes. Can we avoid serializing the inputs which wastefully moves them back to CPU?

This would be cool