[Feature]: Support for Running Classification Task in Online Server

[sam-h-bean]

🚀 The feature, motivation and pitch

I would like it to be easy to stand up models for sequence classification using the vllm online inference pattern. Currently this is available for offline inference but it would be nice to expose this server in kubernetes similar to how we host OpenAI compatible servers.

Alternatives

We could train a causal lm where we treat special tokens as the classification labels. We could then take the softmaxed logprobs for those 2 tokens to threshold. However this is going to require slightly more code on the client side.

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[dipatidar]

@DarkLight1337 I'd like to work on this.

[DarkLight1337]

Thanks for helping out!

[frieda-huang]

Hi @dipatidar Are you still working on this? If not, I would like to pick up the issue!

[frieda-huang]

Hi @DarkLight1337! I’m digging into the “sequence classification” issue. I noticed that OpenAI deprecated the /v1/classifications endpoint in December 2022. If we add our own /v1/classifications route in vLLM for online inference, we’d be diverging from the current OpenAI spec. Do you think it’s worth offering that “bonus” endpoint for ergonomics, or should we stick strictly to OpenAI’s live endpoints (completions/embeddings) for compatibility, as suggested here?

[DarkLight1337]

I think to keep things simple, we can simply create an online version of LLM.classify. To distinguish from the OpenAI endpoint we can have the endpoint located at /classify

[frieda-huang]

I think to keep things simple, we can simply create an online version of LLM.classify. To distinguish from the OpenAI endpoint we can have the endpoint located at /classify

Gotcha. Could you assign the issue to me?

[geraldstanje]

hi,

here some testing i did - it seems /classify route is also missing in V0? is this pr adding it and is it already in vllm/vllm-openai:latest?

Dockerfile

docker run --gpus all \
  -v $(pwd)/modernbert:/model \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model /model \
  --dtype float32 \
  --task classify

./run_openai_docker.sh
INFO 05-23 11:19:07 [__init__.py:239] Automatically detected platform cuda.
INFO 05-23 11:19:11 [api_server.py:1043] vLLM API server version 0.8.5.post1
INFO 05-23 11:19:11 [api_server.py:1044] args: Namespace(host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/model', task='classify', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='float32', max_model_len=None, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)
WARNING 05-23 11:19:21 [arg_utils.py:1658] --dtype torch.float32 is not supported by the V1 Engine. Falling back to V0. 
INFO 05-23 11:19:21 [api_server.py:246] Started engine process with PID 26
INFO 05-23 11:19:25 [__init__.py:239] Automatically detected platform cuda.
INFO 05-23 11:19:27 [llm_engine.py:240] Initializing a V0 LLM engine (v0.8.5.post1) with config: model='/model', speculative_config=None, tokenizer='/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float32, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=/model, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=PoolerConfig(pooling_type=None, normalize=None, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
INFO 05-23 11:19:28 [cuda.py:245] Cannot use FlashAttention-2 backend for dtype other than torch.float16 or torch.bfloat16.
INFO 05-23 11:19:28 [cuda.py:289] Using XFormers backend.
INFO 05-23 11:19:30 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 05-23 11:19:30 [model_runner.py:1108] Starting to load model /model...
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  3.50it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  3.49it/s]

INFO 05-23 11:19:31 [loader.py:458] Loading weights took 0.38 seconds
INFO 05-23 11:19:31 [model_runner.py:1140] Model loading took 1.5020 GiB and 0.544689 seconds
INFO 05-23 11:19:31 [api_server.py:1090] Starting vLLM API server on http://0.0.0.0:8000
INFO 05-23 11:19:31 [launcher.py:28] Available routes are:
INFO 05-23 11:19:31 [launcher.py:36] Route: /openapi.json, Methods: GET, HEAD
INFO 05-23 11:19:31 [launcher.py:36] Route: /docs, Methods: GET, HEAD
INFO 05-23 11:19:31 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 05-23 11:19:31 [launcher.py:36] Route: /redoc, Methods: GET, HEAD
INFO 05-23 11:19:31 [launcher.py:36] Route: /health, Methods: GET
INFO 05-23 11:19:31 [launcher.py:36] Route: /load, Methods: GET
INFO 05-23 11:19:31 [launcher.py:36] Route: /ping, Methods: GET, POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /tokenize, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /detokenize, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /v1/models, Methods: GET
INFO 05-23 11:19:31 [launcher.py:36] Route: /version, Methods: GET
INFO 05-23 11:19:31 [launcher.py:36] Route: /v1/chat/completions, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /v1/completions, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /v1/embeddings, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /pooling, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /score, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /v1/score, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /rerank, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /v1/rerank, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /v2/rerank, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /invocations, Methods: POST
INFO 05-23 11:19:31 [launcher.py:36] Route: /metrics, Methods: GET
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.