Support for Constrained decoding

[ojus1]

For getting structured outputs from custom-finetuned LLMs, extensive use of constrained decoding is standard.

Is there a plan to add support for DisjunctiveConstraint (and others) to vLLM in the near future?
How would one go about implementing this in vLLM (if I were to add a PR)?

[zhuohan123]

Hi! We very much welcome you to contribute to this feature! I believe You can add this functionality by modifying the following places:

1. Add the related parameters to SamplingParams

   vllm/vllm/sampling_params.py

   Line 5 in bdd6b4c

   class SamplingParams:

2. Implement the logic in Sampler

   vllm/vllm/model_executor/layers/sampler.py

   Line 15 in bdd6b4c

   class Sampler(nn.Module):

3. To make our OpenAI frontend support this feature, add the related parameters to CompletionRequest

   vllm/vllm/entrypoints/openai/protocol.py

   Line 68 in bdd6b4c

   class CompletionRequest(BaseModel):

   and add the parameter here when initializing SamplingPramams here:

   vllm/vllm/entrypoints/openai/api_server.py

   Lines 130 to 142 in bdd6b4c

   	sampling_params = SamplingParams(
   	n=request.n,
   	best_of=request.best_of,
   	presence_penalty=request.presence_penalty,
   	frequency_penalty=request.frequency_penalty,
   	temperature=request.temperature,
   	top_p=request.top_p,
   	top_k=request.top_k,
   	stop=request.stop,
   	ignore_eos=request.ignore_eos,
   	max_tokens=request.max_tokens,
   	logprobs=request.logprobs,
   	use_beam_search=request.use_beam_search,

[zacharyblank]

Curious if there's been any progress with this. I've hooked up Microsoft/Guidance and vLLM but the most powerful features aren't yet available because of missing features in vLLM.

Thank you!

[viktor-ferenczi]

Related to #535

[viktor-ferenczi]

Related topics:

• JSON formatting issue #1191: Reliable JSON (or other structured data) generation
• Integration with any of these libraries:
  • https://github.com/outlines-dev/outlines
  • https://github.com/guidance-ai/guidance
  • https://github.com/1rgs/jsonformer

[viktor-ferenczi]

I'm going to implement this.

[MaxZabarka]

I'd like to help implement this aswell

[viktor-ferenczi]

The constraint may change during the generation. For example in case of #1191 it depends on what the JSON schema allows for the next token, but that depends on where the generation is currently in the schema. We cannot use the same constraint over the whole sequence in the general case. It must also work for beam search. How can we handle that efficiently via a REST API?

[viktor-ferenczi]

I think in case of the REST API we could allow for passing a formal description of the constraint in some generic and de facto standard format (if we can talk about it this soon) like guidance. It would allow for "compiling" the constraint inside the server and applying it to all generation of that sequence, including beam search.

In case of direct vLLM calls (from Python) we could let the user to pass a callback to process the logits before the token is chosen, so the probability of any unwanted tokens can be squashed to zero. It would be efficient and allow for using any algorithm. Then we could provide adapters for the above mentioned libraries.

[viktor-ferenczi]

Supporting the outlines library seems to be the best approach, because:

Outlines is compatible with all models. It only interfaces with models via the next-token logits. It can be used with API-based models as well.

While jsonformer is limited only to JSON and guidance does not have a clear way to integrate (it has spaghetti code).

[MaxZabarka]

In case of direct vLLM calls (from Python) we could let the user to pass a callback to process the logits before the token is chosen, so the probability of any unwanted tokens can be squashed to zero. It would be efficient and allow for using any algorithm. Then we could provide adapters for the above mentioned libraries.

This might be inefficient when generating structured data, for example a format like JSON, where a significant portion of the output consists of predetermined fields and symbols. Manipulating logits after a token is generated would be wasteful because we would already know what the majority of tokens are be before generation.

A feature of guidance is that it avoids running generation for tokens that are already known. Given that speed and efficiency is important to vLLM, how would we go about implementing something like this when integrating outlines or another framework?

[viktor-ferenczi]

Let's separate the two features:

1. Ability to constrain the token generated (manipulate logits before the token is chosen)
2. Ability to skip ahead if there is no choice between tokens (next token is dictated by a schema)

Since these features are largely independent I suggest implementing them in the above order.