Support arbitrary json_object in OpenAI and Context Free Grammar #3211
Conversation
simon-mo
changed the title
|
|
||
| def __init__(self, regex_string: str, tokenizer): | ||
| """Compile the FSM that drives the regex-structured generation. | ||
| def adapt_tokenizer(self, tokenizer): |
There was a problem hiding this comment.
simon-mo
changed the title
| JSON = "json" | ||
| REGEX = "regex" | ||
| CHOICE = "choice" | ||
| CFG = "cfg" |
There was a problem hiding this comment.
A dumb question: does CFG mean config? only for grammar config?
There was a problem hiding this comment.
Context free grammar, good point that I should probably change it to grammar to disambiguate
|
|
||
| class CFGLogitsProcessor(BaseLogitsProcessor): | ||
|
|
||
| def __init__(self, cfg: str, tokenizer): |
There was a problem hiding this comment.
Adding type definition for tokenizer in this whole file is better.
| tokenizer.decode = change_decoder(tokenizer.decode) | ||
| setattr(tokenizer, "_outlines_adapted", True) # noqa: B010 | ||
|
|
||
| return tokenizer |
|
Excellent new feature! Thanks everyone! I will definitely test it when the next vLLM release comes out. |
|
Is this available in the new vLLM library? How do I use this feature with Mixtral8x7b? |
|
This is available in latest release. https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html |
|
Please keep this issue closed, but just had to say: Oh man I love vLLM much, you guys are all awesome. Thank you! |
5 participants
Closes #3148
Closes #1229