Output types

[rlouf]

This discussion follows #1220 which I suggest to read beforehand.

Context

In #1120 we propose to stop discriminating between generation method by defining new functions, but instead to dispatch the generation method based on output types. This design is more flexible (we can add a new type without extra boilerplate) and more ergonomic (users do not need to memorize new functions).

This discussion focuses on the interface to define the output types itself.

Interfaces with providers

Before working on the user interface we need to figure out what are the internal interfaces. Most APIs only offer JSON Schema specifications as an interface, and every aspect of the structured generation happens internally:

• OpenAI's structured generation feature
• .txt's API
• Text-generation-inference

Some also offer regular expressions:

• vLLM
• NVIDIA

Or grammars:

• Text-generation-inference
• vLLM
• NVIDIA

Outlines can provide the three interfaces for open models.

Output types and definitions

There are two things to consider: the output type and the object that is used to define the type. For instance, we can use a Pydantic BaseModel to define an output structure, but we have different options when it comes to the output of the generator function:

1. An instance of the BaseModel that contains the information generated by the model;
2. A string that represents the JSON object whose structure corresponds to the JSON Schema implicitly defined by the BaseModel
3. A dictionary [...]
4. A string that represents a YAML object [...]
5. etc.

Parsing or not parsing

The behavior of:

from pydantic import BaseModel
from outlines import generate


class User(BaseModel):
    id: int
    name: str


generator = generate(model, User)

is to build a generator that, given a prompt, returns an instance of User. If we pass a JSON Schema specification in a dictionary it will return a JSON object parsed into a dictionary. It we pass a JSON Schema specification as a string it will return a string that returns the JSON object.

I don't like the fact that it's implicit, and I have come to dislike the fact that we would return parsed objects at all since it does complexify the interface a lot just to avoid at most two lines of code (importing json and run json.loads(result)). It also makes the streaming interface ambiguous and potentially error-prone by making decisions for the users.

Proposal

If we temporarily forget about the interface with model providers, the simplest interface would be to require that the second argument to generator and streamer be a regular expression. We can then define functions that transform a structure definition into a regular expression. For instance to generate json:

from functools import singledispatch


@singledispatch
def json(definition):
    raise NotImplementedError


@json.dispatch(type(BaseModel))
def json_pydantic(definition):
    regex = to_regex(definition.model_json_schema())
    return regex


@json.dispatch(str)
def json_str(definition):
    regex = to_regex(definition)
    return regex

So we can write:

from outlines.types import json

generate = generator(model, json(User))

Similarly, to generate YAML objects, define:

from functools import singledispatch


@singledispatch
def yaml(definition):
    raise NotImplementedError


@yaml.dispatch(type(BaseModel))
def yaml_pydantic(definition):
    regex = to_yaml(definition.model_json_schema())
    return regex


@yaml.dispatch(str)
def yaml_str(definition):
    regex = to_yaml(definition)
    return regex

generate = generator(model, json(User))

And have generator and streamer return strings. The major downside to this is that we cannot interface with API providers any more, including our own API since they use JSON Schema as an interface. We can thus introduce new types:

from typing import Union
from dataclasses import dataclass

@dataclass
class Json:
    definition: Union[str, dict, type(BaseModel)]

@dataclass
class Yaml:
    definition: Union[str, dict, type(BaseModel)]

generate = generator(model, Json(User))
generate = generator(model, Yaml(User))

And provider classes are responsible for choosing the kind of representation they prefer. To generate text that follows a regular expression or a grammar:

from outlines.types import Regex, Grammar

generate = generator(model, Regex(r"[0-9]*"))
generate = generator(model, Grammar("grammar string"))

Work with special types

Following the special types already defined in Outlines in such a way that they can be used in Pydantic models:

from pydantic import WithJsonSchema
from outlines.types import Regex
from typing_extension import Annotated

pattern = r"\d{5}(?:-\d{4})?"

USZipCode = Annotated[
    str,
    WithJsonSchema({"type": "string", "pattern": pattern}),
    Regex(pattern)
]

generate = generator(model, USZipCode)

This generalizes to the case where the special type may be defined via a grammar, cf email adresses.

Open questions

• Do we really want to handle the conversion from the returned text to objects? It complexifies the interface substantially and still may be confusing.

Reviewers

@torymur @lapp0

[torymur]

Interface looks good! 👍

Do we really want to handle the conversion from the returned text to objects? It complexifies the interface substantially and still may be confusing.

I would hesitate here too, conversion is the whole other subject prone to errors & confusions of its own and handling it would leave user always guessing if something went wrong during the conversion or the generation itself.

Leaving it to the user to decide what to do with the result (load this json string or pass it further as it is) will allow not to spend resources on unnecessary cycle of serialization/deserialization due to the format (twice actually) and even don't bring unnecessary dependencies for user just for serialization. Also strings are widely compatible with numerous languages, libraries, and tools.

Perhaps, being highly reliable and predictable structured output provider might be best.

[rlouf]

Perhaps, being highly reliable and predictable structured output provider might be best.

This.