Grammar Builder for JSON Schema

[riedgar-ms]

A very basic guidance grammar builder for JSON schema. Handles the basic types, although the set of valid strings is more restricted than JSON (because making new things acceptable is preferable to rejecting things previously allowed).

There is no support for cross references; that will have to wait until a future PR. There is also no support for constraints. I do not expect constraints such as multipleOf or exclusiveMinimum to be supportable. It is possible that some of the constraints such as minItems, maxLength and pattern may be supportable.

[slundberg]

This is great @riedgar-ms ! Do you have any thoughts about how this might connect to Pedantic as well? For example what if people have a pydantic spec, should they export a schema from pydantic and then we build a grammar from that? I know there was some pydantic work #559 and I was wondering if this could be a layer for that. Thoughts?

[riedgar-ms]

This is great @riedgar-ms ! Do you have any thoughts about how this might connect to Pedantic as well? For example what if people have a pydantic spec, should they export a schema from pydantic and then we build a grammar from that? I know there was some pydantic work #559 and I was wondering if this could be a layer for that. Thoughts?

I was assuming that once this was in place, we could use:
https://docs.pydantic.dev/latest/concepts/json_schema/

[Harsha-Nori]

One thing -- I really liked that I could use Annotated type annotations to specify constraints, e.g. patterns that string fields have to match on.

I couldn't preserve that behavior without making the EarleyParser scream at me though, despite generative forward passes working perfectly (i.e. the EarleyParser gave exceptions for strings that were generated via the grammar). Not sure if that's a bug in the parser or if using it with the gen interface is not a supported use-case.

@hudson-ai, thanks for all the enthusiasm and progress here! Do you have a code snippet to reproduce the Parser issue on your fork? I'm happy to take a look.

[Harsha-Nori]

Seems reasonable to generate compact JSONs, which users can then pretty-print themselves.

[hudson-ai]

@hudson-ai, thanks for all the enthusiasm and progress here! Do you have a code snippet to reproduce the Parser issue on your fork? I'm happy to take a look.

My pleasure. This library is the best thing since sliced bread, and I really wanted the ability to generate structured data according to a schema -- pure coincidence that we made parallel implementations on the same day!

I changed the implementation of gen_str on my fork, which ameliorates this issue, but here's a minimal(ish) snippet to reproduce:

from guidance._grammar import Byte, GrammarFunction, Join, Select, select
from guidance.library._gen import gen
from guidance._parser import EarleyCommitParser, ParserException

_QUOTE = Byte(b'"')
_COMMA = Byte(b',')
_OPEN_BRACKET = Byte(b'[')
_CLOSE_BRACKET = Byte(b']')

def check_string_with_grammar(input_string: str, grammar: GrammarFunction):
    parser = EarleyCommitParser(grammar)

    print(f"Checking {input_string}")
    for c in input_string:
        print(f"Working on: {c}")
        print(f"Valid next bytes: {parser.valid_next_bytes()}")
        next_byte = bytes(c, encoding="utf8")
        print(f"Consuming: {next_byte}")
        parser.consume_byte(next_byte)

def gen_str():
    # using _SAFE_STR instead of gen here fixes the error
    return Join([_QUOTE, gen(stop='"'), _QUOTE])

def gen_list_of_str() -> GrammarFunction:
    s = Select([], capture_name=None, recursive=True)
    s.values = [gen_str(), Join([s,  _COMMA,  gen_str()])]
    return _OPEN_BRACKET + select([_CLOSE_BRACKET, Join([s, _CLOSE_BRACKET])])

def test_list_of_str():
    example = '["a","b","c","d"]'
    grammar = gen_list_of_str()
    check_string_with_grammar(example, grammar)

# This errs, but it shouldn't?
test_list_of_str()

[hudson-ai]

I hypothesize that the issue has to do with the stop kwarg in gen (although check_string_with_grammar('"cake"', gen_str()) is fine...)

[hudson-ai]

@Harsha-Nori I split the parser problem out into its own issue as it doesn't really belong in this PR. Hope I was clear enough over there.
#624

[Harsha-Nori]

I think we should shift this logic over to the GrammarFunction.match() method -- I'd slowly like to use that method as the main check for grammar compliance across the codebase

[riedgar-ms]

Done

[Harsha-Nori]

Assume this can leverage the check_string_with_grammar function defined earlier right?

[riedgar-ms]

This should all be changed to use GrammarFunction.match()

[Harsha-Nori]

@riedgar-ms I think this looks great so far! Would be nice to have a test that leverages an actual guidance.models object -- can be Mock, if we need to -- to verify that the end-to-end grammar constrained generation works against a full model object.

[riedgar-ms]

@riedgar-ms I think this looks great so far! Would be nice to have a test that leverages an actual guidance.models object -- can be Mock, if we need to -- to verify that the end-to-end grammar constrained generation works against a full model object.

Added a smoke test using the Mock model.

[Harsha-Nori]

Thanks @riedgar-ms for writing this up! I think there's a few more open questions here -- layering in commit_point() appropriately, use of Byte strings vs. regular strings, etc. -- but we can handle those later iteratively. Really nice to see some solid tests here, including use of the end to end guidance.models.Mock. I'm comfortable merging this in as a WIP for now, since it isn't user facing yet and we don't want to diverge too much from main, and we can keep working together on pushing support further. Great stuff!

[Harsha-Nori]

Also, thanks @hudson-ai for all the comments and implementations you've done on your fork! Excited to see this continue to progress with your help too :).