-
Notifications
You must be signed in to change notification settings - Fork 426
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add an infilling DSL #182
Comments
I'll tie in one other must-have feature, imo: #305 This is where each infilled generation is accessible via a key, at the end of generation, like in prompt = '''
Here's one sentence: {{gen "SENTENCE_1"}}
Now, another sentence: {{gen "SENTENCE_2"}}
'''
out = guidance(prompt)()
print(out['SENTENCE_1'])
# <prints what the LLM generated>
print(out['SENTENCE_2'])
# <prints the other sentence> As a separate question, could we optionally return, and reuse a kv-cache? This would make your hand-rolled-concatenation example mostly work, already, aside from the nice syntactic sugar you suggest above. |
We opened #190 to discuss this. Does the interface there make sense? This is indeed an important feature that we'll re-prioritise. |
DSLs generally suck, so closing this in favor of #667 which would provide the same functionalities and more. |
Workflow with LLMs often imply recursively calling models where at each step we concatenate the result of the previous call to the prompt. Consider the following example:
The fact that we have to do the concatenation manually can quickly become cumbersome. Worse, we loose the KV cache across generations. I thus suggest to add a thin infilling DSL to outlines with the new API
A few quick points:
stop_at
kwarg from the template;The text was updated successfully, but these errors were encountered: