LlamaCpp doesnt work with generate.fsm for custom FSMs

[Radu1999]

Describe the issue as clearly as possible:

The example with custom fsm from documentation doesnt work for LlamaCpp as

logits, kv_cache = model(token_ids, attention_masks, kv_cache)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'LlamaCpp' object is not callable

Steps/code to reproduce the bug:

from transformers import AutoTokenizer
from outlines import models, generate
from outlines.models.transformers import TransformerTokenizer
from llama_cpp import Llama
import interegular
import torch

if __name__ == "__main__":
    # Create model
    llm = Llama("./models/Mistral-7B-Instruct-v0.2/mistral-7b-instruct-v0.2.Q5_K_M.gguf")
    model = models.LlamaCpp(llm)
    model.tokenizer = TransformerTokenizer(AutoTokenizer.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2', use_fast=True))
    model.device = 'cpu'

    # Create fsm
    list_of_strings_pattern = """\["[^"\s]*"(?:,"[^"\s]*")*\]"""
    pink_elephant_pattern = """.*(pink|elephant).*"""

    list_of_strings_fsm = interegular.parse_pattern(list_of_strings_pattern).to_fsm()
    pink_elephant_fsm = interegular.parse_pattern(pink_elephant_pattern).to_fsm()

    difference_fsm = list_of_strings_fsm - pink_elephant_fsm

    generator = generate.fsm(model, difference_fsm)
    rng = torch.Generator(device="cpu")
    rng.manual_seed(789005)

    response = generator("[INST] Don't talk about pink elephants [/INST]")
    print(response)

Expected result:

I d expect it to work :)

Error message:

No response

Outlines/Python version information:

Version information
latest

Context for the issue:

No response

[lapp0]

It will be a bit before this is merged into main, but you can try it early with

pip install --upgrade git+https://github.com/lapp0/outlines@fix-llamacpp-fsm

Works on my end, please let me know if you run into any issues!