llamacpp tests failing in main

[lapp0]

Describe the issue as clearly as possible:

Failures:

tests/generate/test_integration_llamacpp.py .FF.x......FFFFFFFFFFFFFF

All failures have a similar traceback. Example:

_________________ test_llamacpp_generation_api[regex-params1] __________________

model = <outlines.models.llamacpp.LlamaCpp object at 0x7ff76c1591b0>
generator_type = <function regex at 0x7ff81d254940>, params = ('[0-9]',)

    @pytest.mark.parametrize(
        "generator_type,params",
        (
            (generate.text, []),
            (generate.regex, ("[0-9]",)),
            (generate.cfg, (grammars.arithmetic,)),
        ),
    )
    def test_llamacpp_generation_api(model, generator_type, params):
>       generator = generator_type(model, *params)

tests/generate/test_integration_llamacpp.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/functools.py:889: in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/outlines/generate/regex.py:48: in regex_llamacpp
    logits_processor = RegexLogitsProcessor(regex_str, llm=model.model)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/outlines/integrations/llamacpp.py:154: in __init__
    fsm = RegexGuide(regex_string, tokenizer)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/outlines/fsm/guide.py:145: in __init__
    ) = create_states_mapping(regex_string, tokenizer)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/outlines/caching.py:118: in wrapper
    result = wrapper.__memory__.get(cache_key, default=ENOVAL, retry=True)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/diskcache/core.py:1149: in get
    db_key, raw = self._disk.put(key)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/outlines/caching.py:19: in put
    data = cloudpickle.dumps(key)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/cloudpickle/cloudpickle.py:1479: in dumps
    cp.dump(obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <cloudpickle.cloudpickle.Pickler object at 0x7ff76add7b80>
obj = ('outlines.fsm.guide.create_states_mapping', '[0-9]', <outlines.integrations.llamacpp.LlamaCppTokenizer object at 0x7ff76ab05360>, None)

    def dump(self, obj):
        try:
>           return super().dump(obj)
E           ValueError: ctypes objects containing pointers cannot be pickled

/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/cloudpickle/cloudpickle.py:1245: ValueError

Full run example:
https://github.com/lapp0/outlines/actions/runs/9252615735/job/25450574607

Failures started between May 25 06:24 GMT - May 26 08:04 GMT

Steps/code to reproduce the bug:

pytest -s tests/generate/test_integration_llamacpp.py

Expected result:

Tests pass

[lapp0]

This appears to be related to #911 which updates create_states_mapping() such that it is directly passed the tokenizer.

We're trying to pickle a LlamaCppTokenizer which isn't possible. #911 is a good change-set otherwise, we shouldn't revert. Instead we should ensure the llama.cpp tokenizer passed to fsm/guide.py / create_states_mapping() is pickle-able, as is the case with models/transformers.py / TransformerTokenizer.

[lapp0]

Tests are now passing in main. https://github.com/lapp0/outlines/actions/runs/9281048868/job/25536358249?pr=23

It's not obvious to me which library being changed fixed this. These are the libraries which have new releases since this issue was opened:

2024-05-27T08:30:16: huggingface-hub
2024-05-27T21:45:06: triton
2024-05-28T17:32:52: pydantic_core
2024-05-28T18:38:48: pydantic

Please re-open if you'd like a retrospective analysis.

[lapp0]

Issue is occurring again, will investigate a fix.