KeyError when loading Mistral 7b via Transformers

[lachlancahill]

The bug
attempting to load the model yields the following output:

Loading checkpoint shards: 100%|██████████| 3/3 [00:06<00:00,  2.09s/it]
Traceback (most recent call last):
  File "C:\Users\lachl\PycharmProjects\llm_feedback_analytics\guidance_error.py", line 8, in <module>
    mistral = models.Transformers(model_id, echo=False, device_map=device, torch_dtype=torch.bfloat16)
  File "C:\Users\lachl\PycharmProjects\llm_feedback_analytics\.venv\lib\site-packages\guidance\models\transformers\_transformers.py", line 196, in __init__
    TransformersEngine(model, tokenizer, compute_log_probs, **kwargs),
  File "C:\Users\lachl\PycharmProjects\llm_feedback_analytics\.venv\lib\site-packages\guidance\models\transformers\_transformers.py", line 101, in __init__
    TransformersTokenizer(model, tokenizer),
  File "C:\Users\lachl\PycharmProjects\llm_feedback_analytics\.venv\lib\site-packages\guidance\models\transformers\_transformers.py", line 36, in __init__
    reconstructed += bytes([byte_decoder[c] for c in t.convert_ids_to_tokens(id)])
  File "C:\Users\lachl\PycharmProjects\llm_feedback_analytics\.venv\lib\site-packages\guidance\models\transformers\_transformers.py", line 36, in <listcomp>
    reconstructed += bytes([byte_decoder[c] for c in t.convert_ids_to_tokens(id)])
KeyError: '▁'

To Reproduce

import torch
from guidance import models

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model_id = "mistralai/Mistral-7B-Instruct-v0.2"

mistral = models.Transformers(model_id, echo=False, device_map=device, torch_dtype=torch.bfloat16)

System info (please complete the following information):

• OS: Windows 11
• Guidance Version (guidance.__version__): 0.1.13

[riedgar-ms]

I can reproduce this. @slundberg is this another aspect of the unicode error you pushed a fix for earlier this week?

[riedgar-ms]

I believe that this is at least part of what is going wrong in #716

[JosephGatto]

Any update on this? Also getting same error. On Google Colab.

[riedgar-ms]

I've not heard from @slundberg who knows most about this. I've just started prodding it again

[riedgar-ms]

As a work around @JosephGatto and @lachlancahill , it does appear that if you load mistral via llama-cpp (there is code for this in #716 ), it works.

[MikoAL]

I get this issue no matter what model I'm using, even if it was not based on mistral, any fixes?

[MikoAL]

After a bit of testing, I switched to the commit where the version number was 0.1.12, the same error happened, switched to the commit where the version number was 0.1.11, and it started working again?

[talglobus]

Running into this issue as well, on a model finetuned from Mistral, loaded directly from pytorch

[lachlancahill]

This issue has not been resolved I'm afraid. I have force-reinstalled from git and still get the same error.

[amirabdullah19852020]

Yes, this should still be open. I'm running into this error too on latest version.

[akshaymg99]

Running into the same error, with latest version