llama : add Deepseek support

[ggerganov]

Support is almost complete. There is a dangling issue with the pre-tokenizer: #7036

A useful discussion related to that is here: #7144

---

Outdated below

Creating this issue for more visibility

The main problem is around tokenization support, since the models use some variation of the BPE pre-processing regex. There are also some issues with the conversion scripts.

Anyway, looking for contributions to help with this

Previous unfinished work:

• Update gpt2 preprocess and add deepseek coder preprocess #4070
• Deepseek coder merge #5464

Possible implementation plan: #5464 (comment)

[dragnil1]

Hello, @ggerganov, I'd like to try working on it as my good first issue.

[ggerganov]

Ok 👍 keep us posted

[Kangmo]

Waiting for this to come. deepseek model is famouse for coding well and Korean language ability.

[fostiropoulos]

In the supported models there is Deepseek. I am able to use DeepSeekCoder-33B with comparatively similar results to their API. Can someone please clarify what are the failure cases with the current tokenization?

[Columpio]

@fostiropoulos There may be a problem with deepseek coder 1.3b, which might be somehow irreprodicible in 6.7b, 7b, and 33b.
See this

[mirek190]

any progress with Deepseek support?

[hyperbolic-c]

Hey, so it could support DeepSeek-coder model after #6920 be merged? Thanks !

[ggerganov]

I've updated the description of the issue with the latest state.

Support is pretty much complete, though there are some edge cases during tokenization that are handled incorrectly. For example the letter ü is tokenized to a different token than it should - more info in the referenced links

I think we can declare DeepSeek models supported and handle the problem above into a separate task related to correct processing of added tokens, since it is not specific for DeepSeek models