epic: format generated code in code format (not plain text)

[xx88xx]

Motivation

Seems all the other LLMs do this now, so would be very useful comfort to have in Jan!

[Van-QA]

hi @xx88xx, sorry, is this what you mean?

The copy button will help copy code in the same format that is being displayed 🙏

[mr-september]

I also sometimes run into this issue. Does anyone know if different models maybe use different syntax for things like code blocks? Like how we have multiple different prompt templates/tokens?

[xx88xx]

Hi @Van-QA !
Thanks for the response. Since posting this, noticed it works in the more popular models. I agree with @mr-september sometimes it works as in the picture you attached, but often especially on some of the more bespoke models it shows up unformatted. @mr-september might be onto something with the different syntax.

[Van-QA]

thank you, can you let us know more details of the failed model / step to reproduce? @xx88xx

[mr-september]

With some further testing/hindsight, I think I only experienced this when using Llama3/fine-tunes released in the first week or so, where there were issues with the tokens or something, e.g.

1. Special tokens are not rendered correctly (as empty) -- llama3 specific? ggerganov/llama.cpp#6770
2. https://www.reddit.com/r/LocalLLaMA/comments/1cltac3/part3_cause_to_issue_found_possible_bug_llama3/

Recently I haven't ran into it again. So it's likely some kind of syntax/formatting-specific thing, as previously suspected. No idea if this was also the case for @xx88xx?

[Realmbird]

Can I work on this issue

[imtuyethan]

Can I work on this issue

YES PLEASE :"> @Realmbird

[imtuyethan]

Assigned @Van-QA to follow up when needed.

[imtuyethan]

Hey @Realmbird how is it going so far

[freelerobot]

Works for me on v0.5.3+
closing