main-interactive-mode: optionally allow for special tokens from user in interactive mode for fill-in-middle etal

[hanishkvc]

Add a flag --interactive-specials to examples/main

This controls whether the user entered text in interactive mode is tokenized with parse_special flag set or not.

When main is run with this flag, it will allow users to enter any special tokens for fill-in-mode or equivalent supported by any ai model, so that the model can do the same at the appropriate location, based on the surrounding context in the user text.

@ggerganov @teleprint-me

I havent tested this yet, but logically this should allow for the above mentioned use case.

[hanishkvc]

Tested fill-in-middle sample prompt mentioned wrt Refact 1.6B,

and it correctly generates a suitable comment as expected that should be placed in-between the user provided code context in the sample prompt, when --interactive-specials argument is passed to main. And the proper token ids are generated for <fim_prefix>, <fim_suffix> and <fim_middle>

Without --interactive-specials argument, it generates a some what repeating general note about the code in the given sample prompt. And as expected <fim_prefix> <fim_suffix> and <fim_middle> dont get converted to proper tokens, because parse_special wont be set when tokenizing.

[hanishkvc]

Have force pushed a merge with the master. @mofosyne , I think I overwrote your equivalent merge. Sorry I hadnt noticed that Allow edits by maintainers was enabled.

[mofosyne]

Not a problem. I would prefer that as I can't always guess your full intent. You are a more accurate author than I am. But I was hoping to at least make your life easier :)

[mofosyne]

Double checked that llama_tokenize() forth arguments is actually for special/control token in method definitions. Looks right.

Can observe change to code that would add a new flag that would allow for enabling the ability to add a special token in the middle of the embedding stream.