Add Phi-4 to Tiktoken encoding map

[luisquintanilla]

Phi-4 uses Tiktoken tokenizer (100k vocab).

https://arxiv.org/pdf/[2412.08905v1](https://arxiv.org/pdf/2412.08905v1)

we now use the tiktoken tokenizer (for better multilingual support) with a padded vocabulary size of 100,352 (including unused tokens)

Consider adding it as an option to the encoding map so it's easier to create.

machinelearning/src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs

Lines 1025 to 1035 in 01c4164

	private static readonly (string Prefix, ModelEncoding Encoding)[] _modelPrefixToEncoding =
	[
	// chat
	( "o1-", ModelEncoding.O200kBase ), // e.g. o1-mini
	( "gpt-4o-", ModelEncoding.O200kBase), // e.g., gpt-4o-2024-05-13
	( "gpt-4-", ModelEncoding.Cl100kBase), // e.g., gpt-4-0314, etc., plus gpt-4-32k
	( "gpt-3.5-", ModelEncoding.Cl100kBase), // e.g, gpt-3.5-turbo-0301, -0401, etc.
	( "gpt-35-", ModelEncoding.Cl100kBase ) // Azure deployment name
	];
	private static readonly Dictionary<string, ModelEncoding> _modelToEncoding =

[MaxAkbar]

I got the model to load using the following code, and generate text but I can't seem to workout out the stop sequence:

using Microsoft.Extensions.AI;
using Microsoft.ML.GenAI.Core;
using Microsoft.ML.GenAI.Phi;
using Microsoft.ML.Tokenizers;
using static TorchSharp.torch;
using TorchSharp;
using System.Text.Json;

var weightFolder = @"C:\Users\maxim\source\repos\models\microsoft\phi-4\";
var device = "cuda";
if (device == "cuda")
{
    InitializeDeviceType(DeviceType.CUDA);
}

var defaultType = ScalarType.Float16;
manual_seed(1);
set_default_dtype(defaultType);

var model = Phi3ForCasualLM.FromPretrained(weightFolder, "config.json", layersOnTargetDevice: -1, quantizeToInt4: true);
var tokenizerPath = Path.Combine(weightFolder, "config.json");
var fileConfig = File.ReadAllText(tokenizerPath);
var config = JsonSerializer.Deserialize<Phi3Config>(fileConfig)!;
var tokenizer = TiktokenTokenizer.CreateForModel("gpt-4");
var pipeline = new CausalLMPipeline<Tokenizer, Phi3ForCasualLM>(tokenizer, model, device);
var client = new Phi3CausalLMChatClient(pipeline);

var task = """
            Can you tell me a funny joke?
            """;
var chatMessage = new ChatMessage(ChatRole.User, task);
var options = new ChatOptions 
{
    StopSequences = ["<|endoftext|>"],

};

await foreach (var response in client.CompleteStreamingAsync([chatMessage], options))
{
    Console.Write(response.Text);
}

Console.WriteLine();
Console.WriteLine("End!");