You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I reviewed the Discussions, and have a new bug or useful enhancement to share.
Expected Behavior
Add a public API (e.g., llama_set_logits(ctx, logits)) that allows to manually set the logits. This would open the door to bridging llama.cpp with use cases like FUDGE (code) and Grounded Decoding, which guides the generation process by manipulating the probabilities without the need for fine-tuning.
Current Behavior
I understand that there is currently an API function for getting the current logits (llama_get_logits(ctx)), but no way to modify the logits through a public API.
The text was updated successfully, but these errors were encountered:
The function llama_get_logits() returns the pointer to the logits array in llama.cpp's state. To modify the logits, you just have to write to the array. You can an example in main.cpp itself:
auto logits = llama_get_logits(ctx);
if (params.ignore_eos) {
logits[llama_token_eos()] = 0;
}
Still, maybe not the best API, better would be something like this which makes a copy:
// length from llama_n_vocab()LLAMA_APIintllama_get_logits(structllama_context*ctx, float*logits);
LLAMA_APIintllama_set_logits(structllama_context*ctx, constfloat*logits);
// or maybeLLAMA_APIfloatllama_get_logit(structllama_context*ctx, llama_tokentoken);
LLAMA_APIvoidllama_set_logit(structllama_context*ctx, llama_token, floatlogit);
Prerequisites
Please answer the following questions for yourself before submitting an issue.
Expected Behavior
Add a public API (e.g.,
llama_set_logits(ctx, logits)
) that allows to manually set the logits. This would open the door to bridging llama.cpp with use cases like FUDGE (code) and Grounded Decoding, which guides the generation process by manipulating the probabilities without the need for fine-tuning.Current Behavior
I understand that there is currently an API function for getting the current logits (
llama_get_logits(ctx)
), but no way to modify the logits through a public API.The text was updated successfully, but these errors were encountered: