[FR] Output Confidence Score

[KenjiBaheux]

Enable developers to filter LLM responses based on confidence. This could be achieved by providing a confidence score with each response, potentially derived from per-token log-likelihood. This would improve the reliability of LLM-powered applications by allowing developers to reject low-confidence outputs.

[christianliebel]

The OpenAI API does not return strings as Prompt API does, but chat completion/chunk objects that contain the logprobs if the user has enabled them when creating the chat session. These objects also support reporting back multiple choices if the user has requested them. However, I think it is tedious to unbox the answer from these objects if you didn't request logprobs or additional choices.

[domenic]

Thanks for the extra context @christianliebel. I think we want to align to developer expectations as much as possible by reusing the types of object shapes they've seen elsewhere. Although, I would really prefer if we didn't have to use underscored names like finish_reason in a web API, heh.

I see three possibilities here:

1. Add a second pair of methods that return objects with extra info. (Not sure what to name these.)
2. Allow configuration at session creation time, and have that change the return value from strings to objects.
3. Always return objects, and require the tedious unboxing.

I'm somewhat attracted by (2), but I don't know how web developers feel about it.

[tomayac]

I'm somewhat attracted by (2), but I don't know how web developers feel about it.

+1 to option 2.. This follows the "Make the easy things easy, and the hard things possible" principle.