Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logprobs and loglikelihood #72

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Red-Caesar
Copy link

Update docs with logprobs and loglikelihood.

Additional notebook to run completions with these parameters.

@@ -167,7 +173,8 @@ Parameters
_ **content** *(string)*: The actual text content of the chat completion.
_ **function_call** _(object or null)_: An optional field that may contain information about a function call made within the message. It's usually `null` in standard responses.
_ **delta** *(object or null)*: An optional field that can contain additional metadata about the message, typically `null`.
_ **finish_reason** _(string)_: The reason why the message generation was stopped, such as reaching the maximum length (`"length"`).
_ **finish_reason** _(string)_: The reason why the message generation was stopped, such as reaching the maximum length (`"length"`).
_ **logprobs** _(object)_: An object representing the token, its log probability and the most probable tokens to this one.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the most probable tokens to this one" what does this mean?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Red-Caesar - Are you saying the other tokens most likely to be selected? I.e. if "cat" was chosen, but "dog" and "mouse" were the 2nd and 3rd most likely tokens to be selected, those would be included in the output?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it will be included in the response if we set the top_logprobs > 1. For example, we ask: "Create a story about a cat".
The response will be in the following format:

"choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": " In a quaint, cobblestone town,",
        "tool_calls": null
      },
      "logprobs": {
        "content": [
          {
            "token": " In",
            "logprob": -0.7101921439170837,
            "bytes": null,
            "top_logprobs": [
              {
                "token": " In",
                "logprob": -0.7101921439170837,
                "bytes": null
              },
              {
                "token": " Once",
                "logprob": -1.9485827684402466,
                "bytes": null
              }
            ]
          },
          .... other tokens

@Red-Caesar
Copy link
Author

I'm a bit concerned about the loglikelihood because if users use the OpenAI API, they may have some difficulties sending this parameter correctly. It should be sent in a specific format:

import openai

client = openai.OpenAI(
    base_url=ENDPOINT + "/v1",
    api_key=OCTOAI_TOKEN,
)

completion = client.chat.completions.create(
        model="mistral-7b-instruct",
        messages=[
            {
                "role": "user",
                "content": "Create a story about a cat",
            }
        ],
        extra_body={"loglikelihood":True},
)

So, should we write more about this? If so, I'm not sure where in the documentation it would be best to do so.

@Red-Caesar
Copy link
Author

@BenHamm @devonbrown50

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants