[V1][spec decode] return logprobs for spec decoding #26060

TheEpicDolphin · 2025-10-01T23:30:48Z

Purpose

Add support for returning logprobs for v1 spec decoding.

Test Plan

Automated Tests

Added automated testing for logprobs which compares spec decode LLM per-token output logprobs with those of a reference model. The comparison is done for: raw_logits, raw_logprobs, processed_logits, and processed_logprobs.

(py312conda) bash-5.1$ pytest tests/v1/sample/test_logprobs.py -k test_spec_decode_logprobs
============================================================================================================================================= test session starts =============================================================================================================================================
platform linux -- Python 3.12.9, pytest-8.4.1, pluggy-1.6.0
rootdir: /data/users/gdelfin/gitrepos/vllm
configfile: pyproject.toml
plugins: anyio-4.9.0, asyncio-1.1.0
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 30 items / 26 deselected / 4 selected                                                                                                                                                                                                                                                               

tests/v1/sample/test_logprobs.py ....                                                                                                                                                                                                                                                                   [100%]

============================================================================================================================================== warnings summary ===============================================================================================================================================
tests/v1/sample/test_logprobs.py::test_spec_decode_logprobs[model_setup0-raw_logits]
tests/v1/sample/test_logprobs.py::test_spec_decode_logprobs[model_setup0-raw_logits]
tests/v1/sample/test_logprobs.py::test_spec_decode_logprobs[model_setup0-raw_logprobs]
tests/v1/sample/test_logprobs.py::test_spec_decode_logprobs[model_setup0-raw_logprobs]
tests/v1/sample/test_logprobs.py::test_spec_decode_logprobs[model_setup0-processed_logits]
tests/v1/sample/test_logprobs.py::test_spec_decode_logprobs[model_setup0-processed_logits]
tests/v1/sample/test_logprobs.py::test_spec_decode_logprobs[model_setup0-processed_logprobs]
tests/v1/sample/test_logprobs.py::test_spec_decode_logprobs[model_setup0-processed_logprobs]
  /home/gdelfin/.conda/envs/py312conda/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=2004414) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================================================================== 4 passed, 26 deselected, 8 warnings in 207.60s (0:03:27) ===========================================================================================================================
(py312conda) bash-5.1$

Rejection sampler still works as expected:

(py312conda) bash-5.1$ pytest tests/v1/sample/test_rejection_sampler.py
================================================================================================================================================ test session starts ================================================================================================================================================
platform linux -- Python 3.12.9, pytest-8.4.1, pluggy-1.6.0
rootdir: /data/users/gdelfin/gitrepos/vllm
configfile: pyproject.toml
plugins: anyio-4.9.0, asyncio-1.1.0
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 34 items                                                                                                                                                                                                                                                                                                  

tests/v1/sample/test_rejection_sampler.py ..................................                                                                                                                                                                                                                                  [100%]

================================================================================================================================================ 34 passed in 24.12s ================================================================================================================================================

Manual Test

Setup
Ran the following for spec decode LLM server:

export LLAMA_MODEL=meta-llama/Llama-3.1-8B-Instruct
export DRAFT_MODEL=yuhuili/EAGLE-LLaMA3.1-Instruct-8B
export VLLM_USE_V1=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export SPEC_DEC_CONFIG='{"method": "eagle", "model": "'$DRAFT_MODEL'", "num_speculative_tokens": 3, "draft_tensor_parallel_size": 1, "max_model_len": 2048}'
vllm serve $LLAMA_MODEL --disable-log-requests --tensor-parallel-size=1 --max-num-seqs=64 --max-model-len=32768 --no-enable-prefix-caching --speculative-config="$SPEC_DEC_CONFIG" 2>&1 | tee ~/server_logs/vllm_server.log

And the following for standard decode LLM server:

export LLAMA_MODEL=meta-llama/Llama-3.1-8B-Instruct
export VLLM_USE_V1=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
vllm serve $LLAMA_MODEL --disable-log-requests --tensor-parallel-size=1 --max-num-seqs=64 --max-model-len=32768 --no-enable-prefix-caching 2>&1 | tee ~/server_logs/vllm_server.log

Single Request Test
Use the following to send a request:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain the theory of relativity in simple terms."}],temperature=0.0,logprobs=True,top_logprobs=2,max_tokens=32)

Verified that both standard LLM and spec decode LLM return identical logprobs outputs:

ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=-0.0193302184343338, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.0193302184343338), TopLogprob(token='Albert', bytes=[65, 108, 98, 101, 114, 116], logprob=-4.769330024719238)]), ChatCompletionTokenLogprob(token=' theory', bytes=[32, 116, 104, 101, 111, 114, 121], logprob=-0.01106177270412445, top_logprobs=[TopLogprob(token=' theory', bytes=[32, 116, 104, 101, 111, 114, 121], logprob=-0.01106177270412445), TopLogprob(token=' Theory', bytes=[32, 84, 104, 101, 111, 114, 121], logprob=-4.511061668395996)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0), TopLogprob(token=' Of', bytes=[32, 79, 102], logprob=-19.625)]), ChatCompletionTokenLogprob(token=' rel', bytes=[32, 114, 101, 108], logprob=-1.4424220353248529e-05, top_logprobs=[TopLogprob(token=' rel', bytes=[32, 114, 101, 108], logprob=-1.4424220353248529e-05), TopLogprob(token=' Rel', bytes=[32, 82, 101, 108], logprob=-11.250014305114746)]), ChatCompletionTokenLogprob(token='ativity', bytes=[97, 116, 105, 118, 105, 116, 121], logprob=-1.1920928244535389e-07, top_logprobs=[TopLogprob(token='ativity', bytes=[97, 116, 105, 118, 105, 116, 121], logprob=-1.1920928244535389e-07), TopLogprob(token='ativa', bytes=[97, 116, 105, 118, 97], logprob=-17.0)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.434250146150589, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.434250146150589), TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-1.0592501163482666)]), ChatCompletionTokenLogprob(token=' developed', bytes=[32, 100, 101, 118, 101, 108, 111, 112, 101, 100], logprob=-0.22613893449306488, top_logprobs=[TopLogprob(token=' developed', bytes=[32, 100, 101, 118, 101, 108, 111, 112, 101, 100], logprob=-0.22613893449306488), TopLogprob(token=' proposed', bytes=[32, 112, 114, 111, 112, 111, 115, 101, 100], logprob=-1.851138949394226)]), ChatCompletionTokenLogprob(token=' by', bytes=[32, 98, 121], logprob=-5.483612312673358e-06, top_logprobs=[TopLogprob(token=' by', bytes=[32, 98, 121], logprob=-5.483612312673358e-06), TopLogprob(token=' primarily', bytes=[32, 112, 114, 105, 109, 97, 114, 105, 108, 121], logprob=-12.250005722045898)]), ChatCompletionTokenLogprob(token=' Albert', bytes=[32, 65, 108, 98, 101, 114, 116], logprob=-0.0001212284987559542, top_logprobs=[TopLogprob(token=' Albert', bytes=[32, 65, 108, 98, 101, 114, 116], logprob=-0.0001212284987559542), TopLogprob(token='Albert', bytes=[65, 108, 98, 101, 114, 116], logprob=-9.875121116638184)]), ChatCompletionTokenLogprob(token=' Einstein', bytes=[32, 69, 105, 110, 115, 116, 101, 105, 110], logprob=0.0, top_logprobs=[TopLogprob(token=' Einstein', bytes=[32, 69, 105, 110, 115, 116, 101, 105, 110], logprob=0.0), TopLogprob(token=' Ein', bytes=[32, 69, 105, 110], logprob=-17.75)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.0021873614750802517, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.0021873614750802517), TopLogprob(token=' in', bytes=[32, 105, 110], logprob=-6.127187252044678)]), ChatCompletionTokenLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.018868248909711838, top_logprobs=[TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.018868248909711838), TopLogprob(token=' explains', bytes=[32, 101, 120, 112, 108, 97, 105, 110, 115], logprob=-4.643868446350098)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.012281978502869606, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.012281978502869606), TopLogprob(token=' actually', bytes=[32, 97, 99, 116, 117, 97, 108, 108, 121], logprob=-4.512281894683838)]), ChatCompletionTokenLogprob(token=' fundamental', bytes=[32, 102, 117, 110, 100, 97, 109, 101, 110, 116, 97, 108], logprob=-0.1662641167640686, top_logprobs=[TopLogprob(token=' fundamental', bytes=[32, 102, 117, 110, 100, 97, 109, 101, 110, 116, 97, 108], logprob=-0.1662641167640686), TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-2.291264057159424)]), ChatCompletionTokenLogprob(token=' concept', bytes=[32, 99, 111, 110, 99, 101, 112, 116], logprob=-0.0016553997993469238, top_logprobs=[TopLogprob(token=' concept', bytes=[32, 99, 111, 110, 99, 101, 112, 116], logprob=-0.0016553997993469238), TopLogprob(token=' idea', bytes=[32, 105, 100, 101, 97], logprob=-6.501655578613281)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.0001674750237725675, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.0001674750237725675), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-8.750167846679688)]), ChatCompletionTokenLogprob(token=' physics', bytes=[32, 112, 104, 121, 115, 105, 99, 115], logprob=-0.4768524765968323, top_logprobs=[TopLogprob(token=' physics', bytes=[32, 112, 104, 121, 115, 105, 99, 115], logprob=-0.4768524765968323), TopLogprob(token=' modern', bytes=[32, 109, 111, 100, 101, 114, 110], logprob=-0.9768524765968323)]), ChatCompletionTokenLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.05488456413149834, top_logprobs=[TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.05488456413149834), TopLogprob(token='.', bytes=[46], logprob=-2.929884672164917)]), ChatCompletionTokenLogprob(token=' explains', bytes=[32, 101, 120, 112, 108, 97, 105, 110, 115], logprob=-0.47857409715652466, top_logprobs=[TopLogprob(token=' explains', bytes=[32, 101, 120, 112, 108, 97, 105, 110, 115], logprob=-0.47857409715652466), TopLogprob(token=' changed', bytes=[32, 99, 104, 97, 110, 103, 101, 100], logprob=-2.10357403755188)]), ChatCompletionTokenLogprob(token=' how', bytes=[32, 104, 111, 119], logprob=-0.033877886831760406, top_logprobs=[TopLogprob(token=' how', bytes=[32, 104, 111, 119], logprob=-0.033877886831760406), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-3.4088778495788574)]), ChatCompletionTokenLogprob(token=' space', bytes=[32, 115, 112, 97, 99, 101], logprob=-0.48089107871055603, top_logprobs=[TopLogprob(token=' space', bytes=[32, 115, 112, 97, 99, 101], logprob=-0.48089107871055603), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-1.2308911085128784)]), ChatCompletionTokenLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.04858766868710518, top_logprobs=[TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.04858766868710518), TopLogprob(token=',', bytes=[44], logprob=-3.0485875606536865)]), ChatCompletionTokenLogprob(token=' time', bytes=[32, 116, 105, 109, 101], logprob=-1.1920928244535389e-07, top_logprobs=[TopLogprob(token=' time', bytes=[32, 116, 105, 109, 101], logprob=-1.1920928244535389e-07), TopLogprob(token='.time', bytes=[46, 116, 105, 109, 101], logprob=-16.75)]), ChatCompletionTokenLogprob(token=' are', bytes=[32, 97, 114, 101], logprob=-0.019477184861898422, top_logprobs=[TopLogprob(token=' are', bytes=[32, 97, 114, 101], logprob=-0.019477184861898422), TopLogprob(token=' work', bytes=[32, 119, 111, 114, 107], logprob=-4.269477367401123)]), ChatCompletionTokenLogprob(token=' connected', bytes=[32, 99, 111, 110, 110, 101, 99, 116, 101, 100], logprob=-0.08927204459905624, top_logprobs=[TopLogprob(token=' connected', bytes=[32, 99, 111, 110, 110, 101, 99, 116, 101, 100], logprob=-0.08927204459905624), TopLogprob(token=' related', bytes=[32, 114, 101, 108, 97, 116, 101, 100], logprob=-2.8392720222473145)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=-0.008059008046984673, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=-0.008059008046984673), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-5.258059024810791)]), ChatCompletionTokenLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.7004125118255615, top_logprobs=[TopLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.7004125118255615), TopLogprob(token=' I', bytes=[32, 73], logprob=-1.2004125118255615)]), ChatCompletionTokenLogprob(token="'s", bytes=[39, 115], logprob=-0.15934917330741882, top_logprobs=[TopLogprob(token="'s", bytes=[39, 115], logprob=-0.15934917330741882), TopLogprob(token=' consists', bytes=[32, 99, 111, 110, 115, 105, 115, 116, 115], logprob=-2.534349203109741)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.5355489253997803, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.5355489253997803), TopLogprob(token=' divided', bytes=[32, 100, 105, 118, 105, 100, 101, 100], logprob=-1.5355489253997803)]), ChatCompletionTokenLogprob(token=' bit', bytes=[32, 98, 105, 116], logprob=-0.13530559837818146, top_logprobs=[TopLogprob(token=' bit', bytes=[32, 98, 105, 116], logprob=-0.13530559837818146), TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-2.135305643081665)]), ChatCompletionTokenLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-0.17984026670455933, top_logprobs=[TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-0.17984026670455933), TopLogprob(token=' mind', bytes=[32, 109, 105, 110, 100], logprob=-2.554840326309204)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.00048828122089616954, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.00048828122089616954), TopLogprob(token=' but', bytes=[32, 98, 117, 116], logprob=-7.62548828125)])], refusal=None)

Multiple Request Test
Used the following script to send 4 concurrent requests:

import asyncio
from openai import OpenAI

PROMPTS = [
    "Explain the theory of relativity in simple terms.",
    "Write a poem about the beauty of nature.",
    "How do I make a cup of coffee?",
    "What is the meaning of life?",
]

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

def get_response_sync(prompt: str):
    return client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt}
        ],
        temperature=0.0,
        logprobs=True,
        top_logprobs=2,
        max_tokens=32,
    )

async def main():
    # run blocking requests in threads concurrently
    tasks = [asyncio.to_thread(get_response_sync, prompt) for prompt in PROMPTS]
    results = await asyncio.gather(*tasks)

    for prompt, res in zip(PROMPTS, results):
        print(f"Prompt: {prompt}\nResponse: {res.choices[0].logprobs}\n")

if __name__ == "__main__":
    asyncio.run(main())

The output logprobs for both standard and spec decode LLMs are approximately the same.

Standard decode LLM logprobs:

Prompt: Explain the theory of relativity in simple terms.
Response: ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=-0.0193302184343338, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.0193302184343338), TopLogprob(token='Albert', bytes=[65, 108, 98, 101, 114, 116], logprob=-4.769330024719238)]), ChatCompletionTokenLogprob(token=' theory', bytes=[32, 116, 104, 101, 111, 114, 121], logprob=-0.011061301454901695, top_logprobs=[TopLogprob(token=' theory', bytes=[32, 116, 104, 101, 111, 114, 121], logprob=-0.011061301454901695), TopLogprob(token=' Theory', bytes=[32, 84, 104, 101, 111, 114, 121], logprob=-4.511061191558838)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0), TopLogprob(token=' Of', bytes=[32, 79, 102], logprob=-19.625)]), ChatCompletionTokenLogprob(token=' rel', bytes=[32, 114, 101, 108], logprob=-1.4424220353248529e-05, top_logprobs=[TopLogprob(token=' rel', bytes=[32, 114, 101, 108], logprob=-1.4424220353248529e-05), TopLogprob(token=' Rel', bytes=[32, 82, 101, 108], logprob=-11.250014305114746)]), ChatCompletionTokenLogprob(token='ativity', bytes=[97, 116, 105, 118, 105, 116, 121], logprob=-1.1920928244535389e-07, top_logprobs=[TopLogprob(token='ativity', bytes=[97, 116, 105, 118, 105, 116, 121], logprob=-1.1920928244535389e-07), TopLogprob(token='ativa', bytes=[97, 116, 105, 118, 97], logprob=-17.125)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.43422961235046387, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.43422961235046387), TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-1.0592296123504639)]), ChatCompletionTokenLogprob(token=' developed', bytes=[32, 100, 101, 118, 101, 108, 111, 112, 101, 100], logprob=-0.20736855268478394, top_logprobs=[TopLogprob(token=' developed', bytes=[32, 100, 101, 118, 101, 108, 111, 112, 101, 100], logprob=-0.20736855268478394), TopLogprob(token=' proposed', bytes=[32, 112, 114, 111, 112, 111, 115, 101, 100], logprob=-1.9573686122894287)]), ChatCompletionTokenLogprob(token=' by', bytes=[32, 98, 121], logprob=-5.483612312673358e-06, top_logprobs=[TopLogprob(token=' by', bytes=[32, 98, 121], logprob=-5.483612312673358e-06), TopLogprob(token=' primarily', bytes=[32, 112, 114, 105, 109, 97, 114, 105, 108, 121], logprob=-12.250005722045898)]), ChatCompletionTokenLogprob(token=' Albert', bytes=[32, 65, 108, 98, 101, 114, 116], logprob=-0.00012778419477399439, top_logprobs=[TopLogprob(token=' Albert', bytes=[32, 65, 108, 98, 101, 114, 116], logprob=-0.00012778419477399439), TopLogprob(token='Albert', bytes=[65, 108, 98, 101, 114, 116], logprob=-9.750127792358398)]), ChatCompletionTokenLogprob(token=' Einstein', bytes=[32, 69, 105, 110, 115, 116, 101, 105, 110], logprob=0.0, top_logprobs=[TopLogprob(token=' Einstein', bytes=[32, 69, 105, 110, 115, 116, 101, 105, 110], logprob=0.0), TopLogprob(token=' Ein', bytes=[32, 69, 105, 110], logprob=-17.75)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.0024786246940493584, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.0024786246940493584), TopLogprob(token=' in', bytes=[32, 105, 110], logprob=-6.00247859954834)]), ChatCompletionTokenLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.018199946731328964, top_logprobs=[TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.018199946731328964), TopLogprob(token=' explains', bytes=[32, 101, 120, 112, 108, 97, 105, 110, 115], logprob=-4.643199920654297)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.012180587276816368, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.012180587276816368), TopLogprob(token=' actually', bytes=[32, 97, 99, 116, 117, 97, 108, 108, 121], logprob=-4.512180805206299)]), ChatCompletionTokenLogprob(token=' fundamental', bytes=[32, 102, 117, 110, 100, 97, 109, 101, 110, 116, 97, 108], logprob=-0.16364723443984985, top_logprobs=[TopLogprob(token=' fundamental', bytes=[32, 102, 117, 110, 100, 97, 109, 101, 110, 116, 97, 108], logprob=-0.16364723443984985), TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-2.288647174835205)]), ChatCompletionTokenLogprob(token=' concept', bytes=[32, 99, 111, 110, 99, 101, 112, 116], logprob=-0.0018710264703258872, top_logprobs=[TopLogprob(token=' concept', bytes=[32, 99, 111, 110, 99, 101, 112, 116], logprob=-0.0018710264703258872), TopLogprob(token=' idea', bytes=[32, 105, 100, 101, 97], logprob=-6.376871109008789)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.00018976318824570626, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.00018976318824570626), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-8.625189781188965)]), ChatCompletionTokenLogprob(token=' physics', bytes=[32, 112, 104, 121, 115, 105, 99, 115], logprob=-0.4768531322479248, top_logprobs=[TopLogprob(token=' physics', bytes=[32, 112, 104, 121, 115, 105, 99, 115], logprob=-0.4768531322479248), TopLogprob(token=' modern', bytes=[32, 109, 111, 100, 101, 114, 110], logprob=-0.9768531322479248)]), ChatCompletionTokenLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.054884787648916245, top_logprobs=[TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.054884787648916245), TopLogprob(token='.', bytes=[46], logprob=-2.929884672164917)]), ChatCompletionTokenLogprob(token=' explains', bytes=[32, 101, 120, 112, 108, 97, 105, 110, 115], logprob=-0.47243085503578186, top_logprobs=[TopLogprob(token=' explains', bytes=[32, 101, 120, 112, 108, 97, 105, 110, 115], logprob=-0.47243085503578186), TopLogprob(token=' changed', bytes=[32, 99, 104, 97, 110, 103, 101, 100], logprob=-2.097430944442749)]), ChatCompletionTokenLogprob(token=' how', bytes=[32, 104, 111, 119], logprob=-0.03385080769658089, top_logprobs=[TopLogprob(token=' how', bytes=[32, 104, 111, 119], logprob=-0.03385080769658089), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-3.408850908279419)]), ChatCompletionTokenLogprob(token=' space', bytes=[32, 115, 112, 97, 99, 101], logprob=-0.5190433263778687, top_logprobs=[TopLogprob(token=' space', bytes=[32, 115, 112, 97, 99, 101], logprob=-0.5190433263778687), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-1.1440433263778687)]), ChatCompletionTokenLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.06196795031428337, top_logprobs=[TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.06196795031428337), TopLogprob(token=',', bytes=[44], logprob=-2.8119678497314453)]), ChatCompletionTokenLogprob(token=' time', bytes=[32, 116, 105, 109, 101], logprob=-1.1920928244535389e-07, top_logprobs=[TopLogprob(token=' time', bytes=[32, 116, 105, 109, 101], logprob=-1.1920928244535389e-07), TopLogprob(token='.time', bytes=[46, 116, 105, 109, 101], logprob=-16.75)]), ChatCompletionTokenLogprob(token=' are', bytes=[32, 97, 114, 101], logprob=-0.022042984142899513, top_logprobs=[TopLogprob(token=' are', bytes=[32, 97, 114, 101], logprob=-0.022042984142899513), TopLogprob(token=' work', bytes=[32, 119, 111, 114, 107], logprob=-4.147042751312256)]), ChatCompletionTokenLogprob(token=' connected', bytes=[32, 99, 111, 110, 110, 101, 99, 116, 101, 100], logprob=-0.08926594257354736, top_logprobs=[TopLogprob(token=' connected', bytes=[32, 99, 111, 110, 110, 101, 99, 116, 101, 100], logprob=-0.08926594257354736), TopLogprob(token=' related', bytes=[32, 114, 101, 108, 97, 116, 101, 100], logprob=-2.839265823364258)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=-0.007123666815459728, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=-0.007123666815459728), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-5.3821234703063965)]), ChatCompletionTokenLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.6783666610717773, top_logprobs=[TopLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.6783666610717773), TopLogprob(token=' I', bytes=[32, 73], logprob=-1.1783666610717773)]), ChatCompletionTokenLogprob(token="'s", bytes=[39, 115], logprob=-0.1499624401330948, top_logprobs=[TopLogprob(token="'s", bytes=[39, 115], logprob=-0.1499624401330948), TopLogprob(token=' consists', bytes=[32, 99, 111, 110, 115, 105, 115, 116, 115], logprob=-2.6499624252319336)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.5610803961753845, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.5610803961753845), TopLogprob(token=' divided', bytes=[32, 100, 105, 118, 105, 100, 101, 100], logprob=-1.5610804557800293)]), ChatCompletionTokenLogprob(token=' bit', bytes=[32, 98, 105, 116], logprob=-0.15149393677711487, top_logprobs=[TopLogprob(token=' bit', bytes=[32, 98, 105, 116], logprob=-0.15149393677711487), TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-2.026494026184082)]), ChatCompletionTokenLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-0.17495320737361908, top_logprobs=[TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-0.17495320737361908), TopLogprob(token=' mind', bytes=[32, 109, 105, 110, 100], logprob=-2.5499532222747803)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.00048828122089616954, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.00048828122089616954), TopLogprob(token=' but', bytes=[32, 98, 117, 116], logprob=-7.62548828125)])], refusal=None)

Prompt: Write a poem about the beauty of nature.
Response: ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='In', bytes=[73, 110], logprob=-0.24256405234336853, top_logprobs=[TopLogprob(token='In', bytes=[73, 110], logprob=-0.24256405234336853), TopLogprob(token='Am', bytes=[65, 109], logprob=-2.4925639629364014)]), ChatCompletionTokenLogprob(token=' twilight', bytes=[32, 116, 119, 105, 108, 105, 103, 104, 116], logprob=-0.29958003759384155, top_logprobs=[TopLogprob(token=' twilight', bytes=[32, 116, 119, 105, 108, 105, 103, 104, 116], logprob=-0.29958003759384155), TopLogprob(token=' verd', bytes=[32, 118, 101, 114, 100], logprob=-2.5495800971984863)]), ChatCompletionTokenLogprob(token="'s", bytes=[39, 115], logprob=-0.011126850731670856, top_logprobs=[TopLogprob(token="'s", bytes=[39, 115], logprob=-0.011126850731670856), TopLogprob(token=' skies', bytes=[32, 115, 107, 105, 101, 115], logprob=-5.13612699508667)]), ChatCompletionTokenLogprob(token=' h', bytes=[32, 104], logprob=-0.00367881846614182, top_logprobs=[TopLogprob(token=' h', bytes=[32, 104], logprob=-0.00367881846614182), TopLogprob(token=' hue', bytes=[32, 104, 117, 101], logprob=-6.253678798675537)]), ChatCompletionTokenLogprob(token='ush', bytes=[117, 115, 104], logprob=-1.168244216387393e-05, top_logprobs=[TopLogprob(token='ush', bytes=[117, 115, 104], logprob=-1.168244216387393e-05), TopLogprob(token='ushed', bytes=[117, 115, 104, 101, 100], logprob=-11.500011444091797)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.00011062010162277147, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.00011062010162277147), TopLogprob(token=' where', bytes=[32, 119, 104, 101, 114, 101], logprob=-9.125110626220703)]), ChatCompletionTokenLogprob(token=' where', bytes=[32, 119, 104, 101, 114, 101], logprob=-0.00036090059438720345, top_logprobs=[TopLogprob(token=' where', bytes=[32, 119, 104, 101, 114, 101], logprob=-0.00036090059438720345), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-8.250360488891602)]), ChatCompletionTokenLogprob(token=' shadows', bytes=[32, 115, 104, 97, 100, 111, 119, 115], logprob=-0.16770775616168976, top_logprobs=[TopLogprob(token=' shadows', bytes=[32, 115, 104, 97, 100, 111, 119, 115], logprob=-0.16770775616168976), TopLogprob(token=' sun', bytes=[32, 115, 117, 110], logprob=-2.667707681655884)]), ChatCompletionTokenLogprob(token=' play', bytes=[32, 112, 108, 97, 121], logprob=-0.007661009207367897, top_logprobs=[TopLogprob(token=' play', bytes=[32, 112, 108, 97, 121], logprob=-0.007661009207367897), TopLogprob(token=' dance', bytes=[32, 100, 97, 110, 99, 101], logprob=-5.882660865783691)]), ChatCompletionTokenLogprob(token=',\n', bytes=[44, 10], logprob=-0.00016330339713022113, top_logprobs=[TopLogprob(token=',\n', bytes=[44, 10], logprob=-0.00016330339713022113), TopLogprob(token=',', bytes=[44], logprob=-8.875163078308105)]), ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=-0.422052800655365, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.422052800655365), TopLogprob(token='A', bytes=[65], logprob=-1.4220528602600098)]), ChatCompletionTokenLogprob(token=' forest', bytes=[32, 102, 111, 114, 101, 115, 116], logprob=-0.4898858666419983, top_logprobs=[TopLogprob(token=' forest', bytes=[32, 102, 111, 114, 101, 115, 116], logprob=-0.4898858666419983), TopLogprob(token=' world', bytes=[32, 119, 111, 114, 108, 100], logprob=-1.9898858070373535)]), ChatCompletionTokenLogprob(token=' whispers', bytes=[32, 119, 104, 105, 115, 112, 101, 114, 115], logprob=-0.5475448966026306, top_logprobs=[TopLogprob(token=' whispers', bytes=[32, 119, 104, 105, 115, 112, 101, 114, 115], logprob=-0.5475448966026306), TopLogprob(token=' aw', bytes=[32, 97, 119], logprob=-1.5475449562072754)]), ChatCompletionTokenLogprob(token=' secrets', bytes=[32, 115, 101, 99, 114, 101, 116, 115], logprob=-0.5811936855316162, top_logprobs=[TopLogprob(token=' secrets', bytes=[32, 115, 101, 99, 114, 101, 116, 115], logprob=-0.5811936855316162), TopLogprob(token=',', bytes=[44], logprob=-0.9561936855316162)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.022529365494847298, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.022529365494847298), TopLogprob(token=' all', bytes=[32, 97, 108, 108], logprob=-4.397529602050781)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.001637547742575407, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.001637547742575407), TopLogprob(token=' a', bytes=[32, 97], logprob=-6.6266374588012695)]), ChatCompletionTokenLogprob(token=' day', bytes=[32, 100, 97, 121], logprob=-0.0004532501916401088, top_logprobs=[TopLogprob(token=' day', bytes=[32, 100, 97, 121], logprob=-0.0004532501916401088), TopLogprob(token=' bay', bytes=[32, 98, 97, 121], logprob=-8.250452995300293)]), ChatCompletionTokenLogprob(token=',\n', bytes=[44, 10], logprob=-0.632619321346283, top_logprobs=[TopLogprob(token=',\n', bytes=[44, 10], logprob=-0.632619321346283), TopLogprob(token='.\n', bytes=[46, 10], logprob=-0.757619321346283)]), ChatCompletionTokenLogprob(token='A', bytes=[65], logprob=-0.5877874493598938, top_logprobs=[TopLogprob(token='A', bytes=[65], logprob=-0.5877874493598938), TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.9627874493598938)]), ChatCompletionTokenLogprob(token=' gentle', bytes=[32, 103, 101, 110, 116, 108, 101], logprob=-0.6594477891921997, top_logprobs=[TopLogprob(token=' gentle', bytes=[32, 103, 101, 110, 116, 108, 101], logprob=-0.6594477891921997), TopLogprob(token=' sym', bytes=[32, 115, 121, 109], logprob=-2.15944766998291)]), ChatCompletionTokenLogprob(token=' breeze', bytes=[32, 98, 114, 101, 101, 122, 101], logprob=-0.004482697229832411, top_logprobs=[TopLogprob(token=' breeze', bytes=[32, 98, 114, 101, 101, 122, 101], logprob=-0.004482697229832411), TopLogprob(token=' bro', bytes=[32, 98, 114, 111], logprob=-6.379482746124268)]), ChatCompletionTokenLogprob(token=' st', bytes=[32, 115, 116], logprob=-0.48538821935653687, top_logprobs=[TopLogprob(token=' st', bytes=[32, 115, 116], logprob=-0.48538821935653687), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-1.4853882789611816)]), ChatCompletionTokenLogprob(token='irs', bytes=[105, 114, 115], logprob=-3.8742269680369645e-05, top_logprobs=[TopLogprob(token='irs', bytes=[105, 114, 115], logprob=-3.8742269680369645e-05), TopLogprob(token='rolls', bytes=[114, 111, 108, 108, 115], logprob=-10.250039100646973)]), ChatCompletionTokenLogprob(token=' leaves', bytes=[32, 108, 101, 97, 118, 101, 115], logprob=-0.5631136894226074, top_logprobs=[TopLogprob(token=' leaves', bytes=[32, 108, 101, 97, 118, 101, 115], logprob=-0.5631136894226074), TopLogprob(token=',', bytes=[44], logprob=-0.9381136894226074)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.2595483064651489, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.2595483064651489), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-2.5095481872558594)]), ChatCompletionTokenLogprob(token=' gold', bytes=[32, 103, 111, 108, 100], logprob=-0.32894399762153625, top_logprobs=[TopLogprob(token=' gold', bytes=[32, 103, 111, 108, 100], logprob=-0.32894399762153625), TopLogprob(token=' green', bytes=[32, 103, 114, 101, 101, 110], logprob=-1.5789439678192139)]), ChatCompletionTokenLogprob(token=',\n', bytes=[44, 10], logprob=-0.0036250133998692036, top_logprobs=[TopLogprob(token=',\n', bytes=[44, 10], logprob=-0.0036250133998692036), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-5.62862491607666)]), ChatCompletionTokenLogprob(token='As', bytes=[65, 115], logprob=-0.07874530553817749, top_logprobs=[TopLogprob(token='As', bytes=[65, 115], logprob=-0.07874530553817749), TopLogprob(token='And', bytes=[65, 110, 100], logprob=-2.7037453651428223)]), ChatCompletionTokenLogprob(token=' sun', bytes=[32, 115, 117, 110], logprob=-1.1605862379074097, top_logprobs=[TopLogprob(token=' sun', bytes=[32, 115, 117, 110], logprob=-1.1605862379074097), TopLogprob(token=' nature', bytes=[32, 110, 97, 116, 117, 114, 101], logprob=-1.2855862379074097)]), ChatCompletionTokenLogprob(token='be', bytes=[98, 101], logprob=-0.04718654602766037, top_logprobs=[TopLogprob(token='be', bytes=[98, 101], logprob=-0.04718654602766037), TopLogprob(token='sets', bytes=[115, 101, 116, 115], logprob=-3.5471866130828857)]), ChatCompletionTokenLogprob(token='ams', bytes=[97, 109, 115], logprob=-4.768370445162873e-07, top_logprobs=[TopLogprob(token='ams', bytes=[97, 109, 115], logprob=-4.768370445162873e-07), TopLogprob(token='ats', bytes=[97, 116, 115], logprob=-14.75)]), ChatCompletionTokenLogprob(token=' dance', bytes=[32, 100, 97, 110, 99, 101], logprob=-0.27067670226097107, top_logprobs=[TopLogprob(token=' dance', bytes=[32, 100, 97, 110, 99, 101], logprob=-0.27067670226097107), TopLogprob(token=' filter', bytes=[32, 102, 105, 108, 116, 101, 114], logprob=-1.6456767320632935)])], refusal=None)

Prompt: How do I make a cup of coffee?
Response: ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='Making', bytes=[77, 97, 107, 105, 110, 103], logprob=-0.04564668610692024, top_logprobs=[TopLogprob(token='Making', bytes=[77, 97, 107, 105, 110, 103], logprob=-0.04564668610692024), TopLogprob(token='A', bytes=[65], logprob=-3.4206466674804688)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-2.861018856492592e-06, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-2.861018856492592e-06), TopLogprob(token=' coffee', bytes=[32, 99, 111, 102, 102, 101, 101], logprob=-12.75000286102295)]), ChatCompletionTokenLogprob(token=' cup', bytes=[32, 99, 117, 112], logprob=-0.004693562630563974, top_logprobs=[TopLogprob(token=' cup', bytes=[32, 99, 117, 112], logprob=-0.004693562630563974), TopLogprob(token=' perfect', bytes=[32, 112, 101, 114, 102, 101, 99, 116], logprob=-6.129693508148193)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0), TopLogprob(token=' or', bytes=[32, 111, 114], logprob=-18.375)]), ChatCompletionTokenLogprob(token=' coffee', bytes=[32, 99, 111, 102, 102, 101, 101], logprob=-2.2053474822314456e-05, top_logprobs=[TopLogprob(token=' coffee', bytes=[32, 99, 111, 102, 102, 101, 101], logprob=-2.2053474822314456e-05), TopLogprob(token=' delicious', bytes=[32, 100, 101, 108, 105, 99, 105, 111, 117, 115], logprob=-11.250021934509277)]), ChatCompletionTokenLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.07922647148370743, top_logprobs=[TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.07922647148370743), TopLogprob(token=' can', bytes=[32, 99, 97, 110], logprob=-2.579226493835449)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.08435709029436111, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.08435709029436111), TopLogprob(token=' relatively', bytes=[32, 114, 101, 108, 97, 116, 105, 118, 101, 108, 121], logprob=-2.5843570232391357)]), ChatCompletionTokenLogprob(token=' straightforward', bytes=[32, 115, 116, 114, 97, 105, 103, 104, 116, 102, 111, 114, 119, 97, 114, 100], logprob=-0.48053163290023804, top_logprobs=[TopLogprob(token=' straightforward', bytes=[32, 115, 116, 114, 97, 105, 103, 104, 116, 102, 111, 114, 119, 97, 114, 100], logprob=-0.48053163290023804), TopLogprob(token=' simple', bytes=[32, 115, 105, 109, 112, 108, 101], logprob=-1.3555316925048828)]), ChatCompletionTokenLogprob(token=' process', bytes=[32, 112, 114, 111, 99, 101, 115, 115], logprob=-2.8132995794294402e-05, top_logprobs=[TopLogprob(token=' process', bytes=[32, 112, 114, 111, 99, 101, 115, 115], logprob=-2.8132995794294402e-05), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-10.750027656555176)]), ChatCompletionTokenLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.1243097260594368, top_logprobs=[TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.1243097260594368), TopLogprob(token='.', bytes=[46], logprob=-2.374309778213501)]), ChatCompletionTokenLogprob(token=' requires', bytes=[32, 114, 101, 113, 117, 105, 114, 101, 115], logprob=-0.39766862988471985, top_logprobs=[TopLogprob(token=' requires', bytes=[32, 114, 101, 113, 117, 105, 114, 101, 115], logprob=-0.39766862988471985), TopLogprob(token=' can', bytes=[32, 99, 97, 110], logprob=-1.3976686000823975)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.4394497275352478, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.4394497275352478), TopLogprob(token=' some', bytes=[32, 115, 111, 109, 101], logprob=-1.3144497871398926)]), ChatCompletionTokenLogprob(token=' few', bytes=[32, 102, 101, 119], logprob=-0.0012191252317279577, top_logprobs=[TopLogprob(token=' few', bytes=[32, 102, 101, 119], logprob=-0.0012191252317279577), TopLogprob(token=' coffee', bytes=[32, 99, 111, 102, 102, 101, 101], logprob=-7.001219272613525)]), ChatCompletionTokenLogprob(token=' basic', bytes=[32, 98, 97, 115, 105, 99], logprob=-0.32529085874557495, top_logprobs=[TopLogprob(token=' basic', bytes=[32, 98, 97, 115, 105, 99], logprob=-0.32529085874557495), TopLogprob(token=' simple', bytes=[32, 115, 105, 109, 112, 108, 101], logprob=-1.3252909183502197)]), ChatCompletionTokenLogprob(token=' ingredients', bytes=[32, 105, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-0.5351843237876892, top_logprobs=[TopLogprob(token=' ingredients', bytes=[32, 105, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-0.5351843237876892), TopLogprob(token=' steps', bytes=[32, 115, 116, 101, 112, 115], logprob=-1.660184383392334)]), ChatCompletionTokenLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.00020323118951637298, top_logprobs=[TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.00020323118951637298), TopLogprob(token='.', bytes=[46], logprob=-8.625203132629395)]), ChatCompletionTokenLogprob(token=' equipment', bytes=[32, 101, 113, 117, 105, 112, 109, 101, 110, 116], logprob=-0.7700853943824768, top_logprobs=[TopLogprob(token=' equipment', bytes=[32, 101, 113, 117, 105, 112, 109, 101, 110, 116], logprob=-0.7700853943824768), TopLogprob(token=' tools', bytes=[32, 116, 111, 111, 108, 115], logprob=-1.270085334777832)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=-0.00012718822108581662, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=-0.00012718822108581662), TopLogprob(token='.\n\n', bytes=[46, 10, 10], logprob=-9.375126838684082)]), ChatCompletionTokenLogprob(token=' Here', bytes=[32, 72, 101, 114, 101], logprob=-0.00010990492592100054, top_logprobs=[TopLogprob(token=' Here', bytes=[32, 72, 101, 114, 101], logprob=-0.00010990492592100054), TopLogprob(token=' \n\n', bytes=[32, 10, 10], logprob=-9.250109672546387)]), ChatCompletionTokenLogprob(token="'s", bytes=[39, 115], logprob=-5.686121585313231e-05, top_logprobs=[TopLogprob(token="'s", bytes=[39, 115], logprob=-5.686121585313231e-05), TopLogprob(token=' are', bytes=[32, 97, 114, 101], logprob=-9.875057220458984)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-2.3841855067985307e-07, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-2.3841855067985307e-07), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-16.5)]), ChatCompletionTokenLogprob(token=' step', bytes=[32, 115, 116, 101, 112], logprob=-0.003833447117358446, top_logprobs=[TopLogprob(token=' step', bytes=[32, 115, 116, 101, 112], logprob=-0.003833447117358446), TopLogprob(token=' simple', bytes=[32, 115, 105, 109, 112, 108, 101], logprob=-5.628833293914795)]), ChatCompletionTokenLogprob(token='-by', bytes=[45, 98, 121], logprob=-7.152555099310121e-07, top_logprobs=[TopLogprob(token='-by', bytes=[45, 98, 121], logprob=-7.152555099310121e-07), TopLogprob(token=' by', bytes=[32, 98, 121], logprob=-14.750000953674316)]), ChatCompletionTokenLogprob(token='-step', bytes=[45, 115, 116, 101, 112], logprob=-4.768360213347478e-06, top_logprobs=[TopLogprob(token='-step', bytes=[45, 115, 116, 101, 112], logprob=-4.768360213347478e-06), TopLogprob(token='_step', bytes=[95, 115, 116, 101, 112], logprob=-13.500004768371582)]), ChatCompletionTokenLogprob(token=' guide', bytes=[32, 103, 117, 105, 100, 101], logprob=-3.2186455882765586e-06, top_logprobs=[TopLogprob(token=' guide', bytes=[32, 103, 117, 105, 100, 101], logprob=-3.2186455882765586e-06), TopLogprob(token=' Guide', bytes=[32, 71, 117, 105, 100, 101], logprob=-13.87500286102295)]), ChatCompletionTokenLogprob(token=':\n\n', bytes=[58, 10, 10], logprob=-0.3181779980659485, top_logprobs=[TopLogprob(token=':\n\n', bytes=[58, 10, 10], logprob=-0.3181779980659485), TopLogprob(token=' to', bytes=[32, 116, 111], logprob=-1.3181779384613037)]), ChatCompletionTokenLogprob(token='**', bytes=[42, 42], logprob=-0.22721269726753235, top_logprobs=[TopLogprob(token='**', bytes=[42, 42], logprob=-0.22721269726753235), TopLogprob(token='Ingredients', bytes=[73, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-1.60221266746521)]), ChatCompletionTokenLogprob(token='Ingredients', bytes=[73, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-0.008912426419556141, top_logprobs=[TopLogprob(token='Ingredients', bytes=[73, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-0.008912426419556141), TopLogprob(token='Equipment', bytes=[69, 113, 117, 105, 112, 109, 101, 110, 116], logprob=-5.633912563323975)]), ChatCompletionTokenLogprob(token=':', bytes=[58], logprob=-0.0054407850839197636, top_logprobs=[TopLogprob(token=':', bytes=[58], logprob=-0.0054407850839197636), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-5.255440711975098)]), ChatCompletionTokenLogprob(token='**\n\n', bytes=[42, 42, 10, 10], logprob=-0.00023064337437972426, top_logprobs=[TopLogprob(token='**\n\n', bytes=[42, 42, 10, 10], logprob=-0.00023064337437972426), TopLogprob(token='**\n', bytes=[42, 42, 10], logprob=-8.37523078918457)]), ChatCompletionTokenLogprob(token='*', bytes=[42], logprob=-0.7318389415740967, top_logprobs=[TopLogprob(token='*', bytes=[42], logprob=-0.7318389415740967), TopLogprob(token='1', bytes=[49], logprob=-1.2318389415740967)]), ChatCompletionTokenLogprob(token=' ', bytes=[32], logprob=-0.5708231925964355, top_logprobs=[TopLogprob(token=' ', bytes=[32], logprob=-0.5708231925964355), TopLogprob(token=' Coffee', bytes=[32, 67, 111, 102, 102, 101, 101], logprob=-0.9458231925964355)])], refusal=None)

Prompt: What is the meaning of life?
Response: ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=-0.15441055595874786, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.15441055595874786), TopLogprob(token='One', bytes=[79, 110, 101], logprob=-2.7794106006622314)]), ChatCompletionTokenLogprob(token=' question', bytes=[32, 113, 117, 101, 115, 116, 105, 111, 110], logprob=-0.3961643576622009, top_logprobs=[TopLogprob(token=' question', bytes=[32, 113, 117, 101, 115, 116, 105, 111, 110], logprob=-0.3961643576622009), TopLogprob(token=' meaning', bytes=[32, 109, 101, 97, 110, 105, 110, 103], logprob=-1.2711644172668457)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.005446238908916712, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.005446238908916712), TopLogprob(token=' "', bytes=[32, 34], logprob=-5.380446434020996)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.015167850069701672, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.015167850069701672), TopLogprob(token=' what', bytes=[32, 119, 104, 97, 116], logprob=-4.515167713165283)]), ChatCompletionTokenLogprob(token=' meaning', bytes=[32, 109, 101, 97, 110, 105, 110, 103], logprob=-0.00026794656878337264, top_logprobs=[TopLogprob(token=' meaning', bytes=[32, 109, 101, 97, 110, 105, 110, 103], logprob=-0.00026794656878337264), TopLogprob(token=' "', bytes=[32, 34], logprob=-8.62526798248291)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-19.375)]), ChatCompletionTokenLogprob(token=' life', bytes=[32, 108, 105, 102, 101], logprob=0.0, top_logprobs=[TopLogprob(token=' life', bytes=[32, 108, 105, 102, 101], logprob=0.0), TopLogprob(token=' Life', bytes=[32, 76, 105, 102, 101], logprob=-18.625)]), ChatCompletionTokenLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.2521112859249115, top_logprobs=[TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.2521112859249115), TopLogprob(token=' has', bytes=[32, 104, 97, 115], logprob=-1.5021113157272339)]), ChatCompletionTokenLogprob(token=' one', bytes=[32, 111, 110, 101], logprob=-0.6364416480064392, top_logprobs=[TopLogprob(token=' one', bytes=[32, 111, 110, 101], logprob=-0.6364416480064392), TopLogprob(token=' a', bytes=[32, 97], logprob=-0.7614416480064392)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.03804221749305725, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.03804221749305725), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-3.2880423069000244)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.005728494841605425, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.005728494841605425), TopLogprob(token=' humanity', bytes=[32, 104, 117, 109, 97, 110, 105, 116, 121], logprob=-5.255728721618652)]), ChatCompletionTokenLogprob(token=' most', bytes=[32, 109, 111, 115, 116], logprob=-0.0368226133286953, top_logprobs=[TopLogprob(token=' most', bytes=[32, 109, 111, 115, 116], logprob=-0.0368226133286953), TopLogprob(token=' oldest', bytes=[32, 111, 108, 100, 101, 115, 116], logprob=-3.661822557449341)]), ChatCompletionTokenLogprob(token=' profound', bytes=[32, 112, 114, 111, 102, 111, 117, 110, 100], logprob=-0.03316638246178627, top_logprobs=[TopLogprob(token=' profound', bytes=[32, 112, 114, 111, 102, 111, 117, 110, 100], logprob=-0.03316638246178627), TopLogprob(token=' fundamental', bytes=[32, 102, 117, 110, 100, 97, 109, 101, 110, 116, 97, 108], logprob=-4.283166408538818)]), ChatCompletionTokenLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.004079591948539019, top_logprobs=[TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.004079591948539019), TopLogprob(token=',', bytes=[44], logprob=-5.504079818725586)]), ChatCompletionTokenLogprob(token=' enduring', bytes=[32, 101, 110, 100, 117, 114, 105, 110, 103], logprob=-1.1337802410125732, top_logprobs=[TopLogprob(token=' enduring', bytes=[32, 101, 110, 100, 117, 114, 105, 110, 103], logprob=-1.1337802410125732), TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-1.2587802410125732)]), ChatCompletionTokenLogprob(token=' questions', bytes=[32, 113, 117, 101, 115, 116, 105, 111, 110, 115], logprob=-0.9234352707862854, top_logprobs=[TopLogprob(token=' questions', bytes=[32, 113, 117, 101, 115, 116, 105, 111, 110, 115], logprob=-0.9234352707862854), TopLogprob(token=' mysteries', bytes=[32, 109, 121, 115, 116, 101, 114, 105, 101, 115], logprob=-1.5484352111816406)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.22694909572601318, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.22694909572601318), TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-2.4769492149353027)]), ChatCompletionTokenLogprob(token=' human', bytes=[32, 104, 117, 109, 97, 110], logprob=-0.01479087769985199, top_logprobs=[TopLogprob(token=' human', bytes=[32, 104, 117, 109, 97, 110], logprob=-0.01479087769985199), TopLogprob(token=' philosophy', bytes=[32, 112, 104, 105, 108, 111, 115, 111, 112, 104, 121], logprob=-4.389791011810303)]), ChatCompletionTokenLogprob(token=' history', bytes=[32, 104, 105, 115, 116, 111, 114, 121], logprob=-0.025836575776338577, top_logprobs=[TopLogprob(token=' history', bytes=[32, 104, 105, 115, 116, 111, 114, 121], logprob=-0.025836575776338577), TopLogprob(token=' existence', bytes=[32, 101, 120, 105, 115, 116, 101, 110, 99, 101], logprob=-4.15083646774292)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=-0.2261478751897812, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=-0.2261478751897812), TopLogprob(token=',', bytes=[44], logprob=-1.6011478900909424)]), ChatCompletionTokenLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.2078259289264679, top_logprobs=[TopLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.2078259289264679), TopLogprob(token=' Philosoph', bytes=[32, 80, 104, 105, 108, 111, 115, 111, 112, 104], logprob=-2.3328258991241455)]), ChatCompletionTokenLogprob(token=' has', bytes=[32, 104, 97, 115], logprob=-0.23564612865447998, top_logprobs=[TopLogprob(token=' has', bytes=[32, 104, 97, 115], logprob=-0.23564612865447998), TopLogprob(token="'s", bytes=[39, 115], logprob=-1.86064612865448)]), ChatCompletionTokenLogprob(token=' been', bytes=[32, 98, 101, 101, 110], logprob=-0.09237802028656006, top_logprobs=[TopLogprob(token=' been', bytes=[32, 98, 101, 101, 110], logprob=-0.09237802028656006), TopLogprob(token=' puzzled', bytes=[32, 112, 117, 122, 122, 108, 101, 100], logprob=-3.0923781394958496)]), ChatCompletionTokenLogprob(token=' debated', bytes=[32, 100, 101, 98, 97, 116, 101, 100], logprob=-0.1647709757089615, top_logprobs=[TopLogprob(token=' debated', bytes=[32, 100, 101, 98, 97, 116, 101, 100], logprob=-0.1647709757089615), TopLogprob(token=' explored', bytes=[32, 101, 120, 112, 108, 111, 114, 101, 100], logprob=-2.03977108001709)]), ChatCompletionTokenLogprob(token=' by', bytes=[32, 98, 121], logprob=-0.5897256135940552, top_logprobs=[TopLogprob(token=' by', bytes=[32, 98, 121], logprob=-0.5897256135940552), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-1.0897256135940552)]), ChatCompletionTokenLogprob(token=' philosophers', bytes=[32, 112, 104, 105, 108, 111, 115, 111, 112, 104, 101, 114, 115], logprob=-0.000722266617231071, top_logprobs=[TopLogprob(token=' philosophers', bytes=[32, 112, 104, 105, 108, 111, 115, 111, 112, 104, 101, 114, 115], logprob=-0.000722266617231071), TopLogprob(token=' scholars', bytes=[32, 115, 99, 104, 111, 108, 97, 114, 115], logprob=-7.875722408294678)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=0.0, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=0.0), TopLogprob(token='.', bytes=[46], logprob=-18.625)]), ChatCompletionTokenLogprob(token=' theolog', bytes=[32, 116, 104, 101, 111, 108, 111, 103], logprob=-0.032106488943099976, top_logprobs=[TopLogprob(token=' theolog', bytes=[32, 116, 104, 101, 111, 108, 111, 103], logprob=-0.032106488943099976), TopLogprob(token=' scientists', bytes=[32, 115, 99, 105, 101, 110, 116, 105, 115, 116, 115], logprob=-3.657106399536133)]), ChatCompletionTokenLogprob(token='ians', bytes=[105, 97, 110, 115], logprob=-2.753696753643453e-05, top_logprobs=[TopLogprob(token='ians', bytes=[105, 97, 110, 115], logprob=-2.753696753643453e-05), TopLogprob(token='icans', bytes=[105, 99, 97, 110, 115], logprob=-11.000027656555176)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=0.0, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=0.0), TopLogprob(token='.', bytes=[46], logprob=-20.375)]), ChatCompletionTokenLogprob(token=' scientists', bytes=[32, 115, 99, 105, 101, 110, 116, 105, 115, 116, 115], logprob=-0.0012894895626232028, top_logprobs=[TopLogprob(token=' scientists', bytes=[32, 115, 99, 105, 101, 110, 116, 105, 115, 116, 115], logprob=-0.0012894895626232028), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-7.001289367675781)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-1.1920928244535389e-07, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-1.1920928244535389e-07), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-15.75)])], refusal=None)

Spec decode LLM logprobs:

Prompt: Explain the theory of relativity in simple terms.
ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=-0.0193302184343338, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.0193302184343338), TopLogprob(token='Albert', bytes=[65, 108, 98, 101, 114, 116], logprob=-4.769330024719238)]), ChatCompletionTokenLogprob(token=' theory', bytes=[32, 116, 104, 101, 111, 114, 121], logprob=-0.011061301454901695, top_logprobs=[TopLogprob(token=' theory', bytes=[32, 116, 104, 101, 111, 114, 121], logprob=-0.011061301454901695), TopLogprob(token=' Theory', bytes=[32, 84, 104, 101, 111, 114, 121], logprob=-4.511061191558838)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0), TopLogprob(token=' Of', bytes=[32, 79, 102], logprob=-19.625)]), ChatCompletionTokenLogprob(token=' rel', bytes=[32, 114, 101, 108], logprob=-1.4424220353248529e-05, top_logprobs=[TopLogprob(token=' rel', bytes=[32, 114, 101, 108], logprob=-1.4424220353248529e-05), TopLogprob(token=' Rel', bytes=[32, 82, 101, 108], logprob=-11.250014305114746)]), ChatCompletionTokenLogprob(token='ativity', bytes=[97, 116, 105, 118, 105, 116, 121], logprob=-1.1920928244535389e-07, top_logprobs=[TopLogprob(token='ativity', bytes=[97, 116, 105, 118, 105, 116, 121], logprob=-1.1920928244535389e-07), TopLogprob(token='ativa', bytes=[97, 116, 105, 118, 97], logprob=-17.0)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.4342271387577057, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.4342271387577057), TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-1.0592271089553833)]), ChatCompletionTokenLogprob(token=' developed', bytes=[32, 100, 101, 118, 101, 108, 111, 112, 101, 100], logprob=-0.20746465027332306, top_logprobs=[TopLogprob(token=' developed', bytes=[32, 100, 101, 118, 101, 108, 111, 112, 101, 100], logprob=-0.20746465027332306), TopLogprob(token=' proposed', bytes=[32, 112, 114, 111, 112, 111, 115, 101, 100], logprob=-1.9574646949768066)]), ChatCompletionTokenLogprob(token=' by', bytes=[32, 98, 121], logprob=-5.602820692729438e-06, top_logprobs=[TopLogprob(token=' by', bytes=[32, 98, 121], logprob=-5.602820692729438e-06), TopLogprob(token=' primarily', bytes=[32, 112, 114, 105, 109, 97, 114, 105, 108, 121], logprob=-12.250005722045898)]), ChatCompletionTokenLogprob(token=' Albert', bytes=[32, 65, 108, 98, 101, 114, 116], logprob=-0.0001294529065489769, top_logprobs=[TopLogprob(token=' Albert', bytes=[32, 65, 108, 98, 101, 114, 116], logprob=-0.0001294529065489769), TopLogprob(token='Albert', bytes=[65, 108, 98, 101, 114, 116], logprob=-9.750129699707031)]), ChatCompletionTokenLogprob(token=' Einstein', bytes=[32, 69, 105, 110, 115, 116, 101, 105, 110], logprob=0.0, top_logprobs=[TopLogprob(token=' Einstein', bytes=[32, 69, 105, 110, 115, 116, 101, 105, 110], logprob=0.0), TopLogprob(token=' Ein', bytes=[32, 69, 105, 110], logprob=-17.5)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.002478386741131544, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.002478386741131544), TopLogprob(token=' in', bytes=[32, 105, 110], logprob=-6.00247859954834)]), ChatCompletionTokenLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.01896451972424984, top_logprobs=[TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.01896451972424984), TopLogprob(token=' explains', bytes=[32, 101, 120, 112, 108, 97, 105, 110, 115], logprob=-4.6439642906188965)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.013689680024981499, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.013689680024981499), TopLogprob(token=' actually', bytes=[32, 97, 99, 116, 117, 97, 108, 108, 121], logprob=-4.3886895179748535)]), ChatCompletionTokenLogprob(token=' fundamental', bytes=[32, 102, 117, 110, 100, 97, 109, 101, 110, 116, 97, 108], logprob=-0.1648777425289154, top_logprobs=[TopLogprob(token=' fundamental', bytes=[32, 102, 117, 110, 100, 97, 109, 101, 110, 116, 97, 108], logprob=-0.1648777425289154), TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-2.2898776531219482)]), ChatCompletionTokenLogprob(token=' concept', bytes=[32, 99, 111, 110, 99, 101, 112, 116], logprob=-0.0018755479250103235, top_logprobs=[TopLogprob(token=' concept', bytes=[32, 99, 111, 110, 99, 101, 112, 116], logprob=-0.0018755479250103235), TopLogprob(token=' idea', bytes=[32, 105, 100, 101, 97], logprob=-6.376875400543213)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.00018976318824570626, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.00018976318824570626), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-8.625189781188965)]), ChatCompletionTokenLogprob(token=' physics', bytes=[32, 112, 104, 121, 115, 105, 99, 115], logprob=-0.4768526256084442, top_logprobs=[TopLogprob(token=' physics', bytes=[32, 112, 104, 121, 115, 105, 99, 115], logprob=-0.4768526256084442), TopLogprob(token=' modern', bytes=[32, 109, 111, 100, 101, 114, 110], logprob=-0.9768526554107666)]), ChatCompletionTokenLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.054884787648916245, top_logprobs=[TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.054884787648916245), TopLogprob(token='.', bytes=[46], logprob=-2.929884672164917)]), ChatCompletionTokenLogprob(token=' explains', bytes=[32, 101, 120, 112, 108, 97, 105, 110, 115], logprob=-0.471164345741272, top_logprobs=[TopLogprob(token=' explains', bytes=[32, 101, 120, 112, 108, 97, 105, 110, 115], logprob=-0.471164345741272), TopLogprob(token=' changed', bytes=[32, 99, 104, 97, 110, 103, 101, 100], logprob=-2.0961642265319824)]), ChatCompletionTokenLogprob(token=' how', bytes=[32, 104, 111, 119], logprob=-0.02997696027159691, top_logprobs=[TopLogprob(token=' how', bytes=[32, 104, 111, 119], logprob=-0.02997696027159691), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-3.5299768447875977)]), ChatCompletionTokenLogprob(token=' space', bytes=[32, 115, 112, 97, 99, 101], logprob=-0.5190426111221313, top_logprobs=[TopLogprob(token=' space', bytes=[32, 115, 112, 97, 99, 101], logprob=-0.5190426111221313), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-1.1440426111221313)]), ChatCompletionTokenLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.04858766868710518, top_logprobs=[TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.04858766868710518), TopLogprob(token=',', bytes=[44], logprob=-3.0485875606536865)]), ChatCompletionTokenLogprob(token=' time', bytes=[32, 116, 105, 109, 101], logprob=-1.1920928244535389e-07, top_logprobs=[TopLogprob(token=' time', bytes=[32, 116, 105, 109, 101], logprob=-1.1920928244535389e-07), TopLogprob(token='.time', bytes=[46, 116, 105, 109, 101], logprob=-16.75)]), ChatCompletionTokenLogprob(token=' are', bytes=[32, 97, 114, 101], logprob=-0.019475897774100304, top_logprobs=[TopLogprob(token=' are', bytes=[32, 97, 114, 101], logprob=-0.019475897774100304), TopLogprob(token=' work', bytes=[32, 119, 111, 114, 107], logprob=-4.269475936889648)]), ChatCompletionTokenLogprob(token=' connected', bytes=[32, 99, 111, 110, 110, 101, 99, 116, 101, 100], logprob=-0.08927161246538162, top_logprobs=[TopLogprob(token=' connected', bytes=[32, 99, 111, 110, 110, 101, 99, 116, 101, 100], logprob=-0.08927161246538162), TopLogprob(token=' related', bytes=[32, 114, 101, 108, 97, 116, 101, 100], logprob=-2.8392715454101562)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=-0.0074507687240839005, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=-0.0074507687240839005), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-5.382450580596924)]), ChatCompletionTokenLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.7004333734512329, top_logprobs=[TopLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.7004333734512329), TopLogprob(token=' I', bytes=[32, 73], logprob=-1.200433373451233)]), ChatCompletionTokenLogprob(token="'s", bytes=[39, 115], logprob=-0.15943790972232819, top_logprobs=[TopLogprob(token="'s", bytes=[39, 115], logprob=-0.15943790972232819), TopLogprob(token=' consists', bytes=[32, 99, 111, 110, 115, 105, 115, 116, 115], logprob=-2.534437894821167)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.561939001083374, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.561939001083374), TopLogprob(token=' divided', bytes=[32, 100, 105, 118, 105, 100, 101, 100], logprob=-1.561939001083374)]), ChatCompletionTokenLogprob(token=' bit', bytes=[32, 98, 105, 116], logprob=-0.13531862199306488, top_logprobs=[TopLogprob(token=' bit', bytes=[32, 98, 105, 116], logprob=-0.13531862199306488), TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-2.1353185176849365)]), ChatCompletionTokenLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-0.17495441436767578, top_logprobs=[TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-0.17495441436767578), TopLogprob(token=' mind', bytes=[32, 109, 105, 110, 100], logprob=-2.549954414367676)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.00048828122089616954, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.00048828122089616954), TopLogprob(token=' but', bytes=[32, 98, 117, 116], logprob=-7.62548828125)])], refusal=None)

Prompt: Write a poem about the beauty of nature.
ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='In', bytes=[73, 110], logprob=-0.24256405234336853, top_logprobs=[TopLogprob(token='In', bytes=[73, 110], logprob=-0.24256405234336853), TopLogprob(token='Am', bytes=[65, 109], logprob=-2.4925639629364014)]), ChatCompletionTokenLogprob(token=' twilight', bytes=[32, 116, 119, 105, 108, 105, 103, 104, 116], logprob=-0.27036383748054504, top_logprobs=[TopLogprob(token=' twilight', bytes=[32, 116, 119, 105, 108, 105, 103, 104, 116], logprob=-0.27036383748054504), TopLogprob(token=' verd', bytes=[32, 118, 101, 114, 100], logprob=-2.6453638076782227)]), ChatCompletionTokenLogprob(token="'s", bytes=[39, 115], logprob=-0.010841161012649536, top_logprobs=[TopLogprob(token="'s", bytes=[39, 115], logprob=-0.010841161012649536), TopLogprob(token=' skies', bytes=[32, 115, 107, 105, 101, 115], logprob=-5.135841369628906)]), ChatCompletionTokenLogprob(token=' h', bytes=[32, 104], logprob=-0.0036816690117120743, top_logprobs=[TopLogprob(token=' h', bytes=[32, 104], logprob=-0.0036816690117120743), TopLogprob(token=' hue', bytes=[32, 104, 117, 101], logprob=-6.253681659698486)]), ChatCompletionTokenLogprob(token='ush', bytes=[117, 115, 104], logprob=-1.3112935448589269e-05, top_logprobs=[TopLogprob(token='ush', bytes=[117, 115, 104], logprob=-1.3112935448589269e-05), TopLogprob(token='ushed', bytes=[117, 115, 104, 101, 100], logprob=-11.37501335144043)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-0.00011062010162277147, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-0.00011062010162277147), TopLogprob(token=' where', bytes=[32, 119, 104, 101, 114, 101], logprob=-9.125110626220703)]), ChatCompletionTokenLogprob(token=' where', bytes=[32, 119, 104, 101, 114, 101], logprob=-0.0003351603518240154, top_logprobs=[TopLogprob(token=' where', bytes=[32, 119, 104, 101, 114, 101], logprob=-0.0003351603518240154), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-8.375334739685059)]), ChatCompletionTokenLogprob(token=' shadows', bytes=[32, 115, 104, 97, 100, 111, 119, 115], logprob=-0.17908340692520142, top_logprobs=[TopLogprob(token=' shadows', bytes=[32, 115, 104, 97, 100, 111, 119, 115], logprob=-0.17908340692520142), TopLogprob(token=' sun', bytes=[32, 115, 117, 110], logprob=-2.5540833473205566)]), ChatCompletionTokenLogprob(token=' play', bytes=[32, 112, 108, 97, 121], logprob=-0.007994676008820534, top_logprobs=[TopLogprob(token=' play', bytes=[32, 112, 108, 97, 121], logprob=-0.007994676008820534), TopLogprob(token=' dance', bytes=[32, 100, 97, 110, 99, 101], logprob=-5.757994651794434)]), ChatCompletionTokenLogprob(token=',\n', bytes=[44, 10], logprob=-0.00014673586701974273, top_logprobs=[TopLogprob(token=',\n', bytes=[44, 10], logprob=-0.00014673586701974273), TopLogprob(token=',', bytes=[44], logprob=-9.000146865844727)]), ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=-0.42204034328460693, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.42204034328460693), TopLogprob(token='A', bytes=[65], logprob=-1.422040343284607)]), ChatCompletionTokenLogprob(token=' forest', bytes=[32, 102, 111, 114, 101, 115, 116], logprob=-0.4948474168777466, top_logprobs=[TopLogprob(token=' forest', bytes=[32, 102, 111, 114, 101, 115, 116], logprob=-0.4948474168777466), TopLogprob(token=' world', bytes=[32, 119, 111, 114, 108, 100], logprob=-1.9948474168777466)]), ChatCompletionTokenLogprob(token=' whispers', bytes=[32, 119, 104, 105, 115, 112, 101, 114, 115], logprob=-0.5340423583984375, top_logprobs=[TopLogprob(token=' whispers', bytes=[32, 119, 104, 105, 115, 112, 101, 114, 115], logprob=-0.5340423583984375), TopLogprob(token=' aw', bytes=[32, 97, 119], logprob=-1.5340423583984375)]), ChatCompletionTokenLogprob(token=' secrets', bytes=[32, 115, 101, 99, 114, 101, 116, 115], logprob=-0.579210638999939, top_logprobs=[TopLogprob(token=' secrets', bytes=[32, 115, 101, 99, 114, 101, 116, 115], logprob=-0.579210638999939), TopLogprob(token=',', bytes=[44], logprob=-0.954210638999939)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.022009750828146935, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.022009750828146935), TopLogprob(token=' all', bytes=[32, 97, 108, 108], logprob=-4.39700984954834)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.0014448452275246382, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.0014448452275246382), TopLogprob(token=' a', bytes=[32, 97], logprob=-6.7514448165893555)]), ChatCompletionTokenLogprob(token=' day', bytes=[32, 100, 97, 121], logprob=-0.0004968600696884096, top_logprobs=[TopLogprob(token=' day', bytes=[32, 100, 97, 121], logprob=-0.0004968600696884096), TopLogprob(token=' bay', bytes=[32, 98, 97, 121], logprob=-8.125496864318848)]), ChatCompletionTokenLogprob(token=',\n', bytes=[44, 10], logprob=-0.632619321346283, top_logprobs=[TopLogprob(token=',\n', bytes=[44, 10], logprob=-0.632619321346283), TopLogprob(token='.\n', bytes=[46, 10], logprob=-0.757619321346283)]), ChatCompletionTokenLogprob(token='A', bytes=[65], logprob=-0.5878995656967163, top_logprobs=[TopLogprob(token='A', bytes=[65], logprob=-0.5878995656967163), TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.9628995656967163)]), ChatCompletionTokenLogprob(token=' gentle', bytes=[32, 103, 101, 110, 116, 108, 101], logprob=-0.6560173630714417, top_logprobs=[TopLogprob(token=' gentle', bytes=[32, 103, 101, 110, 116, 108, 101], logprob=-0.6560173630714417), TopLogprob(token=' sym', bytes=[32, 115, 121, 109], logprob=-2.156017303466797)]), ChatCompletionTokenLogprob(token=' breeze', bytes=[32, 98, 114, 101, 101, 122, 101], logprob=-0.004330305848270655, top_logprobs=[TopLogprob(token=' breeze', bytes=[32, 98, 114, 101, 101, 122, 101], logprob=-0.004330305848270655), TopLogprob(token=' bro', bytes=[32, 98, 114, 111], logprob=-6.379330158233643)]), ChatCompletionTokenLogprob(token=' st', bytes=[32, 115, 116], logprob=-0.4852747917175293, top_logprobs=[TopLogprob(token=' st', bytes=[32, 115, 116], logprob=-0.4852747917175293), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-1.4852747917175293)]), ChatCompletionTokenLogprob(token='irs', bytes=[105, 114, 115], logprob=-3.40932747349143e-05, top_logprobs=[TopLogprob(token='irs', bytes=[105, 114, 115], logprob=-3.40932747349143e-05), TopLogprob(token='rolls', bytes=[114, 111, 108, 108, 115], logprob=-10.37503433227539)]), ChatCompletionTokenLogprob(token=' leaves', bytes=[32, 108, 101, 97, 118, 101, 115], logprob=-0.5635104179382324, top_logprobs=[TopLogprob(token=' leaves', bytes=[32, 108, 101, 97, 118, 101, 115], logprob=-0.5635104179382324), TopLogprob(token=',', bytes=[44], logprob=-0.9385104179382324)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.26920822262763977, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.26920822262763977), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-2.5192081928253174)]), ChatCompletionTokenLogprob(token=' gold', bytes=[32, 103, 111, 108, 100], logprob=-0.3095017671585083, top_logprobs=[TopLogprob(token=' gold', bytes=[32, 103, 111, 108, 100], logprob=-0.3095017671585083), TopLogprob(token=' green', bytes=[32, 103, 114, 101, 101, 110], logprob=-1.6845017671585083)]), ChatCompletionTokenLogprob(token=',\n', bytes=[44, 10], logprob=-0.004105948377400637, top_logprobs=[TopLogprob(token=',\n', bytes=[44, 10], logprob=-0.004105948377400637), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-5.504106044769287)]), ChatCompletionTokenLogprob(token='As', bytes=[65, 115], logprob=-0.08875326812267303, top_logprobs=[TopLogprob(token='As', bytes=[65, 115], logprob=-0.08875326812267303), TopLogprob(token='And', bytes=[65, 110, 100], logprob=-2.5887532234191895)]), ChatCompletionTokenLogprob(token=' sun', bytes=[32, 115, 117, 110], logprob=-1.157889485359192, top_logprobs=[TopLogprob(token=' sun', bytes=[32, 115, 117, 110], logprob=-1.157889485359192), TopLogprob(token=' nature', bytes=[32, 110, 97, 116, 117, 114, 101], logprob=-1.282889485359192)]), ChatCompletionTokenLogprob(token='be', bytes=[98, 101], logprob=-0.046694494783878326, top_logprobs=[TopLogprob(token='be', bytes=[98, 101], logprob=-0.046694494783878326), TopLogprob(token='sets', bytes=[115, 101, 116, 115], logprob=-3.54669451713562)]), ChatCompletionTokenLogprob(token='ams', bytes=[97, 109, 115], logprob=-5.960462772236497e-07, top_logprobs=[TopLogprob(token='ams', bytes=[97, 109, 115], logprob=-5.960462772236497e-07), TopLogprob(token='ats', bytes=[97, 116, 115], logprob=-14.500000953674316)]), ChatCompletionTokenLogprob(token=' dance', bytes=[32, 100, 97, 110, 99, 101], logprob=-0.26734375953674316, top_logprobs=[TopLogprob(token=' dance', bytes=[32, 100, 97, 110, 99, 101], logprob=-0.26734375953674316), TopLogprob(token=' filter', bytes=[32, 102, 105, 108, 116, 101, 114], logprob=-1.6423437595367432)])], refusal=None)

Prompt: How do I make a cup of coffee?
ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='Making', bytes=[77, 97, 107, 105, 110, 103], logprob=-0.04564668610692024, top_logprobs=[TopLogprob(token='Making', bytes=[77, 97, 107, 105, 110, 103], logprob=-0.04564668610692024), TopLogprob(token='A', bytes=[65], logprob=-3.4206466674804688)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-2.861018856492592e-06, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-2.861018856492592e-06), TopLogprob(token=' coffee', bytes=[32, 99, 111, 102, 102, 101, 101], logprob=-12.75000286102295)]), ChatCompletionTokenLogprob(token=' cup', bytes=[32, 99, 117, 112], logprob=-0.004596738610416651, top_logprobs=[TopLogprob(token=' cup', bytes=[32, 99, 117, 112], logprob=-0.004596738610416651), TopLogprob(token=' perfect', bytes=[32, 112, 101, 114, 102, 101, 99, 116], logprob=-6.129596710205078)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0), TopLogprob(token=' or', bytes=[32, 111, 114], logprob=-18.25)]), ChatCompletionTokenLogprob(token=' coffee', bytes=[32, 99, 111, 102, 102, 101, 101], logprob=-2.2053474822314456e-05, top_logprobs=[TopLogprob(token=' coffee', bytes=[32, 99, 111, 102, 102, 101, 101], logprob=-2.2053474822314456e-05), TopLogprob(token=' delicious', bytes=[32, 100, 101, 108, 105, 99, 105, 111, 117, 115], logprob=-11.250021934509277)]), ChatCompletionTokenLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.07922118157148361, top_logprobs=[TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.07922118157148361), TopLogprob(token=' can', bytes=[32, 99, 97, 110], logprob=-2.579221248626709)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.0950702354311943, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.0950702354311943), TopLogprob(token=' relatively', bytes=[32, 114, 101, 108, 97, 116, 105, 118, 101, 108, 121], logprob=-2.4700701236724854)]), ChatCompletionTokenLogprob(token=' straightforward', bytes=[32, 115, 116, 114, 97, 105, 103, 104, 116, 102, 111, 114, 119, 97, 114, 100], logprob=-0.4346487522125244, top_logprobs=[TopLogprob(token=' straightforward', bytes=[32, 115, 116, 114, 97, 105, 103, 104, 116, 102, 111, 114, 119, 97, 114, 100], logprob=-0.4346487522125244), TopLogprob(token=' simple', bytes=[32, 115, 105, 109, 112, 108, 101], logprob=-1.4346487522125244)]), ChatCompletionTokenLogprob(token=' process', bytes=[32, 112, 114, 111, 99, 101, 115, 115], logprob=-2.825220326485578e-05, top_logprobs=[TopLogprob(token=' process', bytes=[32, 112, 114, 111, 99, 101, 115, 115], logprob=-2.825220326485578e-05), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-10.750028610229492)]), ChatCompletionTokenLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.13974471390247345, top_logprobs=[TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-0.13974471390247345), TopLogprob(token='.', bytes=[46], logprob=-2.264744758605957)]), ChatCompletionTokenLogprob(token=' requires', bytes=[32, 114, 101, 113, 117, 105, 114, 101, 115], logprob=-0.39767399430274963, top_logprobs=[TopLogprob(token=' requires', bytes=[32, 114, 101, 113, 117, 105, 114, 101, 115], logprob=-0.39767399430274963), TopLogprob(token=' can', bytes=[32, 99, 97, 110], logprob=-1.3976739645004272)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-0.44953301548957825, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-0.44953301548957825), TopLogprob(token=' some', bytes=[32, 115, 111, 109, 101], logprob=-1.3245329856872559)]), ChatCompletionTokenLogprob(token=' few', bytes=[32, 102, 101, 119], logprob=-0.0012240068754181266, top_logprobs=[TopLogprob(token=' few', bytes=[32, 102, 101, 119], logprob=-0.0012240068754181266), TopLogprob(token=' coffee', bytes=[32, 99, 111, 102, 102, 101, 101], logprob=-7.001224040985107)]), ChatCompletionTokenLogprob(token=' basic', bytes=[32, 98, 97, 115, 105, 99], logprob=-0.3247232437133789, top_logprobs=[TopLogprob(token=' basic', bytes=[32, 98, 97, 115, 105, 99], logprob=-0.3247232437133789), TopLogprob(token=' simple', bytes=[32, 115, 105, 109, 112, 108, 101], logprob=-1.324723243713379)]), ChatCompletionTokenLogprob(token=' ingredients', bytes=[32, 105, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-0.5112225413322449, top_logprobs=[TopLogprob(token=' ingredients', bytes=[32, 105, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-0.5112225413322449), TopLogprob(token=' steps', bytes=[32, 115, 116, 101, 112, 115], logprob=-1.7612226009368896)]), ChatCompletionTokenLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.0002019201492657885, top_logprobs=[TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.0002019201492657885), TopLogprob(token='.', bytes=[46], logprob=-8.625202178955078)]), ChatCompletionTokenLogprob(token=' equipment', bytes=[32, 101, 113, 117, 105, 112, 109, 101, 110, 116], logprob=-0.7735154628753662, top_logprobs=[TopLogprob(token=' equipment', bytes=[32, 101, 113, 117, 105, 112, 109, 101, 110, 116], logprob=-0.7735154628753662), TopLogprob(token=' tools', bytes=[32, 116, 111, 111, 108, 115], logprob=-1.2735154628753662)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=-0.0001264730526600033, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=-0.0001264730526600033), TopLogprob(token='.\n\n', bytes=[46, 10, 10], logprob=-9.375126838684082)]), ChatCompletionTokenLogprob(token=' Here', bytes=[32, 72, 101, 114, 101], logprob=-0.00011073929636040702, top_logprobs=[TopLogprob(token=' Here', bytes=[32, 72, 101, 114, 101], logprob=-0.00011073929636040702), TopLogprob(token=' \n\n', bytes=[32, 10, 10], logprob=-9.250110626220703)]), ChatCompletionTokenLogprob(token="'s", bytes=[39, 115], logprob=-6.4490144723095e-05, top_logprobs=[TopLogprob(token="'s", bytes=[39, 115], logprob=-6.4490144723095e-05), TopLogprob(token=' are', bytes=[32, 97, 114, 101], logprob=-9.750064849853516)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=-2.3841855067985307e-07, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=-2.3841855067985307e-07), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-16.5)]), ChatCompletionTokenLogprob(token=' step', bytes=[32, 115, 116, 101, 112], logprob=-0.003838315838947892, top_logprobs=[TopLogprob(token=' step', bytes=[32, 115, 116, 101, 112], logprob=-0.003838315838947892), TopLogprob(token=' simple', bytes=[32, 115, 105, 109, 112, 108, 101], logprob=-5.628838539123535)]), ChatCompletionTokenLogprob(token='-by', bytes=[45, 98, 121], logprob=-7.152555099310121e-07, top_logprobs=[TopLogprob(token='-by', bytes=[45, 98, 121], logprob=-7.152555099310121e-07), TopLogprob(token=' by', bytes=[32, 98, 121], logprob=-14.750000953674316)]), ChatCompletionTokenLogprob(token='-step', bytes=[45, 115, 116, 101, 112], logprob=-4.768360213347478e-06, top_logprobs=[TopLogprob(token='-step', bytes=[45, 115, 116, 101, 112], logprob=-4.768360213347478e-06), TopLogprob(token='_step', bytes=[95, 115, 116, 101, 112], logprob=-13.500004768371582)]), ChatCompletionTokenLogprob(token=' guide', bytes=[32, 103, 117, 105, 100, 101], logprob=-4.0531076592742465e-06, top_logprobs=[TopLogprob(token=' guide', bytes=[32, 103, 117, 105, 100, 101], logprob=-4.0531076592742465e-06), TopLogprob(token=' Guide', bytes=[32, 71, 117, 105, 100, 101], logprob=-13.625003814697266)]), ChatCompletionTokenLogprob(token=':\n\n', bytes=[58, 10, 10], logprob=-0.39204564690589905, top_logprobs=[TopLogprob(token=':\n\n', bytes=[58, 10, 10], logprob=-0.39204564690589905), TopLogprob(token=' to', bytes=[32, 116, 111], logprob=-1.1420456171035767)]), ChatCompletionTokenLogprob(token='**', bytes=[42, 42], logprob=-0.227212592959404, top_logprobs=[TopLogprob(token='**', bytes=[42, 42], logprob=-0.227212592959404), TopLogprob(token='Ingredients', bytes=[73, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-1.6022125482559204)]), ChatCompletionTokenLogprob(token='Ingredients', bytes=[73, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-0.008764252997934818, top_logprobs=[TopLogprob(token='Ingredients', bytes=[73, 110, 103, 114, 101, 100, 105, 101, 110, 116, 115], logprob=-0.008764252997934818), TopLogprob(token='Equipment', bytes=[69, 113, 117, 105, 112, 109, 101, 110, 116], logprob=-5.633764266967773)]), ChatCompletionTokenLogprob(token=':', bytes=[58], logprob=-0.004828459583222866, top_logprobs=[TopLogprob(token=':', bytes=[58], logprob=-0.004828459583222866), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-5.379828453063965)]), ChatCompletionTokenLogprob(token='**\n\n', bytes=[42, 42, 10, 10], logprob=-0.00023064337437972426, top_logprobs=[TopLogprob(token='**\n\n', bytes=[42, 42, 10, 10], logprob=-0.00023064337437972426), TopLogprob(token='**\n', bytes=[42, 42, 10], logprob=-8.37523078918457)]), ChatCompletionTokenLogprob(token='*', bytes=[42], logprob=-0.7318389415740967, top_logprobs=[TopLogprob(token='*', bytes=[42], logprob=-0.7318389415740967), TopLogprob(token='1', bytes=[49], logprob=-1.2318389415740967)]), ChatCompletionTokenLogprob(token=' ', bytes=[32], logprob=-0.6322307586669922, top_logprobs=[TopLogprob(token=' ', bytes=[32], logprob=-0.6322307586669922), TopLogprob(token=' Coffee', bytes=[32, 67, 111, 102, 102, 101, 101], logprob=-0.8822307586669922)])], refusal=None)

Prompt: What is the meaning of life?
ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=-0.15441055595874786, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.15441055595874786), TopLogprob(token='One', bytes=[79, 110, 101], logprob=-2.7794106006622314)]), ChatCompletionTokenLogprob(token=' question', bytes=[32, 113, 117, 101, 115, 116, 105, 111, 110], logprob=-0.3963603675365448, top_logprobs=[TopLogprob(token=' question', bytes=[32, 113, 117, 101, 115, 116, 105, 111, 110], logprob=-0.3963603675365448), TopLogprob(token=' meaning', bytes=[32, 109, 101, 97, 110, 105, 110, 103], logprob=-1.2713603973388672)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.005446120630949736, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.005446120630949736), TopLogprob(token=' "', bytes=[32, 34], logprob=-5.380445957183838)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.01571027562022209, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.01571027562022209), TopLogprob(token=' what', bytes=[32, 119, 104, 97, 116], logprob=-4.515710353851318)]), ChatCompletionTokenLogprob(token=' meaning', bytes=[32, 109, 101, 97, 110, 105, 110, 103], logprob=-0.0002422039397060871, top_logprobs=[TopLogprob(token=' meaning', bytes=[32, 109, 101, 97, 110, 105, 110, 103], logprob=-0.0002422039397060871), TopLogprob(token=' "', bytes=[32, 34], logprob=-8.750242233276367)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=0.0), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-19.375)]), ChatCompletionTokenLogprob(token=' life', bytes=[32, 108, 105, 102, 101], logprob=0.0, top_logprobs=[TopLogprob(token=' life', bytes=[32, 108, 105, 102, 101], logprob=0.0), TopLogprob(token=' Life', bytes=[32, 76, 105, 102, 101], logprob=-18.625)]), ChatCompletionTokenLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.22558236122131348, top_logprobs=[TopLogprob(token=' is', bytes=[32, 105, 115], logprob=-0.22558236122131348), TopLogprob(token=' has', bytes=[32, 104, 97, 115], logprob=-1.6005823612213135)]), ChatCompletionTokenLogprob(token=' one', bytes=[32, 111, 110, 101], logprob=-0.5795155167579651, top_logprobs=[TopLogprob(token=' one', bytes=[32, 111, 110, 101], logprob=-0.5795155167579651), TopLogprob(token=' a', bytes=[32, 97], logprob=-0.8295155167579651)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.042999912053346634, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-0.042999912053346634), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=-3.1679999828338623)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.005114922299981117, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.005114922299981117), TopLogprob(token=' humanity', bytes=[32, 104, 117, 109, 97, 110, 105, 116, 121], logprob=-5.380115032196045)]), ChatCompletionTokenLogprob(token=' most', bytes=[32, 109, 111, 115, 116], logprob=-0.04023458808660507, top_logprobs=[TopLogprob(token=' most', bytes=[32, 109, 111, 115, 116], logprob=-0.04023458808660507), TopLogprob(token=' oldest', bytes=[32, 111, 108, 100, 101, 115, 116], logprob=-3.5402345657348633)]), ChatCompletionTokenLogprob(token=' profound', bytes=[32, 112, 114, 111, 102, 111, 117, 110, 100], logprob=-0.0332377664744854, top_logprobs=[TopLogprob(token=' profound', bytes=[32, 112, 114, 111, 102, 111, 117, 110, 100], logprob=-0.0332377664744854), TopLogprob(token=' fundamental', bytes=[32, 102, 117, 110, 100, 97, 109, 101, 110, 116, 97, 108], logprob=-4.283237934112549)]), ChatCompletionTokenLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.004621301311999559, top_logprobs=[TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-0.004621301311999559), TopLogprob(token=',', bytes=[44], logprob=-5.379621505737305)]), ChatCompletionTokenLogprob(token=' enduring', bytes=[32, 101, 110, 100, 117, 114, 105, 110, 103], logprob=-1.0498623847961426, top_logprobs=[TopLogprob(token=' enduring', bytes=[32, 101, 110, 100, 117, 114, 105, 110, 103], logprob=-1.0498623847961426), TopLogprob(token=' complex', bytes=[32, 99, 111, 109, 112, 108, 101, 120], logprob=-1.2998623847961426)]), ChatCompletionTokenLogprob(token=' questions', bytes=[32, 113, 117, 101, 115, 116, 105, 111, 110, 115], logprob=-1.0001161098480225, top_logprobs=[TopLogprob(token=' questions', bytes=[32, 113, 117, 101, 115, 116, 105, 111, 110, 115], logprob=-1.0001161098480225), TopLogprob(token=' mysteries', bytes=[32, 109, 121, 115, 116, 101, 114, 105, 101, 115], logprob=-1.5001161098480225)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.24117739498615265, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.24117739498615265), TopLogprob(token=' of', bytes=[32, 111, 102], logprob=-2.3661773204803467)]), ChatCompletionTokenLogprob(token=' human', bytes=[32, 104, 117, 109, 97, 110], logprob=-0.014702074229717255, top_logprobs=[TopLogprob(token=' human', bytes=[32, 104, 117, 109, 97, 110], logprob=-0.014702074229717255), TopLogprob(token=' philosophy', bytes=[32, 112, 104, 105, 108, 111, 115, 111, 112, 104, 121], logprob=-4.389701843261719)]), ChatCompletionTokenLogprob(token=' history', bytes=[32, 104, 105, 115, 116, 111, 114, 121], logprob=-0.025838086381554604, top_logprobs=[TopLogprob(token=' history', bytes=[32, 104, 105, 115, 116, 111, 114, 121], logprob=-0.025838086381554604), TopLogprob(token=' existence', bytes=[32, 101, 120, 105, 115, 116, 101, 110, 99, 101], logprob=-4.1508378982543945)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=-0.2528470754623413, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=-0.2528470754623413), TopLogprob(token=',', bytes=[44], logprob=-1.5028470754623413)]), ChatCompletionTokenLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.2028636932373047, top_logprobs=[TopLogprob(token=' It', bytes=[32, 73, 116], logprob=-0.2028636932373047), TopLogprob(token=' Philosoph', bytes=[32, 80, 104, 105, 108, 111, 115, 111, 112, 104], logprob=-2.3278636932373047)]), ChatCompletionTokenLogprob(token=' has', bytes=[32, 104, 97, 115], logprob=-0.2551373839378357, top_logprobs=[TopLogprob(token=' has', bytes=[32, 104, 97, 115], logprob=-0.2551373839378357), TopLogprob(token="'s", bytes=[39, 115], logprob=-1.7551374435424805)]), ChatCompletionTokenLogprob(token=' been', bytes=[32, 98, 101, 101, 110], logprob=-0.09237410873174667, top_logprobs=[TopLogprob(token=' been', bytes=[32, 98, 101, 101, 110], logprob=-0.09237410873174667), TopLogprob(token=' puzzled', bytes=[32, 112, 117, 122, 122, 108, 101, 100], logprob=-3.092374086380005)]), ChatCompletionTokenLogprob(token=' debated', bytes=[32, 100, 101, 98, 97, 116, 101, 100], logprob=-0.16565178334712982, top_logprobs=[TopLogprob(token=' debated', bytes=[32, 100, 101, 98, 97, 116, 101, 100], logprob=-0.16565178334712982), TopLogprob(token=' explored', bytes=[32, 101, 120, 112, 108, 111, 114, 101, 100], logprob=-2.040651798248291)]), ChatCompletionTokenLogprob(token=' by', bytes=[32, 98, 121], logprob=-0.5897237062454224, top_logprobs=[TopLogprob(token=' by', bytes=[32, 98, 121], logprob=-0.5897237062454224), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-1.0897237062454224)]), ChatCompletionTokenLogprob(token=' philosophers', bytes=[32, 112, 104, 105, 108, 111, 115, 111, 112, 104, 101, 114, 115], logprob=-0.0007159530650824308, top_logprobs=[TopLogprob(token=' philosophers', bytes=[32, 112, 104, 105, 108, 111, 115, 111, 112, 104, 101, 114, 115], logprob=-0.0007159530650824308), TopLogprob(token=' scholars', bytes=[32, 115, 99, 104, 111, 108, 97, 114, 115], logprob=-7.875715732574463)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=0.0, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=0.0), TopLogprob(token='.', bytes=[46], logprob=-18.625)]), ChatCompletionTokenLogprob(token=' theolog', bytes=[32, 116, 104, 101, 111, 108, 111, 103], logprob=-0.032515302300453186, top_logprobs=[TopLogprob(token=' theolog', bytes=[32, 116, 104, 101, 111, 108, 111, 103], logprob=-0.032515302300453186), TopLogprob(token=' scientists', bytes=[32, 115, 99, 105, 101, 110, 116, 105, 115, 116, 115], logprob=-3.657515287399292)]), ChatCompletionTokenLogprob(token='ians', bytes=[105, 97, 110, 115], logprob=-2.4199192921514623e-05, top_logprobs=[TopLogprob(token='ians', bytes=[105, 97, 110, 115], logprob=-2.4199192921514623e-05), TopLogprob(token='icans', bytes=[105, 99, 97, 110, 115], logprob=-11.12502384185791)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=0.0, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=0.0), TopLogprob(token='.', bytes=[46], logprob=-20.375)]), ChatCompletionTokenLogprob(token=' scientists', bytes=[32, 115, 99, 105, 101, 110, 116, 105, 115, 116, 115], logprob=-0.0013424679636955261, top_logprobs=[TopLogprob(token=' scientists', bytes=[32, 115, 99, 105, 101, 110, 116, 105, 115, 116, 115], logprob=-0.0013424679636955261), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-7.001342296600342)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=-1.1920928244535389e-07, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=-1.1920928244535389e-07), TopLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=-15.75)])], refusal=None)

mergify · 2025-10-01T23:31:23Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @TheEpicDolphin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

njhill

Thanks again @TheEpicDolphin

njhill · 2025-10-22T15:01:41Z

@TheEpicDolphin could you merge in latest main, this will be needed for the CI to pass.

Edit: Nevermind, I forgot there's a button I can push for this!

njhill · 2025-10-22T17:36:22Z

@TheEpicDolphin it looks like the new test is failing in CI with OOM: https://buildkite.com/vllm/ci/builds/35910#019a0c97-18aa-4e85-bf17-af92608da61e

TheEpicDolphin · 2025-10-22T17:51:18Z

@TheEpicDolphin it looks like the new test is failing in CI with OOM: https://buildkite.com/vllm/ci/builds/35910#019a0c97-18aa-4e85-bf17-af92608da61e

Thanks, looking into it

Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>

TheEpicDolphin · 2025-10-23T05:58:26Z

@njhill i resolved the failing tests :)

22quinn · 2025-10-23T05:59:49Z

Thanks!

Signed-off-by: Giancarlo Delfin <gdelfin@meta.com> Co-authored-by: Nick Hill <nhill@redhat.com>

Signed-off-by: Giancarlo Delfin <gdelfin@meta.com> Co-authored-by: Nick Hill <nhill@redhat.com> Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

…o step_forward * 'step_forward' of https://github.com/raindaywhu/vllm: (148 commits) [Model] Add MoE support for NemotronH (vllm-project#25863) [Metrics] [KVConnector] Add connector prefix cache hit rate stats (vllm-project#26245) [CI] Reorganize entrypoints tests (vllm-project#27403) add SLA information into comparison graph for vLLM Benchmark Suite (vllm-project#25525) [CI/Build] Fix AMD CI: test_cpu_gpu.py (vllm-project#27388) [Bugfix] Fix args settings for guided decoding args (vllm-project#27375) [CI/Build] Fix Prithvi plugin test (vllm-project#27393) [Chore] Remove duplicate `has_` functions in vllm.utils (vllm-project#27372) [Model] Add num_cached_tokens for PoolingRequestOutput (vllm-project#27378) [V1][spec decode] return logprobs for spec decoding (vllm-project#26060) [CORE] Support Prefix Caching with Prompt Embeds (vllm-project#27219) [Bugfix][Core] running queue index leakage exception (vllm-project#26754) [Bugfix] Fix incorrect kv cache metrics in grafana.json (vllm-project#27133) [Bugfix] Fix SLA tuner initialization (vllm-project#27355) [Bugfix] Fix deepseek-ocr multi-image inference and add `merge_by_field_config=True` with tensor schema support (vllm-project#27361) [MLA] Bump FlashMLA (vllm-project#27354) [Chore] Separate out system utilities from vllm.utils (vllm-project#27201) [BugFix] bugfix for Flash Attention MLA with full cuda graph IMA following pr-25490 (vllm-project#27128) [Feature] publisher default set zmq in kv_event config (vllm-project#26915) [Prefix Cache] Use LoRA name for consistent KV-cache block hashing (vllm-project#27211) ...

Signed-off-by: Giancarlo Delfin <gdelfin@meta.com> Co-authored-by: Nick Hill <nhill@redhat.com>

Signed-off-by: Giancarlo Delfin <gdelfin@meta.com> Co-authored-by: Nick Hill <nhill@redhat.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

### What this PR does / why we need it? vllm-project/vllm@c9461e0 Fix ```spec decode rejection sampler```, caused by vllm-project/vllm#26060 Fix some ```import```, caused by vllm-project/vllm#27374 Fix ```scheduler_config.send_delta_data```, caused by #3719 Fix ```init_with_cudagraph_sizes```, caused by vllm-project/vllm#26016 Fix ```vl model```of replacing PatchEmbed's conv3d to linear layer, caused by vllm-project/vllm#27418 ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? CI passed with new added/existing test. - vLLM version: v0.11.0rc3 - vLLM main: vllm-project/vllm@c9461e0 --------- Signed-off-by: Icey <1790571317@qq.com>

mergify bot added the v1 label Oct 1, 2025

mergify bot added the needs-rebase label Oct 1, 2025

TheEpicDolphin force-pushed the v1_spec_decode_logprobs branch from 48bc380 to cc8bc92 Compare October 2, 2025 02:59

mergify bot added speculative-decoding and removed needs-rebase labels Oct 2, 2025

TheEpicDolphin force-pushed the v1_spec_decode_logprobs branch 8 times, most recently from 2cf8c8b to 3969a0c Compare October 3, 2025 00:06

mergify bot added the tpu Related to Google TPUs label Oct 3, 2025

TheEpicDolphin marked this pull request as ready for review October 3, 2025 00:49

TheEpicDolphin requested review from 22quinn, NickLucche, WoosukKwon, alexm-redhat, benchislett, comaniac, houseroad, luccafong, njhill, robertgshaw2-redhat and ywang96 as code owners October 3, 2025 00:49

TheEpicDolphin force-pushed the v1_spec_decode_logprobs branch 2 times, most recently from 2124147 to 5797161 Compare October 3, 2025 17:14

mergify bot removed the tpu Related to Google TPUs label Oct 3, 2025

njhill approved these changes Oct 22, 2025

View reviewed changes

Merge branch 'main' into v1_spec_decode_logprobs

f6b3bcb

njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 22, 2025

TheEpicDolphin force-pushed the v1_spec_decode_logprobs branch from 881eb24 to ac3dbfa Compare October 22, 2025 15:42

TheEpicDolphin force-pushed the v1_spec_decode_logprobs branch 8 times, most recently from 8fe10f3 to fa5c0da Compare October 23, 2025 00:18

[V1][spec decode] return logprobs for spec decoding

8636461

Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>

TheEpicDolphin force-pushed the v1_spec_decode_logprobs branch from fa5c0da to 8636461 Compare October 23, 2025 00:19

22quinn approved these changes Oct 23, 2025

View reviewed changes

22quinn merged commit 6644796 into vllm-project:main Oct 23, 2025
46 checks passed

usberkeley pushed a commit to usberkeley/vllm that referenced this pull request Oct 23, 2025

[V1][spec decode] return logprobs for spec decoding (vllm-project#26060)

e55f05e

Signed-off-by: Giancarlo Delfin <gdelfin@meta.com> Co-authored-by: Nick Hill <nhill@redhat.com>

kingsmad pushed a commit to kingsmad/vllm that referenced this pull request Oct 25, 2025

[V1][spec decode] return logprobs for spec decoding (vllm-project#26060)

038b7a3

Signed-off-by: Giancarlo Delfin <gdelfin@meta.com> Co-authored-by: Nick Hill <nhill@redhat.com>

wxsIcey mentioned this pull request Oct 25, 2025

Upgrade to 0.11.1 newest vllm commit vllm-project/vllm-ascend#3762

Merged

Jialin mentioned this pull request Oct 28, 2025

[Core][Bookkeeping] Update cu_num_accepted_tokens for all req_index #27629

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[V1][spec decode] return logprobs for spec decoding #26060

[V1][spec decode] return logprobs for spec decoding #26060

Uh oh!

TheEpicDolphin commented Oct 1, 2025 •

edited by github-actions bot

Loading

Uh oh!

mergify bot commented Oct 1, 2025

Uh oh!

njhill left a comment

Uh oh!

njhill commented Oct 22, 2025 •

edited

Loading

Uh oh!

njhill commented Oct 22, 2025

Uh oh!

TheEpicDolphin commented Oct 22, 2025 •

edited

Loading

Uh oh!

TheEpicDolphin commented Oct 23, 2025

Uh oh!

22quinn commented Oct 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

[V1][spec decode] return logprobs for spec decoding #26060

[V1][spec decode] return logprobs for spec decoding #26060

Uh oh!

Conversation

TheEpicDolphin commented Oct 1, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Automated Tests

Manual Test

Uh oh!

mergify bot commented Oct 1, 2025

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

njhill commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

njhill commented Oct 22, 2025

Uh oh!

TheEpicDolphin commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TheEpicDolphin commented Oct 23, 2025

Uh oh!

22quinn commented Oct 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TheEpicDolphin commented Oct 1, 2025 •

edited by github-actions bot

Loading

njhill commented Oct 22, 2025 •

edited

Loading

TheEpicDolphin commented Oct 22, 2025 •

edited

Loading