TypeError: argument 'text': 'ChatCompletionMessage' object cannot be converted to 'PyString'

[jstorq]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.8.0 and 2.48.0

Steps to Reproduce

In making calls to OpenAI, we are running into a bug that we cannot handle well down in the Sentry SDK code. If there are no tokens in the response, the code in _calculate_token_usage -> _count_tokens blows up because of changes to the completions object that have not been accounted for in the sdk:

celery-1  |   File "/app/weave_service/core/llm/base.py", line 119, in completion
celery-1  |     return self._client.chat.completions.create(**kwargs)
celery-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/openai.py", line 497, in _sentry_patched_create_sync
celery-1  |     return _execute_sync(f, *args, **kwargs)
celery-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/openai.py", line 486, in _execute_sync
celery-1  |     return gen.send(result)
celery-1  |            ^^^^^^^^^^^^^^^^
celery-1  |   File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/openai.py", line 465, in _new_chat_completion_common
celery-1  |     _set_output_data(span, response, kwargs, integration, finish_span=True)
celery-1  |   File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/openai.py", line 271, in _set_output_data
celery-1  |     _calculate_token_usage(messages, response, span, None, integration.count_tokens)
celery-1  |   File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/openai.py", line 161, in _calculate_token_usage
celery-1  |     output_tokens += count_tokens(choice.message)
celery-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/openai.py", line 88, in count_tokens
celery-1  |     return len(self.tiktoken_encoding.encode_ordinary(s))
celery-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/usr/local/lib/python3.12/site-packages/tiktoken/core.py", line 73, in encode_ordinary
celery-1  |     return self._core_bpe.encode_ordinary(text)
celery-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  | TypeError: argument 'text': 'ChatCompletionMessage' object cannot be converted to 'PyString'

In here:

    def count_tokens(self: "OpenAIIntegration", s: str) -> int:
        if self.tiktoken_encoding is not None:
            return len(self.tiktoken_encoding.encode_ordinary(s))

Expected Result

We would expect this code not to throw an exception but to handle the case.

Actual Result

Exception is thrown

[linear]

PY-2024

[sl0thentr0py]

thx for the report @jstorq