Improve message trimming #787
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| if key == "name": | ||
| num_tokens += tokens_per_name | ||
| num_tokens += 3 # every reply is primed with <|start|>assistant<|message|> | ||
| return num_tokens |
There was a problem hiding this comment.
can we just have one token counter ? Can you add these changes to token_counter ?
There was a problem hiding this comment.
just requesting this one change ^
unless there's a strong reason to have two token counters ? @duc-phamh
There was a problem hiding this comment.
The openai_token_counter recursively calls itself. Of course we can refactor it into one token counter, but leaving the OpenAI code separate like this would make it easier to apply future changes in case OpenAI update this function. Therefore, I believe it's better to leave it this way.
Please let me know if you think otherwise.
|
@duc-phamh going to wait until we can get a new release of our existing code base and then merge this |
|
Thank you, we really look forward to it. Our production relies on this, and occasionally we'd have error with message generation without the fix here. |
Changes:
process_system_message.process_messages.These changes should resolve issue #742, along with some other edge cases as in the test file.