refresh Cohere #141
refresh Cohere #141
Conversation
…s_configs/cohere-chat/configs.yaml and 3 other changes
…oards/evaluators/evaluators_leaderboard.csv
| model_name: "command" | ||
| mode: "chat" | ||
| max_tokens: 2048 | ||
| pretty_name: "Cohere Chat" |
There was a problem hiding this comment.
Why remove the chat model?
if it's because it was updated, can we get the updated results on the leaderboard instead?
There was a problem hiding this comment.
The chat mode of the command model was not really a different model before, just a different prompting style. These have been further unified (essentially turning plain instructions into single-turn chats), making them not worth listing as separate entries.
Having a single entry also will make it easier to give you more regular updates.
There was a problem hiding this comment.
I see, so client.chat is being depreciated?
Then let's remove that from the code also?
There was a problem hiding this comment.
It's more for multi-turn chats, indeed less appropriate for tests like this. Removed in eb52df3
|
Nice improvement @sanderland! Congrats! Updating the command results with the last model makes sense thanks for doing that! However, let's refrain from removing models like cohere-chat unless there are updates. Considering that some papers reference the leaderboard, I think it's important to maintain consistency by keeping all models listed. Let me know your thoughts. |
|
Merged, thanks @sanderland ! |
I see there are various images and results that are derived from the old results. If you need anything else to update these, please let me know.