Feature Request: Support Aya

[maziyarpanahi]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

The new Aya Expanse models require support for CohereTokenizer.

• https://huggingface.co/CohereForAI/aya-expanse-8b
• https://huggingface.co/CohereForAI/aya-expanse-32b

ValueError: Tokenizer class CohereTokenizer does not exist or is not currently imported.

However, I see something similar has been done but only for the Command R/R+ and I was wondering that can be extended to these new models: #7063

Motivation

These are new multilingual models with small sizes and great performance that could be useful to users with languages that are not represented in the existing models

Possible Implementation

#7063

[foldl]

Yes, Command-R and Aya share the same mode arch.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.