Support for `nomic-ai/nomic-embed-text-v2-moe`

[michaelact]

Model description

TEI currently does not seem to support Nomic's latest embedding model: nomic-ai/nomic-embed-text-v2-moe. When I try to load the model using the container image ghcr.io/huggingface/text-embeddings-inference:1.6.0, I get this error:

tei  | 2025-02-21T12:03:12.972121Z  INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "nom**-**/*****-*****-****-**-moe", revision: None, tokenization_workers: None, dtype: None, pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "379cf6f93689", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }
tei  | 2025-02-21T12:03:12.972326Z  INFO hf_hub: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hf-hub-0.3.2/src/lib.rs:55: Token file not found "/root/.cache/huggingface/token"
tei  | 2025-02-21T12:03:13.061604Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:20: Starting download
tei  | 2025-02-21T12:03:13.061624Z  INFO download_artifacts:download_pool_config: text_embeddings_core::download: core/src/download.rs:53: Downloading `1_Pooling/config.json`
tei  | 2025-02-21T12:03:13.061695Z  INFO download_artifacts:download_new_st_config: text_embeddings_core::download: core/src/download.rs:77: Downloading `config_sentence_transformers.json`
tei  | 2025-02-21T12:03:13.061735Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:40: Downloading `config.json`
tei  | 2025-02-21T12:03:13.061771Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:43: Downloading `tokenizer.json`
tei  | 2025-02-21T12:03:13.061798Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:47: Model artifacts downloaded in 195.324µs
tei  | 2025-02-21T12:03:13.529301Z  INFO text_embeddings_router: router/src/lib.rs:188: Maximum number of tokens per request: 512
tei  | 2025-02-21T12:03:13.529350Z  INFO text_embeddings_core::tokenization: core/src/tokenization.rs:28: Starting 4 tokenization workers
tei  | 2025-02-21T12:03:14.590727Z  INFO text_embeddings_router: router/src/lib.rs:230: Starting model backend
tei  | 2025-02-21T12:03:14.590785Z  INFO text_embeddings_backend: backends/src/lib.rs:360: Downloading `model.safetensors`
tei  | 2025-02-21T12:03:14.590856Z  INFO text_embeddings_backend: backends/src/lib.rs:244: Model weights downloaded in 71.202µs
tei  | 2025-02-21T12:03:15.057394Z  INFO text_embeddings_backend_candle: backends/candle/src/lib.rs:332: Starting FlashNomicBert model on Cuda(CudaDevice(DeviceId(1)))
tei  | 2025-02-21T12:03:15.057766Z ERROR text_embeddings_backend: backends/src/lib.rs:255: Could not start Candle backend: Could not start backend: config is not supported
tei  | Error: Could not create backend
tei  |
tei  | Caused by:
tei  |     Could not start backend: Could not start a suitable backend
tei exited with code 1

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

• https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe

[nayanbde]

If I deploy this model using Huggingface TEI script to spin up an endpoint, it gives me the same error again:

{     "timestamp": "2025-06-03T00:15:31.299753Z",     "level": "INFO",     "message": "Starting FlashNomicBert model on Cuda(CudaDevice(DeviceId(1)))",     "target": "text_embeddings_backend_candle",     "filename": "backends/candle/src/lib.rs",     "line_number": 403 }
--
{     "timestamp": "2025-06-03T00:15:31.299983Z",     "level": "ERROR",     "message": "Could not start Candle backend: Could not start backend: config is not supported",     "target": "text_embeddings_backend",     "filename": "backends/src/lib.rs",     "line_number": 388 }
Error: Could not create backend
Caused by:    Could not start backend: Could not start a suitable backend

The script I am using is from the huggingface: https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe

import json
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

try:
	role = sagemaker.get_execution_role()
except ValueError:
	iam = boto3.client('iam')
	role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']

# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'nomic-ai/nomic-embed-text-v2-moe'
}


# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	image_uri=get_huggingface_llm_image_uri("huggingface-tei",version="1.7.0"),
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1,
	instance_type="ml.g5.2xlarge",
  )
  
# send request
predictor.predict({
	"inputs": "My name is Clara and I am",
})