Skip to content

Conversation

@ryan-lempka
Copy link
Collaborator

Problem

Running the embedding fine-tuning tutorial fails at nemo/data-flywheel/embedding-finetuning/2_finetuning_and_inference.ipynb when executing the customization job creation:

customization = client_with_wandb.customization.jobs.create(
    name="llama-3.2-1b-embed-sft",
    ...
)

Error message:

UnprocessableEntityError: Error code: 422 - {'detail': 'Model llama-3.2-nv-embedqa-1b@v2 is not enabled for customization. Contact your administrator to enable it.'}

The llama-3.2-nv-embedqa-1b-v2 embedding model is disabled by default in NeMo Microservices Helm chart v25.10+.

To Confirm Disabled Embedding Model

Pull helm chart

helm pull nmp/nemo-microservices-helm-chart --version 25.10.0 --untar

Check embedding model default configuration

grep -A 5 "nvidia/llama-3.2-nv-embedqa-1b@v2:" nemo-microservices-helm-chart/values.yaml | head -8

nvidia/llama-3.2-nv-embedqa-1b@v2:
  # -- The name for target model.
  name: llama-3.2-nv-embedqa-1b@v2
  # -- The namespace for target model.
  namespace: nvidia
  # -- Whether to enable the model.
  enabled: false

Solution

  • Added a single command to the README that registers the embedding model with NeMo Customizer.
  • The command uses the correct Kubernetes label selector (app.kubernetes.io/name=nemo-customizer), is idempotent and safe to run on existing deployments, and preserves all other registered models.

@shashank3959 shashank3959 self-requested a review November 4, 2025 05:41
@shashank3959 shashank3959 merged commit 7431045 into NVIDIA:main Nov 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants