Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Nomic Embed #5468

Merged
merged 4 commits into from
Feb 13, 2024
Merged

Add support for Nomic Embed #5468

merged 4 commits into from
Feb 13, 2024

Conversation

cebtenzzre
Copy link
Collaborator

@cebtenzzre cebtenzzre commented Feb 12, 2024

Nomic Embed is now working in llama.cpp using these changes (f32 MSE is 3.6e-10 on a simple test).

This PR also changes the llama.cpp names of some of BERT's tensors to match the more accurate ones the conversion script is currently using.

Note: This pull request is currently built on top of #5466, so that should be merged first.

@cebtenzzre cebtenzzre marked this pull request as ready for review February 12, 2024 22:38
@jacoobes
Copy link

ur such a goat ( greatest of all time)

@iamlemec
Copy link
Collaborator

Awesome! I had to do the following to convert-hf-to-gguf.py to get it to run but after that it worked like a charm and matches up with sentence_transformers.

     def set_gguf_parameters(self):
         super().set_gguf_parameters()
-        self.gguf_writer.add_causal_attention(self.hparams["causal"])
+        self.gguf_writer.add_context_length(self.hparams["n_ctx"])
         self.gguf_writer.add_rope_freq_base(self.hparams["rotary_emb_base"])
-        self.gguf_writer.add_pooling_layer(True)
+        self.gguf_writer.add_layer_norm_eps(self.hparams["layer_norm_epsilon"])

@cebtenzzre cebtenzzre merged commit ea9c8e1 into master Feb 13, 2024
51 of 58 checks passed
jordankanter pushed a commit to jordankanter/llama.cpp that referenced this pull request Mar 13, 2024
hodlen pushed a commit to hodlen/llama.cpp that referenced this pull request Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants