Skip to content
This repository has been archived by the owner on Sep 12, 2024. It is now read-only.

What files are compatible? #124

Open
bedcoding opened this issue Mar 23, 2024 · 0 comments
Open

What files are compatible? #124

bedcoding opened this issue Mar 23, 2024 · 0 comments

Comments

@bedcoding
Copy link

I have read the document below.
https://llama-node.vercel.app/docs/start

So, I looked for a model to use in the source code at the site below.
https://huggingface.co/models?search=ggml

Then an error like this appears. What files should I get?

llama.cpp: loading model from model/llama-2-7b-chat.ggmlv3.q2_K.bin
error loading model: unrecognized tensor type 10

llama_init_from_file: failed to load model
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: Failed to initialize LLama context from file: model/llama-2-7b-chat.ggmlv3.q2_K.bin] {
  code: 'GenericFailure'
}

code:

import { LLM } from "llama-node";
import { LLamaCpp } from "llama-node/dist/llm/llama-cpp.js";

const model = "model/llama-2-7b-chat.ggmlv3.q2_K.bin";
const llama = new LLM(LLamaCpp);
const config = {
    modelPath: model,
    enableLogging: true,
    nCtx: 1024,
    seed: 0,
    f16Kv: false,
    logitsAll: false,
    vocabOnly: false,
    useMlock: false,
    embedding: false,
    useMmap: true,
    nGpuLayers: 0
};

const template = `How are you?`;
const prompt = `A chat between a user and an assistant.
USER: ${template}
ASSISTANT:`;

const run = async () => {
  await llama.load(config);

  await llama.createCompletion({
      nThreads: 4,
      nTokPredict: 2048,
      topK: 40,
      topP: 0.1,
      temp: 0.2,
      repeatPenalty: 1,
      prompt,
  }, (response) => {
      process.stdout.write(response.token);
  });
}

run();
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant