Code

Llama.cpp example for cpp backend#2904

Code

Merged

mreso merged 19 commits intomasterpytorch/serve:masterfrom feature/cpp_llama_cpp_rebasepytorch/serve:feature/cpp_llama_cpp_rebaseCopy head branch name to clipboardJan 26, 2024

+564-67

Commits

Commits on Jan 26, 2024

• Version1 of llm inference with cpp backend

  Show description for 26abfd7
  

  shrinath-suresh

  authored and

  mreso

  committed
  26abfd7

  Copy full SHA for 26abfd7

• Adapt to removal of TS backend

  mreso

  committed
  94b9c9c

  Copy full SHA for 94b9c9c

• Re-add test for llama.cpp example

  mreso

  committed
  49649d5

  Copy full SHA for 49649d5

• Add llama.cpp as a submodule

  mreso

  committed
  5804216

  Copy full SHA for 5804216

• Point to correct llama.cpp installation

  mreso

  committed
  6add8da

  Copy full SHA for 6add8da

• Build llama.cpp in build.sh

  mreso

  committed
  ed4114d

  Copy full SHA for ed4114d

• Skip llama.cpp example test if model weights are not available

  mreso

  committed
  4946825

  Copy full SHA for 4946825

• renamed torchscript_model folder into examples

  mreso

  committed
  25992e8

  Copy full SHA for 25992e8

• Adjust to new base_handler interface

  mreso

  committed
  0f00fed

  Copy full SHA for 0f00fed

• Remove debug statement

  mreso

  committed
  d50b19b

  Copy full SHA for d50b19b

• Rename llamacpp class + remove dummy.pt file

  mreso

  committed
  7c5e242

  Copy full SHA for 7c5e242

• Move llamacpp config.json

  mreso

  committed
  ccb9c4d

  Copy full SHA for ccb9c4d

• Moved and created prompt file

  mreso

  committed
  641386c

  Copy full SHA for 641386c

• Reset context for mutiple batch entries

  mreso

  committed
  8e33517

  Copy full SHA for 8e33517

• Add doc for llamacpp example

  mreso

  committed
  9a9b69e

  Copy full SHA for 9a9b69e

• Fix spell check

  mreso

  committed
  d08e4eb

  Copy full SHA for d08e4eb

• Replace output example in llamacpp example

  mreso

  committed
  2d070a2

  Copy full SHA for 2d070a2

• Move cpp example src into main examples folder

  mreso

  committed
  f9dccd3

  Copy full SHA for f9dccd3

• Convert cerr/cout into logs

  mreso

  committed
  2cb7b03

  Copy full SHA for 2cb7b03