Skip to content

Latest commit

 

History

History
259 lines (211 loc) · 12.3 KB

README_en.md

File metadata and controls

259 lines (211 loc) · 12.3 KB
Raw

mnn-llm

mnn-llm

License Download Documentation Status

Chinese

Example Projects

  • cli: Compile using the command line, for Android compilation refer toandroid_build.sh
  • web: Compile using the command line, runtime requires specifyingweb resources
  • android: Open with Android Studio for compilation;
  • ios: Open with Xcode for compilation; 🚀🚀🚀This sample code is 100% generated by ChatGPT🚀🚀🚀
  • python: mnn-llm python api mnnllm；
  • other: Added capabilities for text embedding, vector querying, document parsing, memory bank, and knowledge base 🔥.

model export and download

For exporting the llm model to ONNX or mnn, please usellm-export

Download models from modelscope：

qwen
glm
llama
phi
embedding

Building

Current CI build status:

Build Status Build Status Build Status Build Status Build Status Build Status

Local Compilation

# clone
git clone --recurse-submodules https://github.com/wangzhaode/mnn-llm.git
cd mnn-llm

# linux
./script/build.sh

# windows msvc
./script/build.ps1

# python wheel
./script/py_build.sh

# android
./script/android_build.sh

# android apk
./script/android_app_build.sh

# ios
./script/ios_build.sh

The default backend used is CPU. If you want to use a different backend, you can add a MNN compilation macro:

  • cuda: -DMNN_CUDA=ON
  • opencl: -DMNN_OPENCL=ON
  • metal: -DMNN_METAL=ON

4. Execution

# linux/macos
./cli_demo ./Qwen2-1.5B-Instruct-MNN/config.json # cli demo
./web_demo ./Qwen2-1.5B-Instruct-MNN/config.json ../web # web ui demo

# windows
.\Debug\cli_demo.exe ./Qwen2-1.5B-Instruct-MNN/config.json
.\Debug\web_demo.exe ./Qwen2-1.5B-Instruct-MNN/config.json ../web

# android
adb push android_build/MNN/OFF/arm64-v8a/libMNN.so /data/local/tmp
adb push android_build/MNN/express/OFF/arm64-v8a/libMNN_Express.so /data/local/tmp
adb push android_build/libllm.so android_build/cli_demo /data/local/tmp
adb push Qwen2-1.5B-Instruct-MNN /data/local/tmp
adb shell "cd /data/local/tmp && export LD_LIBRARY_PATH=. && ./cli_demo ./Qwen2-1.5B-Instruct-MNN/config.json"

Reference

reference