Skip to content

Latest commit

 

History

History
201 lines (176 loc) · 20 KB

README.md

File metadata and controls

201 lines (176 loc) · 20 KB
Raw

Introduction

License: MIT

Bolt is a light-weight library for deep learning. Bolt, as a universal deployment tool for all kinds of neural networks, aims to automate the deployment pipeline and achieve extreme acceleration. Bolt has been widely deployed and used in many departments of HUAWEI company, such as 2012 Laboratory, CBG and HUAWEI Product Lines. If you have questions or suggestions, you can submit issue. QQ群: 833345709

Why Bolt is what you need?

  • High Performance: 15%+ faster than existing open source acceleration libraries.
  • Rich Model Conversion: support Caffe, ONNX, TFLite, Tensorflow.
  • Various Inference Precision: support FP32, FP16, INT8, 1-BIT.
  • Multiple platforms: ARM CPU(v7, v8, v8.2+, v9), X86 CPU(AVX2, AVX512), GPU(Mali, Qualcomm, Intel, AMD)
  • Bolt is the first to support NLP and also supports common CV applications.
  • Minimize ROM/RAM
  • Rich Graph Optimization
  • Efficient Thread Affinity Setting
  • Auto Algorithm Tuning
  • Time-Series Data Acceleration

See more excellent features and details here

Building Status

There are some common used platform for inference. More targets can be seen from scripts/target.sh. Please make a suitable choice depending on your environment. If you want to build on-device training module, you can add --train option. If you want to use multi-threads parallel, you can add --openmp option. If you want to build for cortex-M or cortex-A7 with restricted ROM/RAM(Sensor, MCU), you can see docs/LITE.md.

Bolt defaultly link static library, This may cause some problem on some platforms. You can use --shared option to link shared library.

target platform precision build command Linux Windows MacOS
Android(armv7) fp32,int8 ./install.sh --target=android-armv7 Build Status Build Status Build Status
Android(armv8) fp32,int8 ./install.sh --target=android-aarch64 --fp16=off Build Status Build Status Build Status
Android(armv8.2+) fp32,fp16,int8,bnn ./install.sh --target=android-aarch64 Build Status Build Status Build Status
Android(armv9) fp32,fp16,bf16,int8,bnn ./install.sh --target=android-aarch64_v9 Build Status Build Status Build Status
Android(gpu) fp16 ./install.sh --target=android-aarch64 --gpu Build Status Build Status Build Status
Android(x86_64) fp32,int8 ./install.sh --target=android-x86_64 Build Status Build Status Build Status
iOS(armv7) fp32,int8 ./install.sh --target=ios-armv7 / / Build Status
iOS(armv8) fp32,int8 ./install.sh --target=ios-aarch64 --fp16=off / / Build Status
iOS(armv8.2+) fp32,fp16,int8,bnn ./install.sh --target=ios-aarch64 / / Build Status
Linux(armv7) fp32,int8 ./install.sh --target=linux-armv7_blank Build Status / /
Linux(armv8) fp32,int8 ./install.sh --target=linux-aarch64_blank --fp16=off Build Status / /
Linux(armv8.2+) fp32,fp16,int8,bnn ./install.sh --target=linux-aarch64_blank Build Status / /
Linux(x86_64) fp32,int8 ./install.sh --target=linux-x86_64 Build Status / /
Linux(x86_64_avx2) fp32 ./install.sh --target=linux-x86_64_avx2 Build Status / /
Linux(x86_64_avx512) fp32,int8 ./install.sh --target=linux-x86_64_avx512 Build Status / /
Windows(x86_64) fp32,int8 ./install.sh --target=windows-x86_64 / Build Status /
Windows(x86_64_avx2) fp32 ./install.sh --target=windows-x86_64_avx2 / Build Status /
Windows(gpu) fp16 ./install.sh --target=windows-x86_64_avx2 --gpu --fp16=on / Build Status /
Windows(x86_64_avx512) fp32,int8 ./install.sh --target=windows-x86_64_avx512 / Build Status /
Windows(armv8.2+) fp32,fp16,int8,bnn ./install.sh --target=windows-aarch64 / / Build Status
MacOS(x86_64) fp32,int8 ./install.sh --target=macos-x86_64 / / Build Status
MacOS(x86_64_avx2) fp32 ./install.sh --target=macos-x86_64_avx2 / / Build Status
MacOS(x86_64_avx512) fp32,int8 ./install.sh --target=macos-x86_64_avx512 / / Build Status
MacOS(armv8.2+) fp32,fp16,int8,bnn ./install.sh --target=macos-aarch64 / / Build Status

Quick Start

Two steps to get started with bolt.

  1. Conversion: use X2bolt to convert your model from caffe, onnx, tflite or tensorflow to .bolt file;

  2. Inference: run benchmark with .bolt and data to get the inference result.

    For more details about the usage of X2bolt and benchmark tools, see docs/USER_HANDBOOK.md.

DL Applications in Bolt

Here we show some interesting and useful applications in bolt.

Image Classification
android ios		 Face Detection
ios exe		 Pose Detection
android
Semantics Analysis
android		 Reading Comprehension
android		 Chinese Speech Recognition
android ios

Verified Networks

Bolt has shown its high performance in the inference of common CV, NLP and Recommendation neural networks. Some of the representative networks that we have verified are listed below. You can find detailed benchmark information in docs/BENCHMARK.md.

Application Models
CV Resnet50, Shufflenet, Squeezenet, Densenet, Efficientnet, Mobilenet_v1, Mobilenet_v2, Mobilenet_v3, BiRealNet, ReActNet, Ghostnet, unet, LCNet, Pointnet, hair-segmentation, duc, fcn, retinanet, SSD, Faster-RCNN, Mask-RCNN, Yolov2, Yolov3, Yolov4, Yolov5, ViT, TNT, RepVGG, VitAE, CMT, EfficientFormer ...
NLP Bert, Albert, Tinybert, Neural Machine Translation, Text To Speech(Tactron,Tactron2,FastSpeech+hifigan,melgan), Automatic Speech Recognition, DFSMN, Conformer, Tdnn, FRILL, T5, GPT-2, Roberta, Wenet ...
Recommendation NFM, AFM, ONN, wide&deep, DeepFM, MMOE
More DL Tasks ...

More models than these mentioned above are supported, users are encouraged to further explore.

On-Device Training

On-Device Training has come, it's a beta vesion which supports Lenet, Mobilenet_v1 and Resnet18 for training on the embedded devices and servers. Want more details of on-device training in bolt? Get with the official training tutorial.

Documentations

Everything you want to know about bolt is recorded in the detailed documentations stored in docs.

Articles

教程

Acknowledgement

Bolt refers to the following projects: caffe, onnx, tensorflow, ncnn, mnn, dabnn.

License

The MIT License(MIT)