OpenVINO Model Server 2020.4

OpenVINO™ Model Server 2020.4 introduces support for Inference Engine in version 2020.4.
Read OpenVINO-Release Notes to learn more about changes. The most important for the model server scenarios is:

• introduced support for the Bfloat16 data type for inferencing using the 3rd generation Intel® Xeon® Scalable processor

You can use an OVMS public Docker image based on clearlinux via the following command:
docker pull intelaipg/openvino-model-server:2020.4

OpenVINO Model Server 2020.3

OpenVINO™ Model Server 2020.3 introduces support for Inference Engine in version 2020.3.
Refer to OpenVINO-Release Notes to learn more about enhancements. The most important for the model server scenarios are:

• Introducing Long-Term Support (LTS), a new release type that provides longer-term maintenance and support with a focus on stability and compatibility
• Added support for new FP32 and INT8 models to enable more vision and text use cases: 3D U-Net, MobileFace, EAST, OpenPose, RetinaNet, and FaceNet
• Improved the support of AVX2 and AVX512 instruction sets in the CPU preprocessing module
• Added support for new model operations
• Introduced support for bfloat16 (BF16) data type for inferencing
• Included security, functionality bug fixes, and minor capability changes

OpenVINO Model Server 2020.3 release has the following changes and enhancements:

• Documentation for Multi-Device Plugin usage to enable load balancing across multiple devices for a single model.
• Added a Quick start guide
• Documentation improvements

Bug fixes:

• Fixed unnecessary model reload that occurred for multiple versions of the model
• Fixed race condition for simultaneous loading and unloading of the same version
• Fixed bug in face detection example

You can use an OVMS public Docker image based on clearlinux via the following command:
docker pull intelaipg/openvino-model-server:2020.3

OpenVINO Model Server 2020.1

OpenVINO™ Model Server 2020.1 introduces support for Inference Engine in version 2020.1.
Refer to OpenVINO-Release Notes to learn more about enhancements. The most relevant for the model server use case are:

• Inference Engine integrated with ngraph
• Low-precision runtime for INT8
• Added support for multiple new layers and operations
• Numerous improvements in the plugin implementation

OpenVINO Model Server 2020.1 release has the following new features and changes:

• Speeded up inference output serialization – up to 40x faster – models with big outputs will have noticeably shorter latency
• Added exemplary client sending inference requests from multiple cameras in parallel
• Added support for tensorflow 2.0 and python3.8 with backward compatibility
• Updated functional tests to use IR models from OpenVINO Model Zoo
• Updated functional tests to use mino for S3 compatible model storage

Bug fixes:

• Fixed model files detection and import for certain name patterns
• Corrected kubernetes demo in GCP

Note: In version 2020.1 CPU extensions library was removed. Extensions are include into the CPU plugin.
Extension library is now optional to include custom layers only.

You can use an OVMS public Docker image based on OpenVINO runtime image via the following command:

docker pull intelaipg/openvino-model-server:2020.1

OpenVINO Model Server 2019 R3

OpenVINO™ Model Server 2019 R3 introduces support for Inference Engine in version 2019 R3.
Refer to OpenVINO-Release Notes to learn more about enhancements. The most relevant for the model server use case are:

• Improved performance through network loading optimizations and sped up inference by reducing model loading time. This is useful when shape size changes between inferences.
• Added support for Ubuntu* 18.04
• Added support for multiple new layers and operations
• Numerous improvements in the plugin implementation for all supported devices

OpenVINO Model Server 2019 R3 release has the following new features and changes:

• Ability to start the server with multi-worker configuration and parallel inference execution. A new set of parameters are introduced for controlling the number of server threads and parallel inference executions:
  -- grpc_workers
  -- rest_workers
  -- nireq
  Read more about this in performance tuning guide.
  This new feature improves throughput results when employing hardware accelerators like Intel® Movidius™ VPU HDDL.
• The target device is now configurable on the model level for running inference operations by adding the parameter target_device in the command line and in the service configuration file. The DEVICE environment variable is no longer used.
• Added the option to pass additional configuration to the employed plugins with parameter plugin_config
• Included recommendation to use CPU affinity with multiple replicas in Kubernetes via a CPU manager and a static assignment policy.

You can use a public Docker image based on clearlinux base image via the following command:

docker pull intelaipg/openvino-model-server:2019.3

OpenVINO Model Server 2019 R2

OpenVINO™ Model Server 2019 R2 introduces support for Inference Engine in 2019 R2 version of the Intel® Distribution of OpenVINO toolkit. Refer to the Release Notes to learn more about enhancements. The most relevant enhancements for model server use cases:

• Added new non-vision topologies: GNMT, BERT, TDNN (NNet3), ESPNet, etc. to enable machine translation, natural language processing and speech use cases
• Added support for model in FP16 precision in CPU plugin
• Performance improvements with CPU execution
• Added support for multiple new layers and operations

OpenVINO Model Server 2019 R2 release brings additional new capabilities:

• Added option to change model shape in runtime - it is now possible to change the model input data shapes without recreating it. The model server can also adjust the served model parameters to fit to the input data in the received request. Learn more about it
• Public docker image is now based on clearlinux. It is expected to bring execution optimization
• Added support for Intel Movidius™ Myriad™ X VPU HDDL accelerators
• Added exemplary face detection Python application to demonstrate automatic model shape reconfiguration

You can use a public docker image based on clearlinux base image via a command:

docker pull intelaipg/openvino-model-server:2019.2

OpenVINO Model Server 2019 R1.1

In OpenVINO Model Server 2019 R1.1 there is introduced support for Inference Engine in version 2019 R1.1.
Refer to OpenVINO-Release Notes to learn more about introduced improvements. Most important enhancements are:

• alignment with Intel® Movidius™ Myriad™ X Development Kit R7 release.
• support for mPCIe and M.2 form factor versions of Intel® Vision Accelerator Design with Intel® Movidius™ VPUs.
• Myriad plugin is now available in open source

Release OpenVINO Model Server 2019 R1.1 brings also the following new features and changes:

• Added RESTful API - all implemented functions can be accessed using gRPC and REST interfaces according to TensorFlow Serving API. Check the client examples and Jupyter notebook to learn how to use the new interface.
• added exemplary kubeflow pipelines which demo OpenVINO Model Server deployment in Kubernetes and TensorFlow model optimization using Model Optimizer from OpenVINO Toolkit
• Added implementation of GetModelStatus function - it reports state of served models
• Model version update can be disabled by setting FILE_SYSTEM_POLL_WAIT_SECONDS to 0 or negative value.
• Improved error handling for model loading issues like network problems or access permissions
• Updated versions of python dependencies

You can use a public docker image based on Intel python base image via a command:

docker pull intelaipg/openvino-model-server:2019.1.1

OpenVINO Model Server 2019 R1

There is now changed the naming convention for OpenVINO Model Server versions. It became consistent with OpenVINO SDK release names. It should be now easier to map which version of OVMS is using which inference engine backend.

In OpenVINO Model Server 2019 R1 there is introduced support for Inference Engine in version 2019 R1.
Refer to OpenVINO-RelNotes to learn more about introduced improvements. Most important enhancements are:

• Added support for many new operations in ONNX*, TensorFlow* and MXNet* frameworks. Topologies like Tiny YOLO v3, full DeepLab v3, bi-direction
• More than 10 new pre-trained models are added including gaze estimation, action recognition encoder/decoder, text recognition, instance segmentation networks to expand to newer use cases.
• Improved support for Low-Precision 8-bit Integer inference
• upgraded mkl-dnn version to v0.18
• Added support for many new layers, activation types and operations.

Exemplary grpc client has now option to transpose the input data in two directions: NCHW>NHWC and NHWC>NCHW.

Special kudos for @joakimr-axis for his contribution is dockerfiles cleanup and enhancements.

You can use a public docker image based on Intel python base image via a command:
docker pull intelaipg/openvino-model-server:2019.1

OpenVINO Model Server v0.5

Release 0.5 adds the following improvements:

• added version policy which controls filtering conditions for the served model versions
• automatic update of served model versions based on file system changes
• demonstrative Jupyter notebook showing OVMS deployment and evaluation
• added custom Minio configuration options which support S3 compliant storage providers - PR23
• added support for anonymous access to S3 and GS cloud storage
• added support for Movidius stick

You can use a public docker image based on Intel python base image via a command:
docker pull intelaipg/openvino-model-server:0.5

OpenVINO Model Server v0.4

Release 0.4 brings the following improvements:

Docker images are updated to use OpenVINO release R5. Corespondent docker image is intelaipg/openvino-model-server:0.4. It supports models created by the OpenVINO model optimizer version R5. It can be used with low precision models boosting execution performance.

Additional optional parameter is supported to start the server --batch_size. It accepts the following values:

• integer value : it changes the batch size set by be model optimizer at server startup
• phrase 'auto' : it changes the batch size at run time based on the inference requests. It adds one time delay for the model reloading but it is recommended for sequential requests with the same batch size.
• empty value sets the served batch size based on the imported model settings from the model optimizer.

You can use a public docker image based on Intel python base image via a command:
docker pull intelaipg/openvino-model-server:0.4

OpenVINO Model Server v0.3

Release 0.3 introduces the following new features:

• GetModelMetadata call returns now a complete information about the model inputs and outputs. There is included an exemplary gRPC client which can retrieve the model details
• OpenVINO Model Server is compatible with the Inference Engine from release R4 and R5 :
  • The docker build with DLDT sources supports now the Inference Engine from OpenVINO release R4
  • The docker build with Intel® Distribution of OpenVINO™ toolkit supports now the release R4 and R5
  • It is not compatible with Inference Engine from release R3 and earlier

Refer to OpenVINO toolkit release notes to learn about new capabilities in both versions.

You can use a public docker image based on intel python base image via a command:
docker pull intelaipg/openvino-model-server:0.3