V2 Dataplane in Executor/Operator + Triton Server

doc/source/examples/notebooks.rst

@@ -2,6 +2,16 @@
 Notebooks
 =========
 
+Seldon Core Setup
+----
+
+.. toctree::
+   :titlesonly:
+
+   Install Seldon Core <seldon_core_setup>
+
+
+
 Prepackaged Inference Server Examples
 -----
 
@@ -34,7 +44,7 @@ Specialised Framework Examples
 .. toctree::
    :titlesonly:
 
-   NVIDIA TensorRT MNIST <nvidia_mnist>
+   NVIDIA TensorRT MNIST <tensorrt>
    OpenVINO ImageNet <openvino>
    OpenVINO ImageNet Ensemble <openvino_ensemble>
    ONNX ResNet with Intel nGraph<onnx_resnet>

doc/source/examples/tensorrt.nblink

@@ -0,0 +1,3 @@
+{
+  "path": "../../../examples/models/tensorrt/triton_tensorrt.ipynb"
+}

doc/source/graph/protocols.md

@@ -4,8 +4,9 @@ Tensorflow protocol is only available in version >=1.1.
 
 Seldon Core supports the following data planes:
 
- * REST and gRPC Seldon protocol
- * REST and gRPC Tensorflow Serving Protocol
+ * [REST and gRPC Seldon protocol](#rest-and-grpc-seldon-protocol)
+ * [REST and gRPC Tensorflow Serving Protocol](#rest-and-grpc-tensorflow-protocol)
+ * [REST and gRPC V2 KFServing Protocol](#v2-kfserving-protocol)
 
 ## REST and gRPC Seldon Protocol
 
@@ -39,3 +40,10 @@ General considerations:
   * The name of the model in the `graph` section of the SeldonDeployment spec must match the name of the model loaded onto the Tensorflow Server.
 
 
+## V2 KFServing Protocol 
+
+Seldon has collaborated with the [NVIDIA Triton Server Project](https://github.com/triton-inference-server/server) and the [KFServing Project](https://github.com/kubeflow/kfserving) to create a new ML inference protocol.
+
+This protocol can be used by specifing `protocol: kfserving`. At present we only support NVIDIA Triton prepackaged servers but will have support for custom and other server types in the near future.
+
+See [example notebook](../examples/protocol_examples.html). 

doc/source/index.rst

@@ -59,6 +59,7 @@ Documentation Index
 
    MLflow Server <servers/mlflow.md>
    SKLearn server <servers/sklearn.md>
+   Triton Inference Server <servers/triton.md>   
    Tensorflow Serving <servers/tensorflow.md>
    XGBoost server <servers/xgboost.md>
 

doc/source/python/api/seldon_core.proto.rst

@@ -2,9 +2,9 @@ seldon\_core.proto package
 ==========================
 
 .. automodule:: seldon_core.proto
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 Submodules
 ----------
@@ -13,16 +13,15 @@ seldon\_core.proto.prediction\_pb2 module
 -----------------------------------------
 
 .. automodule:: seldon_core.proto.prediction_pb2
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.proto.prediction\_pb2\_grpc module
 -----------------------------------------------
 
 .. automodule:: seldon_core.proto.prediction_pb2_grpc
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
+   :members:
+   :undoc-members:
+   :show-inheritance:
 

doc/source/python/api/seldon_core.rst

@@ -2,16 +2,16 @@ seldon\_core package
 ====================
 
 .. automodule:: seldon_core
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 Subpackages
 -----------
 
 .. toctree::
 
-    seldon_core.proto
+   seldon_core.proto
 
 Submodules
 ----------
@@ -20,9 +20,9 @@ seldon\_core.api\_tester module
 -------------------------------
 
 .. automodule:: seldon_core.api_tester
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.app module
 -----------------------
@@ -36,128 +36,127 @@ seldon\_core.batch\_processor module
 ------------------------------------
 
 .. automodule:: seldon_core.batch_processor
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.flask\_utils module
 --------------------------------
 
 .. automodule:: seldon_core.flask_utils
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.imports\_helper module
 -----------------------------------
 
 .. automodule:: seldon_core.imports_helper
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.metadata module
 ----------------------------
 
 .. automodule:: seldon_core.metadata
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.metrics module
 ---------------------------
 
 .. automodule:: seldon_core.metrics
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.microservice module
 --------------------------------
 
 .. automodule:: seldon_core.microservice
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.microservice\_tester module
 ----------------------------------------
 
 .. automodule:: seldon_core.microservice_tester
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.persistence module
 -------------------------------
 
 .. automodule:: seldon_core.persistence
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.seldon\_client module
 ----------------------------------
 
 .. automodule:: seldon_core.seldon_client
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.seldon\_methods module
 -----------------------------------
 
 .. automodule:: seldon_core.seldon_methods
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.serving\_test\_gen module
 --------------------------------------
 
 .. automodule:: seldon_core.serving_test_gen
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.storage module
 ---------------------------
 
 .. automodule:: seldon_core.storage
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.user\_model module
 -------------------------------
 
 .. automodule:: seldon_core.user_model
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.utils module
 -------------------------
 
 .. automodule:: seldon_core.utils
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.version module
 ---------------------------
 
 .. automodule:: seldon_core.version
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:
 
 seldon\_core.wrapper module
 ---------------------------
 
 .. automodule:: seldon_core.wrapper
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
+   :members:
+   :undoc-members:
+   :show-inheritance:
 

doc/source/servers/triton.md

@@ -0,0 +1,25 @@
+# Triton Inference Server
+
+If you have a model that can be run on [NVIDIA Triton Inference Server](https://github.com/triton-inference-server/server) you can use Seldon's Prepacked Triton Server.
+
+Triton has multiple supported backends including support for TensorRT, Tensorflow, PyTorch and ONNX models. For further details see the [Triton supported backends documentation](https://docs.nvidia.com/deeplearning/triton-inference-server/master-user-guide/docs/model_repository.html#section-framework-model-definition).
+
+## Example
+
+```
+apiVersion: machinelearning.seldon.io/v1alpha2
+kind: SeldonDeployment
+metadata:
+  name: triton
+spec:
+  protocol: kfserving
+  predictors:
+  - graph:
+      implementation: TRITON_SERVER
+      modelUri: gs://seldon-models/trtis/simple-model
+      name: simple
+    name: simple
+    replicas: 1
+```
+
+Try out a [worked notebook](../examples/protocol_examples.html)