Merge pull request #1016 from cliveseldon/executor

Executor

.gitignore

@@ -194,3 +194,4 @@ testing/scripts/go
 .tox
 
 operator/operator.tar
+operator/helm/tmp/

components/routers/epsilon-greedy/Makefile

@@ -1,8 +1,14 @@
 IMAGE_VERSION=1.3
 IMAGE_NAME=seldonio/mab_epsilon_greedy
 
-build:
-	s2i build . seldonio/seldon-core-s2i-python3:0.4 $(IMAGE_NAME):$(IMAGE_VERSION)
+build_rest:
+	s2i build . -E environment_rest seldonio/seldon-core-s2i-python3:0.15 $(IMAGE_NAME)_rest:$(IMAGE_VERSION)
 
-push_to_dockerhub:
-	docker push $(IMAGE_NAME):$(IMAGE_VERSION)
+push_to_dockerhub_rest:
+	docker push $(IMAGE_NAME)_rest:$(IMAGE_VERSION)
+
+build_grpc:
+	s2i build . -E environment_grpc seldonio/seldon-core-s2i-python3:0.15 $(IMAGE_NAME)_grpc:$(IMAGE_VERSION)
+
+push_to_dockerhub_grpc:
+	docker push $(IMAGE_NAME)_grpc:$(IMAGE_VERSION)

components/routers/epsilon-greedy/environment_grpc

@@ -0,0 +1,4 @@
+MODEL_NAME=EpsilonGreedy
+API_TYPE=GRPC
+SERVICE_TYPE=ROUTER
+PERSISTENCE=0

doc/source/analytics/analytics.md

@@ -1,22 +1,17 @@
-# Seldon Core Analytics
+# Metrics
 
-Seldon Core exposes metrics that can be scraped by Prometheus. The core metrics are exposed by the service orchestrator (```engine```) and API gateway (```server_ingress```).
+Seldon Core exposes metrics that can be scraped by Prometheus. The core metrics are exposed by the service orchestrator (```executor```).
 
 The metrics are:
 
 **Prediction Requests**
 
- * ```seldon_api_engine_server_requests_duration_seconds_(bucket,count,sum) ``` : Requests to the service orchestrator from an ingress, e.g. API gateway or Ambassador
- * ```seldon_api_engine_client_requests_duration_seconds_(bucket,count,sum) ``` : Requests from the service orchestrator to a component, e.g., a model
- * ```seldon_api_server_ingress_requests_duration_seconds_(bucket,count,sum) ``` : Requests to the API Gateway from an external client
-
-**Feedback Requests**
-
- * ```seldon_api_model_feedback_reward_total``` : Reward sent via Feedback API
- * ```seldon_api_model_feedback_total``` : Total feedback requests
+ * ```seldon_api_executor_server_requests_duration_seconds_(bucket,count,sum) ``` : Requests to the service orchestrator from an ingress, e.g. API gateway or Ambassador
+ * ```seldon_api_executor_client_requests_duration_seconds_(bucket,count,sum) ``` : Requests from the service orchestrator to a component, e.g., a model
 
 Each metric has the following key value pairs for further filtering which will be taken from the SeldonDeployment custom resource that is running:
 
+  * service
   * deployment_name
   * predictor_name
   * predictor_version
@@ -31,10 +26,11 @@ Each metric has the following key value pairs for further filtering which will b
 Seldon Core provides an example Helm analytics chart that displays the above Prometheus metrics in Grafana. You can install it with:
 
 ```bash
-helm install seldon-core-analytics --name seldon-core-analytics \
-     --repo https://storage.googleapis.com/seldon-charts \
-     --set grafana_prom_admin_password=password \
-     --set persistence.enabled=false \
+helm install seldon-core-analytics seldon-core-analytics \
+   --repo https://storage.googleapis.com/seldon-charts \
+   --set grafana_prom_admin_password=password \
+   --set persistence.enabled=false \
+   --namespace seldon-system
 ```
 
 The available parameters are:
@@ -45,10 +41,14 @@ The available parameters are:
 Once running you can expose the Grafana dashboard with:
 
 ```bash
-kubectl port-forward $(kubectl get pods -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') 3000:3000
+kubectl port-forward $(kubectl get pods -n seldon-system -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') 3000:3000 -n seldon-system
 ```
 
 You can then view the dashboard at http://localhost:3000/dashboard/db/prediction-analytics?refresh=5s&orgId=1
 
 ![dashboard](./dashboard.png)
 
+## Example
+
+There is [an example notebook you can use to test the metrics](../examples/metrics.html).
+

doc/source/analytics/logging.md

@@ -1,116 +1,50 @@
-# Centralised Logging with the Elastic Stack 
+# Payload Logging
 
-## Introduction
-
-Here we will set up EFK (elasticsearch, fluentd/fluentbit, kibana) as a stack to gather logs from SeldonDeployments and make them searchable.
-
-This demo is aimed at minikube.
-
-Alternatives are available and if you are running in cloud then you can consider a managed service from your cloud provider.
-
-If you just want to bootstrap a full logging and request tracking setup for minikube, run ./full-setup.sh. That includes the [request logging setup](./request-logging/README.md)
-
-## Setup
-
-If helm is not already set up then it needs to be configured:
-
-```
-kubectl -n kube-system create sa tiller
-kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
-helm init --service-account tiller
-```
-
-Install elasticsearch with minikube configuration:
-
-```
-helm install --name elasticsearch elasticsearch --version 7.1.1 --namespace=logs -f elastic-minikube.yaml --repo https://helm.elastic.co
-```
-
-Then fluentd as a collection agent (chosen in preference to fluentbit - see notes at end):
-
-```
-helm install fluentd-elasticsearch --name fluentd --namespace=logs -f fluentd-values.yaml --repo https://kiwigrid.github.io
-```
-
-And kibana UI:
-
-```
-helm install kibana --version 7.1.1 --name=kibana --namespace=logs --set service.type=NodePort --repo https://helm.elastic.co
-```
-
-## Generating Logging
-
-First we need seldon and a seldon deployment.
-
-Install seldon operator:
-
-```
-helm install --name seldon-core ../../helm-charts/seldon-core-operator/ --namespace seldon-system
-```
-
-Check that it now recognises the seldon CRD by running `kubectl get sdep`.
-
-Now a model:
-
-```
-helm install --name seldon-single-model ../../helm-charts/seldon-single-model/ --set engine.env.LOG_MESSAGES_EXTERNALLY="false"
-```
-
-And the loadtester:
+Logging of request and response payloads from your Seldon Deployment can be accomplished by adding a logging section to any part of the Seldon deployment graph. An example is shown below:
 
 ```
-kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust --overwrite
-helm install --name seldon-core-loadtesting ../../helm-charts/seldon-core-loadtesting/ --set locust.host=http://seldon-single-model-seldon-single-model:8000 --set oauth.enabled=false --set oauth.key=oauth-key --set oauth.secret=oauth-secret --set locust.hatchRate=1 --set locust.clients=1 --set loadtest.sendFeedback=0 --set locust.minWait=0 --set locust.maxWait=0 --set replicaCount=1
-```
-
-## Inspecting Logging and Search for Requests
-
-To find kibana URL
+apiVersion: machinelearning.seldon.io/v1
+kind: SeldonDeployment
+metadata:
+  name: seldon-model
+spec:
+  name: test-deployment
+  predictors:
+  - componentSpecs:
+    - spec:
+        containers:
+        - image: seldonio/mock_classifier:1.3
+          name: classifier
+    graph:
+      children: []
+      endpoint:
+        type: REST
+      name: classifier
+      type: MODEL
+      logger:
+        url: http://mylogging-endpoint
+        mode: all
+    name: example
+    replicas: 1
 
 ```
-echo $(minikube ip)":"$(kubectl get svc kibana-kibana -n logs -o=jsonpath='{.spec.ports[?(@.port==5601)].nodePort}')
-```
-
-When Kibana appears for the first time there will be a brief animation while it initializes.
-On the Welcome page click Explore on my own.
-From the top-left or from the `Visualize and Explore Data` panel select the `Discover` item.
-In the form field Index pattern enter logstash-*
-It should read "Success!" and Click the `> Next` step button on the right.
-In the next form select timestamp from the dropdown labeled `Time Filter` field name.
-From the bottom-right of the form select `Create index pattern`.
-In a moment a list of fields will appear.
-From the top-left or the home screen's `Visualize and Explore Data` panel, select the `Discover` item.
-The log list will appear.
-Refine the list a bit by selecting `log` near the bottom the left-hand Selected fields list.
-When you hover over or click on the word `log`, click the `Add` button to the right of the label.
-You can create a filter using the `Add Filter` button under `Search`. The field can be `kubernetes.labels.seldon-app` and the value can be an 'is' match on `seldon-single-model-seldon-single-model`.
-
-The custom fields in the request bodies may not currently be in the index. If you hover over one in a request you may see `No cached mapping for this field`.
-
-To add mappings, go to `Management` at the bottom-left and then `Index Patterns`. Hit `Refresh` on the index created earlier. The number of fields should increase and `request.data.names` should be present.
-
-Now we can go back and add a further filter for `data.names` with the operator `exists`. We can add further filters if we want, such as the presence of a feature name or the presence of a feature value.
-
-![picture](https://raw.githubusercontent.com/SeldonIO/seldon-core/master/examples/centralised-logging/kibana-custom-search.png)
-
-## Notes
-
-The fluentd setup is configured to ensure only labelled pods are logged and seldon pods are automatically labelled.
 
-Fluentbit can be chosen instead. This could be installed with:
+The logging for the top level requets response is provided by:
 
 ```
-helm install stable/fluent-bit --name=fluent-bit --namespace=logs --set backend.type=es --set backend.es.host=elasticsearch-master
+      logger:
+        url: http://mylogging-endpoint
+        mode: all
 ```
 
-In that case pods would be logged. At the time of writing fluentbit only supports [excluding pods by label, not including](https://github.com/fluent/fluent-bit/issues/737).
+In this example both request and response payloads as specified by the `mode` attribute are sent as CloudEvents to the url `http://mylogging-endpoint`.
 
-Seldon can also be used to log full HTTP requests. See [request logging guide](./request-logging/README.md)
+The specification is:
 
-The elasticsearch backend is not available externally by default but can be exposed if needed for debugging with `kubectl patch svc elasticsearch-master -n logs -p '{"spec": {"type": "LoadBalancer"}}'`
+ * url: Any url. Optional. If not provided then it will default to the default knative borker in the namespace of the Seldon Deployment.
+ * mode: Either `request`, `response` or `all`
 
-## Credits
+## Example Notebook
 
-Loosely based on https://www.katacoda.com/javajon/courses/kubernetes-observability/efk
-Fluentd filtering based on https://blog.ptrk.io/tweaking-an-efk-stack-on-kubernetes/
+You can try out an [example notebook with logging](../examples/payload_logging.html)
 

doc/source/examples/metrics.nblink

@@ -0,0 +1,3 @@
+{
+  "path": "../../../examples/models/metrics/metrics.ipynb"
+}

doc/source/examples/notebooks.rst

@@ -29,6 +29,7 @@ Notebooks
    Chainer MNIST <chainer_mnist>
    Kubeflow Seldon E2E Pipeline <kubeflow_seldon_e2e_pipeline>
    Max gRPC Message Size <max_grpc_msg_size>
+   Metrics <metrics>
    Model with Custom Metrics <tmpl_model_with_metrics>
    MLflow <mlflow>
    MLflow Pre-packaged MOdel Server A/B Test <mlflow_server_ab_test_ambassador>
@@ -37,7 +38,9 @@ Notebooks
    NVIDIA TensorRT MNIST <nvidia_mnist>
    OpenVINO ImageNet <openvino>
    OpenVINO ImageNet Ensemble <openvino_ensemble>
-   ONNX ResNet with Intel nGraph<onnx_resnet>   
+   ONNX ResNet with Intel nGraph<onnx_resnet>
+   Payload Logging <payload_logging>
+   Protocol Examples <protocol_examples>
    R Iris Classifier <r_iris>
    R MNIST Classifier <r_mnist>   
    REST timeouts <rest_timeouts>	         
@@ -48,6 +51,7 @@ Notebooks
    Sklearn, Xgboost, Tensorflow Server Examples <server_examples>
    Tensorflow MNIST <deep_mnist>
    TFserving MNIST <tfserving_mnist>
+   Tracing <tracing>
    Spam Classification TRANSFORMER component <transformer_spam_model>
    Spam Classification TRANSFORMER + COMBINER Component <combiner_spam_model>
 

doc/source/examples/payload_logging.nblink

@@ -0,0 +1,3 @@
+{
+  "path": "../../../examples/models/payload_logging/payload_logging.ipynb"
+}

doc/source/examples/protocol_examples.nblink

@@ -0,0 +1,3 @@
+{
+  "path": "../../../notebooks/protocol_examples.ipynb"
+}

doc/source/examples/tracing.nblink

@@ -0,0 +1,3 @@
+{
+  "path": "../../../examples/models/tracing/tracing.ipynb"
+}

doc/source/graph/annotations.md

@@ -32,36 +32,6 @@ You can configure aspects of Seldon Core via annotations in the SeldonDeployment
     * Locations : SeldonDeployment.spec.annotations
     * [gRPC headless example](grpc_load_balancing_ambassador.md)
 
-Otherwise any annotations starting with `seldon.io/engine-` will be interpreted as specifying environment variables for the engine container. These include:
-
-  * ```seldon.io/engine-java-opts``` : Java Opts for Service Orchestrator
-    * Locations : SeldonDeployment.spec.annotations
-    * [Java Opts example](model_engine_java_opts.md)
-    * Translates to the environment variable JAVA_OPTS
-  * ```seldon.io/engine-seldon-log-requests``` : Whether to log raw requests from engine
-    * Locations : SeldonDeployment.spec.annotations
-    * Translates to the environment variable SELDON_LOG_REQUESTS
-  * ```seldon.io/engine-seldon-log-responses``` : Whether to log raw responses from engine
-    * Locations : SeldonDeployment.spec.annotations
-    * Translates to the environment variable SELDON_LOG_RESPONSES
-  * ```seldon.io/engine-seldon-log-messages-externally``` : Option to turn on logging of requests via a logging service
-    * Locations : SeldonDeployment.spec.annotations
-    * Translates to the environment variable SELDON_LOG_MESSAGES_EXTERNALLY
-  * ```seldon.io/engine-seldon-log-message-type``` : Option to override type set on messages when sending to logging service. Used to determine which logger impl
-    * Locations : SeldonDeployment.spec.annotations
-    * Translates to the environment variable SELDON_LOG_MESSAGE_TYPE
-  * ```seldon.io/engine-seldon-message-logging-service``` : Option to override url to broker that sends to logging service
-    * Locations : SeldonDeployment.spec.annotations
-    * Translates to the environment variable SELDON_MESSAGE_LOGGING_SERVICE
-
-More details on logging-related variables can be seen in the [request-logging example](https://github.com/SeldonIO/seldon-core/tree/master/examples/centralised-logging/README.md).
-
-Environment variables for the engine can also be set in the `svcOrchSpec` section of the SeldonDeployment, alongside engine resources. For examples see the helm charts or the [distributed tracing example](./distributed-tracing.md).
-
-If both annotations and `svcOrchSpec` environment variables are used to set an environment variable for the engine container then `svcOrchSpec` environment variables take priority.
-
-The above are the key engine env vars. For a full listing of engine env vars see the application.properties file of the engine source code.
-
 
 ### Misc
 

doc/source/graph/distributed-tracing.md

@@ -1,14 +1,10 @@
 # Distributed Tracing
 
-You can use Jaeger Open Tracing to trace your API calls to Seldon Core.
-
-This feature is available from versions >=0.2.5-SNAPSHOT of the core images and presently in:
-
- * Python wrappers >=0.5-SNAPSHOT
+You can use Open Tracing to trace your API calls to Seldon Core.
 
 ## Install Jaeger
 
-You will need to install Jaeger on your Kubernetes cluster. Follow their [documentation](https://github.com/jaegertracing/jaeger-kubernetes).
+You will need to install Jaeger on your Kubernetes cluster. Follow their [documentation](https://www.jaegertracing.io/docs/1.16/operator/)
 
 ## Configuration
 
@@ -143,4 +139,4 @@ An example is show below:
 
 ## Worked Example
 
-[A fully worked template example](../examples/tmpl_model_tracing.html) is provided.
+[A fully worked template example](../examples/tracing.html) is provided.

doc/source/graph/protocols.md

@@ -0,0 +1,37 @@
+# Protocols
+
+Tensorflow protocol is only available in version >=1.1.
+
+Seldon Core supports the following data planes:
+
+ * REST and gRPC Seldon protocol
+ * REST and gRPC Tensorflow Serving Protocol
+
+## REST and gRPC Seldon Protocol
+
+ * [REST Seldon Protocol](../reference/apis/index.html)
+
+Seldon is the default protocol for SeldonDeployment resources. You can specify the gRPC protocol by setting `transport: grpc` in your SeldonDeployment resource or ensuring all components in the graph have endpoint.tranport set ot grpc.
+
+See [example notebook](../examples/protcol_examples.html). 
+
+## REST and gRPC Tensorflow Protocol
+
+   * [REST Tensorflow Protocol definition](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/api_rest.md).
+   * [gRPC Tensorflow Procotol definition](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_service.proto).
+
+Activate this protocol by speicfying `protocol: tensorflow` and `transport: rest` or `transport: grpc` in your Seldon Deployment. See [example notebook](../examples/protcol_examples.html). 
+
+For Seldon graphs the protocol will work as expected for single model graphs for Tensorflow Serving servers running as the single model in the graph. For more complex graphs you can chain models:
+
+ * Sending the response from the first as a request to the second. This will be done automatically when you defined a chain of models as a Seldon graph. It is up to the user to ensure the response of each changed model can be fed a request to the next in the chain.
+ * Only Predict calls can be handled in multiple model chaining.
+
+General considerations:
+
+  * Seldon components marked as MODELS, INPUT_TRANSFORMER and OUTPUT_TRANSFORMERS will allow a PredictionService Predict method to be called.
+  * GetModelStatus for any model in the graph is available.
+  * GetModelMetadata for any model in the graph is available.
+  * Combining and Routing with the Tensorflow protocol is not presently supported.
+
+