V2 Dataplane in Executor/Operator + Triton Server #2023

ukclivecox · 2020-06-28T07:48:31Z

Allows executor to use v2 dataplane
- Adds local example to executor
Adds v2 protocol to operator
Adds examples to protocol_examples notebook
Initial docs

review-notebook-app · 2020-06-28T07:48:37Z

Check out this pull request on

Review Jupyter notebook visual diffs & provide feedback on notebooks.

Powered by ReviewNB

seldondev · 2020-06-28T07:50:00Z

Sun Jun 28 07:49:59 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=1

seldondev · 2020-06-28T07:51:00Z

Sun Jun 28 07:50:59 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=2

adriangonz

This looks great @cliveseldon!!!

executor/Makefile

executor/api/grpc/kfserving/client.go

executor/api/grpc/kfserving/server.go

executor/api/rest/server.go

servers/triton/samples/simple.yaml

gaocegege

Thanks for the awesome feature! Looking forward to it.

executor/api/constants.go

operator/apis/machinelearning.seldon.io/v1/seldondeployment_types.go

seldondev · 2020-09-04T18:06:15Z

Fri Sep 4 18:06:11 UTC 2020
The logs for [lint] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=4

seldondev · 2020-09-04T18:06:15Z

Fri Sep 4 18:06:10 UTC 2020
The logs for [pr-build] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=3

seldondev · 2020-09-05T07:17:52Z

Sat Sep 5 07:17:50 UTC 2020
The logs for [pr-build] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=5

seldondev · 2020-09-05T07:17:54Z

Sat Sep 5 07:17:54 UTC 2020
The logs for [lint] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=6

seldondev · 2020-09-05T08:23:26Z

Sat Sep 5 08:23:25 UTC 2020
The logs for [lint] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=8

seldondev · 2020-09-05T08:23:36Z

Sat Sep 5 08:23:34 UTC 2020
The logs for [pr-build] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=7

seldondev · 2020-09-05T10:40:47Z

Sat Sep 5 10:40:42 UTC 2020
The logs for [pr-build] [9] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/9.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=9

seldondev · 2020-09-05T10:40:47Z

Sat Sep 5 10:40:42 UTC 2020
The logs for [lint] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=10

operator/controllers/seldondeployment_prepackaged_servers.go

seldondev · 2020-09-22T10:47:06Z

Tue Sep 22 10:47:05 UTC 2020
The logs for [pr-build] [29] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/29.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=29

seldondev · 2020-09-22T10:47:13Z

Tue Sep 22 10:47:12 UTC 2020
The logs for [lint] [30] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/30.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=30

seldondev · 2020-09-22T12:46:07Z

Tue Sep 22 12:46:06 UTC 2020
The logs for [lint] [32] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/32.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=32

seldondev · 2020-09-22T12:46:10Z

Tue Sep 22 12:46:09 UTC 2020
The logs for [pr-build] [31] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/31.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=31

ukclivecox · 2020-09-22T13:06:17Z

/test integration

ukclivecox · 2020-09-22T13:06:26Z

/test notebooks

seldondev · 2020-09-22T13:07:43Z

Tue Sep 22 13:07:41 UTC 2020
The logs for [integration] [33] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/33.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=33

seldondev · 2020-09-22T13:07:53Z

Tue Sep 22 13:07:52 UTC 2020
The logs for [notebooks] [34] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/34.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=34

seldondev · 2020-09-22T19:32:54Z

Tue Sep 22 19:32:53 UTC 2020
The logs for [lint] [36] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/36.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=36

seldondev · 2020-09-22T19:32:56Z

Tue Sep 22 19:32:55 UTC 2020
The logs for [pr-build] [35] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/35.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=35

ukclivecox · 2020-09-22T22:36:32Z

/test integration

ukclivecox · 2020-09-22T22:36:41Z

/test notebooks

seldondev · 2020-09-22T22:38:07Z

Tue Sep 22 22:38:05 UTC 2020
The logs for [integration] [37] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/37.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=37

seldondev · 2020-09-22T22:38:29Z

Tue Sep 22 22:38:28 UTC 2020
The logs for [notebooks] [38] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/38.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=38

seldondev · 2020-09-23T00:54:37Z

@cliveseldon: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
integration	`747fc45`	link	`/test integration`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

axsaucedo

Looks great! I added a couple of points/questions

axsaucedo · 2020-09-23T07:15:56Z

executor/api/constants.go

@@ -2,6 +2,7 @@ package api

 const ProtocolSeldon = "seldon"
 const ProtocolTensorflow = "tensorflow"
+const ProtocolKFserving = "kfserving"


Nit: KFserving -> KfServing/KFServing

they are all lowercase

Oh i meant ProtocolKFserving -> ProtocolKFServing

executor/api/grpc/kfserving/client.go

axsaucedo · 2020-09-23T07:18:34Z

executor/api/grpc/kfserving/client.go

+			grpc.WithInsecure(),
+		}
+		opts = append(opts, grpc2.AddClientInterceptors(s.Predictor, s.DeploymentName, modelName, s.annotations, s.Log))
+		conn, err := grpc.Dial(fmt.Sprintf("%s:%d", host, port), opts...)


Seems first iteration won't have ssl, which is reasonable - but may be worth mentioning briefly on docs or failing in the operator if ssl is enabled with protocl kfserving

I don't see SSL on any of the grpc clients. Sounds like this is a wider issue to address?

Yeah sounds good, I think we can just open a gh issue

axsaucedo · 2020-09-23T07:23:36Z

executor/api/grpc/kfserving/server.go

+	panic("implement me")
+}
+
+func (g GrpcKFServingServer) ModelMetadata(ctx context.Context, request *inference.ModelMetadataRequest) (*inference.ModelMetadataResponse, error) {


It seems in the client, model metadata is not implemented, but metadata is - would this not be using th emodelMetadata function instead of the metadata function?

As I understand it client modelMetaData is only used by the Seldon specific Graph Metadata call. But I agree this terminology is confusing.

axsaucedo · 2020-09-23T07:24:44Z

executor/api/rest/server.go

@@ -160,6 +159,12 @@ func (r *SeldonRestApi) Initialise() {
 			r.Router.NewRoute().Path("/v1/models:predict").Methods("OPTIONS", "POST").HandlerFunc(r.wrapMetrics(metric.PredictionHttpServiceName, r.predictions))  // Nonstandard path - Seldon extension
 			r.Router.NewRoute().Path("/v1/models/{"+ModelHttpPathVariable+"}").Methods("GET", "OPTIONS").HandlerFunc(r.wrapMetrics(metric.StatusHttpServiceName, r.status))
 			r.Router.NewRoute().Path("/v1/models/{"+ModelHttpPathVariable+"}/metadata").Methods("GET", "OPTIONS").HandlerFunc(r.wrapMetrics(metric.MetadataHttpServiceName, r.metadata))
+		case api.ProtocolKFserving:


It seems metadata is implemented but not defined here, is that intentional?

seldondev · 2020-09-23T07:59:03Z

Wed Sep 23 07:59:01 UTC 2020
The logs for [pr-build] [39] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/39.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=39

seldondev · 2020-09-23T07:59:11Z

Wed Sep 23 07:59:10 UTC 2020
The logs for [lint] [40] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/40.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=40

seldondev · 2020-09-23T08:42:20Z

Wed Sep 23 08:42:19 UTC 2020
The logs for [pr-build] [41] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/41.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=41

seldondev · 2020-09-23T08:42:30Z

Wed Sep 23 08:42:29 UTC 2020
The logs for [lint] [42] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/42.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=42

seldondev · 2020-09-23T08:45:35Z

Wed Sep 23 08:45:33 UTC 2020
The logs for [pr-build] [43] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/43.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=43

seldondev · 2020-09-23T08:46:24Z

Wed Sep 23 08:46:23 UTC 2020
The logs for [lint] [44] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2023/44.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2023 --build=44

ukclivecox added 8 commits May 25, 2020 14:37

initial additions for triton server

6f21e36

Merge branch 'master' into 1648_v2_dataplane

21d975f

Add kfserving API initial integration to executor for REST

8f41829

Add v2 grpc implementation and local sample

822b19a

rebase

1b45df8

update due to executor interface change and go.sum additions

3a3a2b6

Update protos and local examples

dd9d16f

Add examples to protocol notebook

c284bdf

seldondev added the do-not-merge/work-in-progress label Jun 28, 2020

seldondev added the size/XXL label Jun 28, 2020

adriangonz reviewed Jul 2, 2020

View reviewed changes

gaocegege reviewed Jul 15, 2020

View reviewed changes

executor/api/constants.go Outdated Show resolved Hide resolved

operator/apis/machinelearning.seldon.io/v1/seldondeployment_types.go Show resolved Hide resolved

rebase

943a7cd

Update to triton 20-08 triton

7564b69

Update protocol example

f59936c

Update docs

f161cdb

ukclivecox changed the title ~~WIP: V2 Dataplane in Executor/Operator + Triton Server~~ V2 Dataplane in Executor/Operator + Triton Server Sep 5, 2020

seldondev removed the do-not-merge/work-in-progress label Sep 5, 2020

adriangonz reviewed Sep 11, 2020

View reviewed changes

operator/controllers/seldondeployment_prepackaged_servers.go Outdated Show resolved Hide resolved

Merge branch 'master' into 1648_v2_dataplane

dd8ac14

rebase

dde0b68

fix executor samples

747fc45

axsaucedo reviewed Sep 23, 2020

View reviewed changes

Update to not impletemted message

a8237d5

Update KFServing constant

36b279a

Change v1 to v2 in kfserving typo

b07758d

axsaucedo merged commit 02f6bd0 into SeldonIO:master Sep 23, 2020

adriangonz mentioned this pull request Oct 1, 2020

Add pre-packaged server for ONNX #1458

Closed

V2 Dataplane in Executor/Operator + Triton Server #2023

V2 Dataplane in Executor/Operator + Triton Server #2023

Conversation

ukclivecox commented Jun 28, 2020 • edited Loading

review-notebook-app bot commented Jun 28, 2020

seldondev commented Jun 28, 2020

seldondev commented Jun 28, 2020

adriangonz left a comment

Choose a reason for hiding this comment

gaocegege left a comment

Choose a reason for hiding this comment

seldondev commented Sep 4, 2020

seldondev commented Sep 4, 2020

seldondev commented Sep 5, 2020

seldondev commented Sep 5, 2020

seldondev commented Sep 5, 2020

seldondev commented Sep 5, 2020

seldondev commented Sep 5, 2020

seldondev commented Sep 5, 2020

seldondev commented Sep 22, 2020

seldondev commented Sep 22, 2020

seldondev commented Sep 22, 2020

seldondev commented Sep 22, 2020

ukclivecox commented Sep 22, 2020

ukclivecox commented Sep 22, 2020

seldondev commented Sep 22, 2020

seldondev commented Sep 22, 2020

seldondev commented Sep 22, 2020

seldondev commented Sep 22, 2020

ukclivecox commented Sep 22, 2020

ukclivecox commented Sep 22, 2020

seldondev commented Sep 22, 2020

seldondev commented Sep 22, 2020

seldondev commented Sep 23, 2020

axsaucedo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seldondev commented Sep 23, 2020

seldondev commented Sep 23, 2020

seldondev commented Sep 23, 2020

seldondev commented Sep 23, 2020

seldondev commented Sep 23, 2020

seldondev commented Sep 23, 2020

ukclivecox commented Jun 28, 2020 •

edited

Loading