Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add executor changes for HTTP/gRPC multiplexing #1772

Merged
merged 15 commits into from
May 26, 2020
Merged

Add executor changes for HTTP/gRPC multiplexing #1772

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
0a9064e
Add executor changes for HTTP/gRPC multiplexing
Apr 30, 2020
2e766a2
Remove gRPC port for executor container creation in operator
May 1, 2020
2b7f398
Close executor listener
May 6, 2020
3c474ad
Remove transport flag from operator controller
May 11, 2020
7524b4a
Revert changes in Seldon Deployment Engine
May 13, 2020
a30cbfe
Merge remote-tracking branch 'upstream/master' into multiplex
May 13, 2020
21a0cff
Add Metrics Port Name removed by merge
May 13, 2020
144659b
remove ProbesOnly from SeldonRestApi
May 18, 2020
6c1bc89
Remove probesOnly from Executor tests
May 18, 2020
04e9cd2
Merge remote-tracking branch 'upstream/master' into multiplex
May 18, 2020
5a01b78
Revert "fix after rebase"
May 18, 2020
5abd007
Undo revert
May 18, 2020
1898e2a
Revert "Remove probesOnly from Executor tests"
May 18, 2020
bfd0ab5
Revert "remove ProbesOnly from SeldonRestApi"
May 18, 2020
82eac94
Undo changes to REST server
May 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions executor/api/rest/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ package rest
import (
"context"
"fmt"
"io/ioutil"
"net/http"
"net/url"

"github.com/go-logr/logr"
guuid "github.com/google/uuid"
"github.com/gorilla/mux"
Expand All @@ -15,10 +19,7 @@ import (
"github.com/seldonio/seldon-core/executor/api/metric"
"github.com/seldonio/seldon-core/executor/api/payload"
"github.com/seldonio/seldon-core/executor/predictor"
"github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1"
"io/ioutil"
"net/http"
"net/url"
v1 "github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1"
logf "sigs.k8s.io/controller-runtime/pkg/runtime/log"
)

Expand Down
6 changes: 4 additions & 2 deletions executor/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@ go 1.12

require (
github.com/cloudevents/sdk-go v0.10.2
github.com/fullstorydev/grpcurl v1.5.1 // indirect
github.com/ghodss/yaml v1.0.0
github.com/go-logr/logr v0.1.0
github.com/golang/protobuf v1.3.2
github.com/golang/protobuf v1.3.5
github.com/google/uuid v1.1.1
github.com/gorilla/mux v1.7.3
github.com/grpc-ecosystem/go-grpc-middleware v1.1.0
Expand All @@ -17,12 +18,13 @@ require (
github.com/prometheus/client_golang v1.3.0
github.com/prometheus/common v0.7.0
github.com/seldonio/seldon-core/operator v0.0.0-20200401123312-d4c435ea5217
github.com/soheilhy/cmux v0.1.4
github.com/tensorflow/tensorflow v1.14.0 // indirect
github.com/tensorflow/tensorflow/tensorflow/go/core v0.0.0-00010101000000-000000000000
github.com/uber/jaeger-client-go v2.21.1+incompatible
github.com/uber/jaeger-lib v2.2.0+incompatible // indirect
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7
google.golang.org/grpc v1.27.0
google.golang.org/grpc v1.28.0
gotest.tools v2.2.0+incompatible
k8s.io/api v0.17.2
k8s.io/apimachinery v0.17.2
Expand Down
20 changes: 20 additions & 0 deletions executor/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XL
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cloudevents/sdk-go v0.10.2 h1:CAqHqDHmBkCG4OUeUBt7q2Ql8KV25U+bgPUtlcJelZ4=
github.com/cloudevents/sdk-go v0.10.2/go.mod h1:EHG6NmU3XkIeuueER6+vbnhYfWlgVlfUQVzPC+UK7ao=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f h1:WBZRG4aNOuI15bLRrCgN8fCq8E5Xuty6jGbmSNEvSsU=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
Expand Down Expand Up @@ -92,7 +94,11 @@ github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFP
github.com/elazarl/goproxy v0.0.0-20170405201442-c4fc26588b6e/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4 h1:rEvIZUSZ3fx39WIi3JkQqQBitGwpELBIYWeBVh6wn+E=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/protoc-gen-validate v0.1.0 h1:EQciDnbrYxy13PgWoY8AqoxGiPrpgBZ1R8UNe3ddc+A=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/evanphx/json-patch v4.2.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
github.com/evanphx/json-patch v4.5.0+incompatible h1:ouOWdg56aJriqS0huScTkVXPC5IcNrDCXZ6OoTAWu7M=
Expand All @@ -101,6 +107,8 @@ github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5Kwzbycv
github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fullstorydev/grpcurl v1.5.1 h1:MyN3rJxuk1etiBeNpXzjSM7GjhUndohhkgnRQ5LlPh8=
github.com/fullstorydev/grpcurl v1.5.1/go.mod h1:gfDWAYMHBN+GjNeO30K9mAfpbYWtWd/bOebWSV0l6kY=
github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
Expand Down Expand Up @@ -182,6 +190,9 @@ github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5y
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.5 h1:F768QJ1E9tib+q5Sc8MkdJi1RxLTbRcTf8LJV56aRls=
github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
Expand Down Expand Up @@ -239,6 +250,8 @@ github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJ
github.com/imdario/mergo v0.3.7 h1:Y+UAYTZ7gDEuOfhxKWy+dvb5dRQ6rJjFSdX2HZY1/gI=
github.com/imdario/mergo v0.3.7/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/jhump/protoreflect v1.5.0 h1:NgpVT+dX71c8hZnxHof2M7QDK7QtohIJ7DYycjnkyfc=
github.com/jhump/protoreflect v1.5.0/go.mod h1:eaTn3RZAmMBcV0fifFvlm6VHNz3wSkYyXYWUh7ymB74=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/json-iterator/go v0.0.0-20180612202835-f2b4162afba3/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.5 h1:gL2yXlmiIo4+t+y32d4WGwOjKGYcGOuyrg46vadswDE=
Expand Down Expand Up @@ -387,6 +400,7 @@ github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAm
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/soheilhy/cmux v0.1.4 h1:0HKaf1o97UwFjHH9o5XsHUOF+tqmdA7KEzXLpiyaw0E=
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
Expand Down Expand Up @@ -470,6 +484,7 @@ golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHl
golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180530234432-1e491301e022/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
Expand Down Expand Up @@ -583,6 +598,7 @@ google.golang.org/appengine v1.5.0 h1:KxkO13IPW4Lslp2bz+KHP2E3gtFlrIGNThxkZQ3g+4
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20170818010345-ee236bd376b0/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8 h1:Nw54tB0rB7hY/N0NQvRW8DG4Yk3Q6T9cu9RcFQDu1tc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
Expand All @@ -595,6 +611,7 @@ google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55 h1:gSJIx1SDwno+2El
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190916214212-f660b8655731 h1:Phvl0+G5t5k/EUFUi0wPdUUeTL2HydMQUXHnunWgSb0=
google.golang.org/genproto v0.0.0-20190916214212-f660b8655731/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/grpc v1.8.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.19.1/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
Expand All @@ -603,8 +620,11 @@ google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyac
google.golang.org/grpc v1.23.1/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.24.0 h1:vb/1TCsVn3DcJlQ0Gs1yB1pKI6Do2/QNwxdKqmc/b0s=
google.golang.org/grpc v1.24.0/go.mod h1:XDChyiUovWa60DnaeDeZmSW86xtLtjtZbwvSiRnRtcA=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0 h1:rRYRFMVgRv6E0D70Skyfsr28tDXIuuPZyWGMPdMcnXg=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.28.0 h1:bO/TA4OxCOummhSf10siHuG7vJOiwh7SpRpFZDkOgl4=
google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60=
gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
Expand Down
96 changes: 51 additions & 45 deletions executor/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,15 @@ import (
"encoding/json"
"flag"
"fmt"
"io/ioutil"
"net"
"net/url"
"os"
"os/signal"
"strings"
"syscall"
"time"

"github.com/ghodss/yaml"
"github.com/go-logr/logr"
"github.com/prometheus/common/log"
Expand All @@ -20,28 +29,19 @@ import (
"github.com/seldonio/seldon-core/executor/k8s"
loghandler "github.com/seldonio/seldon-core/executor/logger"
"github.com/seldonio/seldon-core/executor/proto/tensorflow/serving"
"github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1"
"io/ioutil"
"net"
"net/url"
"os"
"os/signal"
v1 "github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1"
"github.com/soheilhy/cmux"
logf "sigs.k8s.io/controller-runtime/pkg/runtime/log"
"strings"
"syscall"
"time"
)

var (
configPath = flag.String("config", "", "Path to kubconfig")
sdepName = flag.String("sdep", "", "Seldon deployment name")
namespace = flag.String("namespace", "", "Namespace")
predictorName = flag.String("predictor", "", "Name of the predictor inside the SeldonDeployment")
httpPort = flag.Int("http_port", 8080, "Executor port")
grpcPort = flag.Int("grpc_port", 8000, "Executor port")
port = flag.Int("port", 8080, "Executor port")
wait = flag.Duration("graceful_timeout", time.Second*15, "Graceful shutdown secs")
protocol = flag.String("protocol", "seldon", "The payload protocol")
transport = flag.String("transport", "rest", "The network transport http or grpc")
filename = flag.String("file", "", "Load graph from file")
hostname = flag.String("hostname", "localhost", "The hostname of the running server")
logWorkers = flag.Int("logger_workers", 5, "Number of workers handling payload logging")
Expand Down Expand Up @@ -92,7 +92,7 @@ func getServerUrl(hostname string, port int) (*url.URL, error) {
return url.Parse(fmt.Sprintf("http://%s:%d/", hostname, port))
}

func runHttpServer(logger logr.Logger, predictor *v1.PredictorSpec, client seldonclient.SeldonApiClient, port int,
func runHttpServer(lis net.Listener, logger logr.Logger, predictor *v1.PredictorSpec, client seldonclient.SeldonApiClient, port int,
probesOnly bool, serverUrl *url.URL, namespace string, protocol string, deploymentName string, prometheusPath string) {

// Create REST API
Expand All @@ -101,7 +101,8 @@ func runHttpServer(logger logr.Logger, predictor *v1.PredictorSpec, client seldo
srv := seldonRest.CreateHttpServer(port)

go func() {
if err := srv.ListenAndServe(); err != nil {
logger.Info("server started")
if err := srv.Serve(lis); err != nil {
logger.Error(err, "Server error")
}
}()
Expand Down Expand Up @@ -129,11 +130,7 @@ func runHttpServer(logger logr.Logger, predictor *v1.PredictorSpec, client seldo

}

func runGrpcServer(logger logr.Logger, predictor *v1.PredictorSpec, client seldonclient.SeldonApiClient, port int, serverUrl *url.URL, namespace string, protocol string, deploymentName string, annotations map[string]string) {
lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
func runGrpcServer(lis net.Listener, logger logr.Logger, predictor *v1.PredictorSpec, client seldonclient.SeldonApiClient, port int, serverUrl *url.URL, namespace string, protocol string, deploymentName string, annotations map[string]string) {
grpcServer, err := grpc.CreateGrpcServer(predictor, deploymentName, annotations, logger)
if err != nil {
log.Fatalf("Failed to create grpc server: %v", err)
Expand Down Expand Up @@ -171,20 +168,14 @@ func main() {
log.Fatal("Protocol must be seldon or tensorflow")
}

if !(*transport == "rest" || *transport == "grpc") {
log.Fatal("Only rest and grpc supported")
}

serverUrl, err := getServerUrl(*hostname, *httpPort)
serverUrl, err := getServerUrl(*hostname, *port)
if err != nil {
log.Fatal("Failed to create server url from", *hostname, *httpPort)
log.Fatal("Failed to create server url from", *hostname, *port)
}

logf.SetLogger(logf.ZapLogger(false))
logger := logf.Log.WithName("entrypoint")

logger.Info("Flags", "transport", *transport)

var predictor *v1.PredictorSpec
if *filename != "" {
logger.Info("Trying to get predictor from file")
Expand Down Expand Up @@ -231,25 +222,40 @@ func main() {
}
defer closer.Close()

if *transport == "rest" {
clientRest, err := rest.NewJSONRestClient(*protocol, *sdepName, predictor, annotations)
if err != nil {
log.Fatalf("Failed to create http client: %v", err)
}
logger.Info("Running http server ", "port", *httpPort)
runHttpServer(logger, predictor, clientRest, *httpPort, false, serverUrl, *namespace, *protocol, *sdepName, *prometheusPath)
} else {
logger.Info("Running http probes only server ", "port", *httpPort)
go runHttpServer(logger, predictor, nil, *httpPort, true, serverUrl, *namespace, *protocol, *sdepName, *prometheusPath)
logger.Info("Running grpc server ", "port", *grpcPort)
var clientGrpc seldonclient.SeldonApiClient
if *protocol == "seldon" {
clientGrpc = seldon.NewSeldonGrpcClient(predictor, *sdepName, annotations)
} else {
clientGrpc = tensorflow.NewTensorflowGrpcClient(predictor, *sdepName, annotations)
}
runGrpcServer(logger, predictor, clientGrpc, *grpcPort, serverUrl, *namespace, *protocol, *sdepName, annotations)
// Create a listener at the desired port.
lis, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
if err != nil {
log.Fatalf("failed to create listener: %v", err)
}
defer lis.Close()

// Create a cmux object.
tcpm := cmux.New(lis)

// Declare the match for different services required.
httpl := tcpm.Match(cmux.HTTP1Fast())
grpcl := tcpm.MatchWithWriters(
cmux.HTTP2MatchHeaderFieldSendSettings("content-type", "application/grpc"))

clientRest, err := rest.NewJSONRestClient(*protocol, *sdepName, predictor, annotations)
if err != nil {
log.Fatalf("Failed to create http client: %v", err)
}
logger.Info("Running HTTP server ", "port", *port)
go runHttpServer(httpl, logger, predictor, clientRest, *port, false, serverUrl, *namespace, *protocol, *sdepName, *prometheusPath)
ukclivecox marked this conversation as resolved.
Show resolved Hide resolved

var clientGrpc seldonclient.SeldonApiClient
if *protocol == "seldon" {
clientGrpc = seldon.NewSeldonGrpcClient(predictor, *sdepName, annotations)
} else {
clientGrpc = tensorflow.NewTensorflowGrpcClient(predictor, *sdepName, annotations)
}
logger.Info("Running gRPC server ", "port", *port)
go runGrpcServer(grpcl, logger, predictor, clientGrpc, *port, serverUrl, *namespace, *protocol, *sdepName, annotations)

// Start cmux serving.
if err := tcpm.Serve(); !strings.Contains(err.Error(),
"use of closed network connection") {
log.Fatal(err)
}
}
6 changes: 3 additions & 3 deletions executor/samples/local/logger/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ BASE=../../..
## REST

run_rest_executor:
${BASE}/executor --sdep seldon-model --namespace default --predictor example --file ./model_rest.yaml --http_port 8000
${BASE}/executor --sdep seldon-model --namespace default --predictor example --file ./model_rest.yaml --port 8000


run_dummy_rest_model:
Expand All @@ -23,12 +23,12 @@ curl_rest_big:
## GRPC

run_grpc_executor:
${BASE}/executor --sdep seldon-model --namespace default --predictor example --file ./model_grpc.yaml --http_port 8000 --grpc_port 5000 --transport grpc
${BASE}/executor --sdep seldon-model --namespace default --predictor example --file ./model_grpc.yaml --port 8000

run_dummy_grpc_model:
cd ${BASE}/../examples/models/mean_classifier && make run_grpc_local

grpc_test:
cd ${BASE}/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0]]}}' -plaintext -proto ./prediction.proto 0.0.0.0:5000 seldon.protos.Seldon/Predict
cd ${BASE}/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0]]}}' -plaintext -proto ./prediction.proto 0.0.0.0:8000 seldon.protos.Seldon/Predict
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This failed for me with

panic: interface conversion: interface {} is nil, not string

goroutine 76 [running]:
github.com/seldonio/seldon-core/executor/predictor.(*PredictorProcess).logPayload(0xc00009aaf0, 0xc0000e8e40, 0xa, 0xc00038eee0, 0x178a2d3, 0x7, 0x19c7800, 0xc0001cf920, 0xc00013e840, 0xc00013e840)
	/home/clive/work/seldon-core/fork-seldon-core/executor/predictor/predictor_process.go:263 +0x3ff
github.com/seldonio/seldon-core/executor/predictor.(*PredictorProcess).Predict(0xc00009aaf0, 0xc0000bcaa0, 0x19c7800, 0xc0001cf920, 0xc000110720, 0x415c4f, 0x7fdd55fac490, 0x203000)
	/home/clive/work/seldon-core/fork-seldon-core/executor/predictor/predictor_process.go:271 +0xd1
github.com/seldonio/seldon-core/executor/api/grpc/seldon.GrpcSeldonServer.Predict(0x19f36e0, 0xc00009b360, 0xc0000ee1b0, 0x19e6020, 0xc000168680, 0xc0000d4000, 0x7ffc970cfc8c, 0x7, 0x19dc640, 0xc00017f860, ...)
	/home/clive/work/seldon-core/fork-seldon-core/executor/api/grpc/seldon/server.go:37 +0x28e
github.com/seldonio/seldon-core/executor/api/grpc/seldon/proto._Seldon_Predict_Handler.func1(0x19dc640, 0xc00017f860, 0x17073c0, 0xc0001582c0, 0x17a60ba, 0x1d, 0x19dc640, 0xc00017f860)
	/home/clive/work/seldon-core/fork-seldon-core/executor/api/grpc/seldon/proto/prediction.pb.go:1538 +0x86
github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1(0x19dc640, 0xc00017f3b0, 0x17073c0, 0xc0001582c0, 0xc000110560, 0xc000110580, 0x48a1d6, 0x5ec23f4b, 0x1dfcd1fb, 0x137bb6277ab6b)
	/home/clive/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/tracing/opentracing/server_interceptors.go:31 +0x165
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x19dc640, 0xc00017f3b0, 0x17073c0, 0xc0001582c0, 0x20, 0x20, 0x163a180, 0xc0001582c0)
	/home/clive/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
github.com/seldonio/seldon-core/executor/api/metric.(*ServerMetrics).UnaryServerInterceptor.func1(0x19dc640, 0xc00017f3b0, 0x17073c0, 0xc0001582c0, 0xc000110560, 0xc0001105c0, 0x12ad3aa, 0x163a180, 0xc0001105e0, 0xc000110560)
	/home/clive/work/seldon-core/fork-seldon-core/executor/api/metric/server.go:43 +0xa0
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x19dc640, 0xc00017f3b0, 0x17073c0, 0xc0001582c0, 0xc000156200, 0x0, 0xc000251b18, 0x40e488)
	/home/clive/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0x19dc640, 0xc00017f3b0, 0x17073c0, 0xc0001582c0, 0xc000110560, 0xc000110580, 0xc00041bb88, 0x4932e8, 0x166efc0, 0xc00017f3b0)
	/home/clive/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:34 +0xd5
github.com/seldonio/seldon-core/executor/api/grpc/seldon/proto._Seldon_Predict_Handler(0x158e480, 0xc000158400, 0x19dc640, 0xc00017f3b0, 0xc00010e6c0, 0xc000256900, 0x19dc640, 0xc00017f3b0, 0xc0003a4a40, 0x1e)
	/home/clive/work/seldon-core/fork-seldon-core/executor/api/grpc/seldon/proto/prediction.pb.go:1540 +0x14b
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000120b60, 0x19ef320, 0xc000298000, 0xc000156200, 0xc000256a20, 0x25fe620, 0x0, 0x0, 0x0)
	/home/clive/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1082 +0x50a
google.golang.org/grpc.(*Server).handleStream(0xc000120b60, 0x19ef320, 0xc000298000, 0xc000156200, 0x0)
	/home/clive/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1405 +0xcc9
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0000446a0, 0xc000120b60, 0x19ef320, 0xc000298000, 0xc000156200)
	/home/clive/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:746 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/home/clive/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:744 +0xa1
Makefile:26: recipe for target 'run_grpc_executor' failed
make: *** [run_grpc_executor] Error 2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the line failing:

seldon-core/executor/predictor/predictor_process.go

Line 263 in 21a0cff

RequestId: p.Ctx.Value(payload.SeldonPUIDHeader).(string),

Which has been raised in a separate issue #1763 which will be fixed once PR #1790 is merged.



4 changes: 2 additions & 2 deletions executor/samples/local/metrics/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ BASE=../../..
## REST

run_rest_executor:
${BASE}/executor --sdep seldon-model --namespace default --predictor example --file ./model_rest.yaml --http_port 8000
${BASE}/executor --sdep seldon-model --namespace default --predictor example --file ./model_rest.yaml --port 8000


run_dummy_rest_model:
Expand All @@ -16,7 +16,7 @@ curl_rest:
## GRPC

run_grpc_executor:
${BASE}/executor --sdep seldon-model --namespace default --predictor example --file ./model_grpc.yaml --http_port 8000 --grpc_port 5000 --transport grpc
${BASE}/executor --sdep seldon-model --namespace default --predictor example --file ./model_grpc.yaml --port 8000

run_dummy_grpc_model:
cd ${BASE}/../examples/models/mean_classifier && make run_grpc_local
Expand Down
Loading