Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Seldon Manager Config Map has an extra comma that can cause parsing error #3652

Closed
gangpeng opened this issue Oct 9, 2021 — with Board Genius Sync · 1 comment · Fixed by #3693
Closed

Seldon Manager Config Map has an extra comma that can cause parsing error #3652

gangpeng opened this issue Oct 9, 2021 — with Board Genius Sync · 1 comment · Fixed by #3693
Assignees
Labels

Comments

Copy link

gangpeng commented Oct 9, 2021

Describe the bug

Seldon Manager config map has an extra "," in prediction servers section at following line:

This causes the webhook to panic with following stack:
/usr/local/go/src/net/http/server.go:2993 +0x39b
{"level":"info","ts":1633731619.884156,"logger":"seldondeployment","msg":"Validating v1 Webhook called for CREATE","name":"iris-model"}
2021/10/08 22:20:19 http: panic serving 172.17.0.1:1969: Unable to unmarshall predictor_servers
json string due to invalid character '}'
looking for beginning of object key string
goroutine 6050 [running]:
net/http.(*conn).serve.func1(0xc000626640)
/usr/local/go/src/net/http/server.go:1804 +0x153
panic(0x1a13620, 0xc000a01160)
/usr/local/go/src/runtime/panic.go:971 +0x499

github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1.getPredictorServerConfigsFromMap(0xc00043af00, 0x1ec6f88, 0xc000120008, 0xc00004616e)
/workspace/apis/machinelearning.seldon.io/v1/prepack.go:143 +0x1d7
github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1.getPredictorServerConfigs(0x203000, 0x7fc1ecda2ce0, 0x3f)
/workspace/apis/machinelearning.seldon.io/v1/prepack.go:135 +0x249
github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1.(*SeldonDeploymentSpec).checkPredictiveUnits(0xc0005b67d8, 0xc0005164c0, 0xc0005164b0, 0xc0007a8bd0, 0x0, 0x0, 0x0, 0xc000516208, 0x4dbf53, 0xc0001340c0)
/workspace/apis/machinelearning.seldon.io/v1/seldondeployment_webhook.go:100 +0xc8b
github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1.(*SeldonDeploymentSpec).ValidateSeldonDeployment(0xc0005b67d8, 0x1ca3d90, 0x27)
/workspace/apis/machinelearning.seldon.io/v1/seldondeployment_webhook.go:251 +0x573
github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1.(*SeldonDeployment).ValidateCreate(0xc0005b66c0, 0xc00095bb00, 0x24)
/workspace/apis/machinelearning.seldon.io/v1/seldondeployment_webhook.go:290 +0xe5
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatingHandler).Handle(0xc0005974b8, 0x1ec6f50, 0xc000a96800, 0xc00095bb00, 0x24, 0xc0009a3e20, 0x19, 0xc000a15178, 0x2, 0xc000a15180, ...)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/webhook/admission/validator.go:71 +0xe37
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(0xc000e8c600, 0x1ec6f50, 0xc000a96800, 0xc00095bb00, 0x24, 0xc0009a3e20, 0x19, 0xc000a15178, 0x2, 0xc000a15180, ...)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/webhook/admission/webhook.go:146 +0xd5
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc000e8c600, 0x7fc1ecd69d90, 0xc000930d20, 0xc00097cd00)
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/webhook/admission/http.go:99 +0x1045

To reproduce

  1. Install Seldon on Minikube:
    $helm install seldon-core seldon-core-operator \
    --repo https://storage.googleapis.com/seldon-charts
    --set ambassador.enabled=false
    --set managerCreateResources=true
    --namespace seldon-system

  2. Deploy Iris following Seldon doc:
    $kubectl create namespace seldon
    $ kubectl apply -f - << END
    apiVersion: machinelearning.seldon.io/v1
    kind: SeldonDeployment
    metadata:
    name: iris-model
    namespace: seldon
    spec:
    name: iris
    predictors:

  • graph:
    implementation: SKLEARN_SERVER
    modelUri: gs://seldon-models/sklearn/iris
    name: classifier
    name: default
    replicas: 1
    END

I also created a go program to reproduce the error. Please open attached zip file and extract both files to the same directory and run "go predictor.go".

Expected behaviour

Controller should accept the deployment.

Environment

minikube version
minikube version: v1.22.0
commit: a03fbcf166e6f74ef224d4a63be4277d017bb62e
Archive.zip

@gangpeng gangpeng added bug triage Needs to be triaged and prioritised accordingly labels Oct 9, 2021
@ukclivecox ukclivecox removed the triage Needs to be triaged and prioritised accordingly label Oct 11, 2021
@ukclivecox
Copy link
Contributor

Thanks for this. Will look into it for next release

@seldondev seldondev added the triage Needs to be triaged and prioritised accordingly label Oct 11, 2021 — with Board Genius Sync
@ukclivecox ukclivecox removed the triage Needs to be triaged and prioritised accordingly label Oct 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants