Internal error occurred: failed calling webhook

[gosandhya]

My seldon deployment appears when I check:
kubectl get sdep but not when
kubectl get deployments

Also, Nothing gets returned when I check the status of the deployment:

kubectl get seldondeployments.machinelearning.seldon.io seldon-model -n seldon-model -o jsonpath='{.status.state}'

Also, there are no pods allocated to the deployment.

Further investigating seldon-core operator controller manager pod, I found this error message:

.....{"level":"error","ts":1568906205.4712648,"logger":"kubebuilder.controller","msg":" *Reconciler error* ","controller":"seldondeployment-controller","request":"seldon-model/seldon-model","error":" *Internal error occurred: failed calling webhook* \"kfservice.kfserving-webhook-server.deployment-mutator\": Post https://kfserving-webhook-server-service.kubeflow.svc:443/mutate-deployments?timeout=30s: service \"kfserving-webhook-server-service\" not found","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error......."}

The same deployment works okay on my local machine. So, I do not think there is issue in yaml file. But this server does have kubeflow installed. I installed seldon-core separately using Helm.

The deployment seem to be looking for kubeflow component kfserving-webhook. Is there a way to create a deployment without it interacting with kubeflow?

[elouayas]

Can you try kubectl describe sdep , you can see if what is the deployment failed or succeed there.

[ryandawsonuk]

I think the error relates to the webhook for KFServing. It checks all Pods that get deployed, including Pods created by the Seldon operator (and any other operator too). I guess this is a kubeflow v0.6 install?

kubectl logs kfserving-controller-manager-0 -n kubeflow manager or kubectl get pods -n kubeflow might help pin down what is wrong with it.

[gosandhya]

@ryandawsonuk I have only these pods in kubeflow namespace

kubeflow               admission-webhook-bootstrap-stateful-set-0     1/1     Running   0          4d22h
kubeflow               admission-webhook-deployment-b77bd65c5-z4bnl   1/1     Running   0          4d22h
kubeflow               application-controller-stateful-set-0          1/1     Running   0          4d22h
kubeflow               argo-ui-6db54c878-ww85g                        1/1     Running   0          4d22h
kubeflow               centraldashboard-559bc9fb6d-bw2nz              1/1     Running   0          4d22h
kubeflow               metacontroller-0                               1/1     Running   0          4d22h
kubeflow               workflow-controller-5b4764bc47-jgbvg           1/1     Running   0          4d22h

and I installed seldon-core in separate workspace:
seldon-core seldon-operator-controller-manager-0 1/1 Running 0 3d18h

[gosandhya]

@elouayas I do not see any failed or success status!

Name:         seldon-model
Namespace:    seldon-model
Labels:       app=seldon
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"machinelearning.seldon.io/v1alpha2","kind":"SeldonDeployment","metadata":{"annotations":{},"labels":{"app":"seldon"},"name"...
API Version:  machinelearning.seldon.io/v1alpha3
Kind:         SeldonDeployment
Metadata:
  Creation Timestamp:  2019-09-19T20:24:29Z
  Generation:          1
  Resource Version:    294796
  Self Link:           /apis/machinelearning.seldon.io/v1alpha3/namespaces/seldon-model/seldondeployments/seldon-model
  UID:                 34124283-1bc4-48cc-8c32-cbe724c46b0f
Spec:
  Annotations:
    deployment_version:  v1
    project_name:        seldon-model
  Name:                  seldon-model
  oauth_key:             oauth-key
  oauth_secret:          oauth-secret
  Predictors:
    Annotations:
      predictor_version:  v1
    Component Specs:
      Spec:
        Containers:
          Image:              seldon-model:1.0
          Image Pull Policy:  IfNotPresent
          Name:               seldon-model
          Resources:
            Requests:
              Memory:  20Mi
          Volume Mounts:
            Mount Path:                    /etc/podinfo
            Name:                          podinfo
        Termination Grace Period Seconds:  20
        Volumes:
          Downward API:
            Default Mode:  420
            Items:
              Field Ref:
                API Version:  v1
                Field Path:   metadata.annotations
              Path:           annotations
          Name:               podinfo
    Graph:
      Children:
      Endpoint:
        service_host:  localhost
        service_port:  9000
        Type:          REST
      Implementation:  UNKNOWN_IMPLEMENTATION
      Name:            seldon-model
      Type:            MODEL
    Labels:
      Version:  single-model
    Name:       single-model
    Replicas:   1
Events:         <none>

[elouayas]

You should have something like that

Status:
  Predictor Status:
    Name:                seldon-deployment-single-model-2123895
    Replicas:            1
    Replicas Available:  1
  State:                 Available
Events:                  <none>

I think something is wrong in your yaml definition

[ryandawsonuk]

Could you check kubectl get mutatingwebhookconfigurations? I think you might be hitting something I hit before where the kfserving webhook was invoked even when I removed kfserving

[gosandhya]

@ryandawsonuk I get this:

admission-webhook-mutating-webhook-configuration   2019-09-18T14:45:30Z
experiment-mutating-webhook-config                 2019-09-18T14:45:37Z
istio-sidecar-injector                             2019-09-18T14:45:28Z
kfservice.serving.kubeflow.org                     2019-09-18T14:46:22Z
mutating-webhook-configuration                     2019-09-18T14:45:43Z

Can you explain, why did you need to remove kfserving?

My deployment only needs to use Seldon Operator really and not kubeflow. Is there any way to separate those two out?

[ryandawsonuk]

The error getting reported is "failed calling webhook* "kfservice.kfserving-webhook-server.deployment-mutator". It may be reported by seldon but it's not actually coming from Seldon. The seldon operator is just trying to create resources and they're failing because the kfserving webhook intercepts the requests to the kubernetes API and tries to modify the resources.

[gosandhya]

It was to do with kubeflow installation - It was not done properly. Some components were not up. Uninstalling it helped getting rid of that issue - at the very least.
Thank you everyone!

[ukclivecox]

Glad it fixed!

[cyrilthank]

Hi Sandhya @SandhyaaGopchandani
can you please share the version of kubeflow you installed and the version of kubernetes
requesting since I am grounded on this issue
kubeflow/kubeflow#4228
kubeflow/kubeflow#4184 (comment)