Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SeldonDeployment 的pod没有ready #189

Open
gongyu143 opened this issue Sep 30, 2021 · 1 comment
Open

SeldonDeployment 的pod没有ready #189

gongyu143 opened this issue Sep 30, 2021 · 1 comment
Labels
question Further information is requested

Comments

@gongyu143
Copy link

gongyu143 commented Sep 30, 2021

[root@xxxx ormb]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/tfserving-default-0-mnist-model-6cdfd5d-6vncx 0/2 Running 0 108m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 443/TCP 3d21h
service/tfserving-default ClusterIP 10.102.59.194 8000/TCP,5001/TCP 109m
service/tfserving-default-mnist-model ClusterIP 10.109.93.101 9000/TCP 109m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tfserving-default-0-mnist-model 0/1 1 0 108m

NAME DESIRED CURRENT READY AGE
replicaset.apps/tfserving-default-0-mnist-model-6cdfd5d 1 1 0 108m

kubectl describe pod/tfserving-default-0-mnist-model-6cdfd5d-6vncx

Events:
Type Reason Age From Message


Warning Unhealthy 40s (x1699 over 4h18m) kubelet Liveness probe errored: strconv.Atoi: parsing "http": invalid syntax

kubectl logs -f tfserving-default-0-mnist-model-6cdfd5d-6vncx mnist-model

2021-09-30 06:10:13.660649: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:268] No versions of servable mnist-model found under base path /mnt/models. Did you forget to name your leaf directory as a number (eg. '/1/')?
2021-09-30 06:10:14.660861: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:268] No versions of servable mnist-model found under base path /mnt/models. Did you forget to name your leaf directory as a number (eg. '/1/')?
2021-09-30 06:10:15.662074: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:268] No versions of servable mnist-model found under base path /mnt/models. Did you forget to name your leaf directory as a number (eg. '/1/')?
2021-09-30 06:10:16.662489: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:268] No versions of servable mnist-model found under base path /mnt/models. Did you forget to name your leaf directory as a number (eg. '/1/')?

kubectl logs -f tfserving-default-0-mnist-model-6cdfd5d-6vncx mnist-model-model-initializer

Using / as the root path
Logging to the remote registry demo.goharbor.io
Username: ormbtest
Login insecurely
time="2021-09-30T01:55:43Z" level=debug msg="attempting v2 login to registry endpoint https://demo.goharbor.io/v2/"
Login succeeded
Using / as the root path
v1: Pulling from demo.goharbor.io/tensorflow/fashion_model
ref: demo.goharbor.io/tensorflow/fashion_model:v1
digest: 62c7d273602a9dc6833a1401e0186f50708e1e03b717edc4b0b39781b53c367a
size: 162.1 KiB
format: SavedModel
Status: Downloaded newer model for demo.goharbor.io/tensorflow/fashion_model:v1
ref: demo.goharbor.io/tensorflow/fashion_model:v1
digest: 62c7d273602a9dc6833a1401e0186f50708e1e03b717edc4b0b39781b53c367a
size: 162.1 KiB
format: SavedModel
Moving /mnt/models/model/saved_model.pb to /mnt/models/saved_model.pb
Moving /mnt/models/model/variables to /mnt/models/variables

有没有大神知道怎么解决

@gaocegege
Copy link
Member

Maybe you can attach to the container and find if there is a model downloaded.

@gaocegege gaocegege added the question Further information is requested label Nov 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants