创建graphscope.session失败 #3

jmzhoulab · 2020-12-16T02:07:56Z

jmzhoulab
Dec 16, 2020

本地环境：CentOs7, python 3.6.5
按照安装流程操作：
第一步
./scripts/prepare_env.sh
没有报错：日志如下：

Requirement already satisfied: cachetools<5.0,>=2.0.0 in /data/zhoujm/anaconda3/lib/python3.6/site-packages (from google-auth>=1.0.1->kubernetes->graphscope) (4.2.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /data/zhoujm/anaconda3/lib/python3.6/site-packages (from google-auth>=1.0.1->kubernetes->graphscope) (4.6)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /data/zhoujm/anaconda3/lib/python3.6/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes->graphscope) (0.4.8)
Requirement already satisfied: decorator>=4.3.0 in /data/zhoujm/anaconda3/lib/python3.6/site-packages (from networkx->graphscope) (4.3.0)
Requirement already satisfied: pytz>=2011k in /data/zhoujm/anaconda3/lib/python3.6/site-packages (from pandas->graphscope) (2018.4)
Requirement already satisfied: idna<3,>=2.5 in /data/zhoujm/anaconda3/lib/python3.6/site-packages (from requests->kubernetes->graphscope) (2.6)
Requirement already satisfied: chardet<4,>=3.0.2 in /data/zhoujm/anaconda3/lib/python3.6/site-packages (from requests->kubernetes->graphscope) (3.0.4)
Requirement already satisfied: oauthlib>=3.0.0 in /data/zhoujm/anaconda3/lib/python3.6/site-packages (from requests-oauthlib->kubernetes->graphscope) (3.1.0)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  2 38.3M    2  901k    0     0   5105      0  2:11:20  0:03:00  2:08:20     0
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:01:06 --:--:--     0

第二步
graphscope客户端按照成功，但创建session失败，如下：

(graphscope) [zhoujm@ml0 GraphScope]$ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import graphscope
>>>
>>> sess = graphscope.session()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/zhoujm/anaconda3/lib/python3.6/site-packages/graphscope/client/session.py", line 319, in __init__
    self._proc, self._endpoint = self._connect()
  File "/data/zhoujm/anaconda3/lib/python3.6/site-packages/graphscope/client/session.py", line 573, in _connect
    api_client = kube_config.new_client_from_config()
  File "/data/zhoujm/anaconda3/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 848, in new_client_from_config
    persist_config=persist_config)
  File "/data/zhoujm/anaconda3/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 794, in load_kube_config
    persist_config=persist_config)
  File "/data/zhoujm/anaconda3/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 752, in _get_kube_config_loader
    'Invalid kube-config file. '
kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.
>>>

kubernetes.config 应该是如何配置的呢？上述错误如何解决

Answered by lidongze0629

Dec 17, 2020

sess = graphscope.session(show_log=True)

日志如下：

>>> sess = graphscope.session(show_log=True)
2020-12-17 12:27:05,570 [INFO][session:266]: Initializing graphscope session with parameters: {'num_workers': 2, 'log_level': 'info', 'show_log': True, 'k8s_namespace': None, 'k8s_gs_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:latest', 'k8s_etcd_image': 'quay.io/coreos/etcd:v3.4.13', 'k8s_image_pull_policy': 'IfNotPresent', 'k8s_image_pull_secrets': [], 'k8s_gie_graph_manager_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/maxgraph_standalone_manager:1.0', 'k8s_zookeeper_image': 'zookeeper:3.4.14', 'k8s_coordinator_cpu': 1.0, 'k8s_coordinator_mem': '4Gi', 'k8s_vineyard…

View full answer

sighingnow · 2020-12-16T02:43:45Z

sighingnow
Dec 16, 2020
Maintainer

From the logs we could see error occurs (curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104) when downloading kubectl and minikube, could you please verify if /usr/local/bin/kubectl and /usr/local/bin/minikube exist?

0 replies

siyuan0322 · 2020-12-16T02:48:36Z

siyuan0322
Dec 16, 2020
Maintainer

kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.

报错信息代表没有配置minikube环境。

curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

看起来是在下载 kubectl 或 minikube 的时候出错了。可以手动check一下这两个文件是否存在并且可用。

之后可以再次执行prepare_env.sh 来初始化minikube 配置，或者手动执行

GraphScope/scripts/prepare_env.sh

Lines 56 to 66 in e162f11

    
           export CHANGE_MINIKUBE_NONE_USER=true 
        
           echo "$(date '+%Y-%m-%d %H:%M:%S') launch k8s cluster" 
        
           sudo sysctl fs.protected_regular=0 || true 
        
           sudo minikube start --vm-driver=none --kubernetes-version="${K8S_VERSION}" 
        
           sudo cp -r /root/.kube /root/.minikube "${HOME}" || true 
        
           sudo chown -R "$(id -u)":"$(id -g)" "${HOME}"/.minikube || true 
        
           sudo chown -R "$(id -u)":"$(id -g)" "${HOME}"/.kube || true 
        
           sed -i 's@/root@'"${HOME}"'@g' "${HOME}"/.kube/config || true 
        
           minikube update-context

0 replies

jmzhoulab · 2020-12-16T06:16:08Z

jmzhoulab
Dec 16, 2020
Author

重新准备环境现在这个错误不出现了，但是出现了另外一个问题：

>>> sess = graphscope.session()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/zhoujm/anaconda3/lib/python3.6/site-packages/graphscope/client/session.py", line 319, in __init__
    self._proc, self._endpoint = self._connect()
  File "/data/zhoujm/anaconda3/lib/python3.6/site-packages/graphscope/client/session.py", line 619, in _connect
    enable_k8s=self._config_params["enable_k8s"],
  File "/data/zhoujm/anaconda3/lib/python3.6/site-packages/graphscope/client/rpc.py", line 120, in waiting_service_ready
    raise ConnectionError("Connect coordinator timeout, {}".format(msg))
graphscope.framework.errors.ConnectionError: 'Connect coordinator timeout, grpc connnect failed.'

gprc 连接失败

去查看/var/log/messages日志，提示：The image pull may not succeed，详细日志如下

Dec 16 14:12:23 ml0 sslocal: 2020-12-16 14:12:23 INFO     connecting 192.168.1.21:30786 from 127.0.0.1:57346
Dec 16 14:12:48 ml0 kubelet: W1216 14:12:48.018525  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-etcd-pizpoz-6fbd855fc-ndgl5 due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:12:56 ml0 kubelet: W1216 14:12:56.018560  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-engine-ccyzlp-bs9lz due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:12:58 ml0 sslocal: 2020-12-16 14:12:58 INFO     connecting 192.168.1.21:30786 from 127.0.0.1:57684
Dec 16 14:13:03 ml0 kubelet: W1216 14:13:03.019063  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-graphmanager-ohqxhd-5d67b678f8-5t6xz due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:13:12 ml0 kubelet: W1216 14:13:12.018288  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-engine-ccyzlp-snjkz due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:13:59 ml0 kubelet: W1216 14:13:59.018385  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-engine-ccyzlp-bs9lz due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:14:04 ml0 kubelet: W1216 14:14:04.018583  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-graphmanager-ohqxhd-5d67b678f8-5t6xz due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:14:05 ml0 sslocal: 2020-12-16 14:14:05 INFO     connecting 192.168.1.21:30786 from 127.0.0.1:58320
Dec 16 14:14:05 ml0 sslocal: 2020-12-16 14:14:05 INFO     connecting 192.168.1.21:30290 from 127.0.0.1:58322
Dec 16 14:14:17 ml0 kubelet: W1216 14:14:17.018540  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-engine-ccyzlp-snjkz due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:14:17 ml0 kubelet: W1216 14:14:17.018601  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-etcd-pizpoz-6fbd855fc-ndgl5 due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:14:49 ml0 systemd-logind: New session 57760 of user root.
Dec 16 14:14:49 ml0 systemd: Started Session 57760 of user root.
Dec 16 14:14:49 ml0 systemd: Starting Session 57760 of user root.
Dec 16 14:14:49 ml0 dbus[1123]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Dec 16 14:14:49 ml0 dbus-daemon: dbus[1123]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Dec 16 14:14:49 ml0 dbus[1123]: [system] Successfully activated service 'org.freedesktop.problems'
Dec 16 14:14:49 ml0 dbus-daemon: dbus[1123]: [system] Successfully activated service 'org.freedesktop.problems'
Dec 16 14:14:58 ml0 sslocal: 2020-12-16 14:14:58 INFO     connecting 192.168.1.21:8443 from 127.0.0.1:58812
Dec 16 14:15:01 ml0 kubelet: W1216 14:15:01.018589  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-engine-ccyzlp-bs9lz due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:15:08 ml0 kubelet: W1216 14:15:08.018354  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-graphmanager-ohqxhd-5d67b678f8-5t6xz due to secret "\"\"" not found.  The image pull may not succeed.
Dec 16 14:15:08 ml0 sslocal: 2020-12-16 14:15:08 INFO     connecting 192.168.1.21:8443 from 127.0.0.1:58912
Dec 16 14:15:18 ml0 sslocal: 2020-12-16 14:15:18 INFO     connecting 192.168.1.21:8443 from 127.0.0.1:59010
Dec 16 14:15:28 ml0 sslocal: 2020-12-16 14:15:28 INFO     connecting 192.168.1.21:8443 from 127.0.0.1:59108
Dec 16 14:15:31 ml0 kubelet: W1216 14:15:31.018569  190895 kubelet_pods.go:880] Unable to retrieve pull secret gs-mlkiox/"" for gs-mlkiox/gs-engine-ccyzlp-snjkz due to secret "\"\"" not found.  The image pull may not succeed.

0 replies

siyuan0322 · 2020-12-16T06:23:58Z

siyuan0322
Dec 16, 2020
Maintainer

sess = graphscope.session()

能请您这边加入一个参数 show_log=True 再重新运行一下, 然后贴一下 LOG 吗？

sess = graphscope.session(show_log=True)

0 replies

lidongze0629 · 2020-12-16T08:46:45Z

lidongze0629
Dec 16, 2020
Maintainer

这边在 centos 7.8 + python3.6.5 + minikube1.15.1 (start with vm-driver=none) 环境下没有复现该问题。

但是从grpc连接失败上看，像是k8s集群暴露的service没有被client端访问到，所以麻烦您提供一下k8s集群的部署方式。

0 replies

jmzhoulab · 2020-12-17T04:32:16Z

jmzhoulab
Dec 17, 2020
Author

sess = graphscope.session(show_log=True)

日志如下：

>>> sess = graphscope.session(show_log=True)
2020-12-17 12:27:05,570 [INFO][session:266]: Initializing graphscope session with parameters: {'num_workers': 2, 'log_level': 'info', 'show_log': True, 'k8s_namespace': None, 'k8s_gs_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:latest', 'k8s_etcd_image': 'quay.io/coreos/etcd:v3.4.13', 'k8s_image_pull_policy': 'IfNotPresent', 'k8s_image_pull_secrets': [], 'k8s_gie_graph_manager_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/maxgraph_standalone_manager:1.0', 'k8s_zookeeper_image': 'zookeeper:3.4.14', 'k8s_coordinator_cpu': 1.0, 'k8s_coordinator_mem': '4Gi', 'k8s_vineyard_cpu': 0.5, 'k8s_vineyard_mem': '512Mi', 'k8s_vineyard_shared_mem': '4Gi', 'k8s_engine_cpu': 0.5, 'k8s_engine_mem': '4Gi', 'k8s_waiting_for_delete': False, 'timeout_seconds': 600}
2020-12-17 12:27:06,719 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Successfully assigned gs-sjwwiu/coordinator-ohkvpn-99d87dc87-wfv2c to ml0.sz-local.haizhi.com
2020-12-17 12:27:07,365 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Container image "registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:latest" already present on machine
2020-12-17 12:27:07,395 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Created container coordinator
2020-12-17 12:27:07,489 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Started container coordinator
2020-12-17 12:27:18,890 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Readiness probe failed: dial tcp 172.17.0.3:59631: connect: connection refused
2020-12-17 12:27:35,208 [INFO][cluster:484]: Coordinator service is ready.

后续就没有日志了

0 replies

jmzhoulab · 2020-12-17T06:10:14Z

jmzhoulab
Dec 17, 2020
Author

这边在 centos 7.8 + python3.6.5 + minikube1.15.1 (start with vm-driver=none) 环境下没有复现该问题。

但是从grpc连接失败上看，像是k8s集群暴露的service没有被client端访问到，所以麻烦您提供一下k8s集群的部署方式。

按照脚本执行的，应该是docker

0 replies

lidongze0629 · 2020-12-17T06:30:43Z

lidongze0629
Dec 17, 2020
Maintainer

sess = graphscope.session(show_log=True)

日志如下：

>>> sess = graphscope.session(show_log=True)
2020-12-17 12:27:05,570 [INFO][session:266]: Initializing graphscope session with parameters: {'num_workers': 2, 'log_level': 'info', 'show_log': True, 'k8s_namespace': None, 'k8s_gs_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:latest', 'k8s_etcd_image': 'quay.io/coreos/etcd:v3.4.13', 'k8s_image_pull_policy': 'IfNotPresent', 'k8s_image_pull_secrets': [], 'k8s_gie_graph_manager_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/maxgraph_standalone_manager:1.0', 'k8s_zookeeper_image': 'zookeeper:3.4.14', 'k8s_coordinator_cpu': 1.0, 'k8s_coordinator_mem': '4Gi', 'k8s_vineyard_cpu': 0.5, 'k8s_vineyard_mem': '512Mi', 'k8s_vineyard_shared_mem': '4Gi', 'k8s_engine_cpu': 0.5, 'k8s_engine_mem': '4Gi', 'k8s_waiting_for_delete': False, 'timeout_seconds': 600}
2020-12-17 12:27:06,719 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Successfully assigned gs-sjwwiu/coordinator-ohkvpn-99d87dc87-wfv2c to ml0.sz-local.haizhi.com
2020-12-17 12:27:07,365 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Container image "registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:latest" already present on machine
2020-12-17 12:27:07,395 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Created container coordinator
2020-12-17 12:27:07,489 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Started container coordinator
2020-12-17 12:27:18,890 [INFO][cluster:462]: [coordinator-ohkvpn-99d87dc87-wfv2c]: Readiness probe failed: dial tcp 172.17.0.3:59631: connect: connection refused
2020-12-17 12:27:35,208 [INFO][cluster:484]: Coordinator service is ready.

后续就没有日志了

Coordinator service is ready. 代表目前GraphScope集群已经成功拉起，后续会有client端与coordinator的连接日志，或超时报错等。如果连接成功，完整的日志应该如下:

0 replies

sighingnow · 2020-12-21T05:37:24Z

sighingnow
Dec 21, 2020
Maintainer

Closed since this issue requires further information from the user. Feel free to reopen if you have further questions.

7 replies

MultiChen Feb 8, 2021

This is the log output after following your suggestion, stuck here for a long time and there is no more

sighingnow Feb 8, 2021
Maintainer

Hi @MultiChen,

Thanks for providing the feedback here.

I have no extra information about your kubernetes cluster settings. The log says "1 insufficient cpu". There seems no enough resources to launch a graphscope session.

Note that we have announce GraphScope Playground to help users for a quick get started, without any extra prerequisite about kubernetes. We encourge you to try graphscope in the playground notebooks!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

创建graphscope.session失败 #3

{{title}}

Replies: 9 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

创建graphscope.session失败 #3

jmzhoulab Dec 16, 2020

Replies: 9 comments · 7 replies

sighingnow Dec 16, 2020 Maintainer

siyuan0322 Dec 16, 2020 Maintainer

jmzhoulab Dec 16, 2020 Author

siyuan0322 Dec 16, 2020 Maintainer

lidongze0629 Dec 16, 2020 Maintainer

jmzhoulab Dec 17, 2020 Author

jmzhoulab Dec 17, 2020 Author

lidongze0629 Dec 17, 2020 Maintainer

sighingnow Dec 21, 2020 Maintainer

MultiChen Feb 8, 2021

sighingnow Feb 8, 2021 Maintainer

MultiChen Feb 9, 2021

dakeleblack May 24, 2021

dakeleblack May 24, 2021

jmzhoulab
Dec 16, 2020

Replies: 9 comments 7 replies

sighingnow
Dec 16, 2020
Maintainer

siyuan0322
Dec 16, 2020
Maintainer

jmzhoulab
Dec 16, 2020
Author

siyuan0322
Dec 16, 2020
Maintainer

lidongze0629
Dec 16, 2020
Maintainer

jmzhoulab
Dec 17, 2020
Author

jmzhoulab
Dec 17, 2020
Author

lidongze0629
Dec 17, 2020
Maintainer

sighingnow
Dec 21, 2020
Maintainer

sighingnow Feb 8, 2021
Maintainer