Failed to enable kubeflow #958
Comments
|
I'm having a similar problem enabling kubeflow, slightly different error |
|
suggested but that did not help |
|
My error message looks like this: Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 2 |
|
I think this is the same as #943 |
|
@titsuki I think the problem you are facing is slow internet connection. If I read the curl output correctly you are downloading with 71k and the juju client is 116M so curl at some point gave up. @mattk08 are you behind a firewall/proxy? For some reason curl failed to initiate the ssl handshake and download the juju client. @andrewcheny in your case the juju client is available but the bootstrap failed. Could you share the |
@ktsakalozos I am behind a firewall (no proxy), however, this is the first issue I've had downloading any packages with any package manager. I was able to find the URL curl was using in enable.juju.sh I was able to download the tar file fine outside of the script. I went ahead and just ran each command in the enable.juju.sh script by hand to enable juju and then tried running |
|
@ktsakalozos seems I see the same problem, and it only affects ubuntu desktop for whatever reason. $ microk8s.kubectl get nodes $ microk8s.kubectl get po -A $ microk8s.inspect WARNING: Docker is installed. |
|
The issue starts happening when rbac is enabled. In this case I can't spin up pods with "mountvolume.setup failed for volume "default-token-XXX": failed to sync secret cache: timed out waiting for the condition" error. |
|
@grebennikov could you help me reproduce this? When you say "it only affects ubuntu desktop" what do you mean? If you |
|
@ktsakalozos Thanks for your advice! |
Would you be able to manualy run the commands found in the |
|
@ktsakalozos |
|
@ktsakalozos I could observe the same error as |
|
You could |
|
@ktsakalozos Your idea works fine to me. snap seems 30~40 times faster than launchpad w.r.t. downloading speed. |
|
@ktsakalozos This also works to enable juju. But when I then try to enable Kubeflow, it gives me the following message. |
|
I am also facing the "Problem with the SSL CA cert" issue. It seems curl from snapd has different behavior than curl in OS. I sniffed the connection and curl-snapd has a FYN,ACK after the TCP handshake, while curl-OS issues a TLS1.2 Client Hello. No idea why.
|
|
@dgrahn I faced the same error but this error itself could solve by changing the permission.
|
|
@ktsakalozos Thanks for your advice. Fortunately, I could make juju enabled (by trying |
|
I follow the kubeflow document to install kubeflow with Microk8s: But with this error messages: |
|
@dgrahn: Running these commands should fix the issue you're facing with the permission denied error: https://github.com/ubuntu/microk8s/blob/050e98b/snap/hooks/configure#L380-L382 Where Additionally, #989 will fix this issue generally, as it just includes the juju binary in the snap, instead of downloading it after snap install. |
|
@JIElite: I haven't seen that error before. Can you create a new issue with the output from |
|
Same problem here on centos 7.7. Installed via snap: Getting the error: |
|
@egkiastas: Can you download https://launchpad.net/juju/2.7/2.7.3/+download/juju-2.7.3-k8s.tar.xz, and place the unzipped https://github.com/ubuntu/microk8s/blob/0ebbaed/snap/hooks/configure#L378-L385 #989 will fix the issue you're running into by including the juju binary in the snap instead of downloading it. If you'd like to try that instead of the above, you should be able to by switching microk8s to the |
I'm getting the same error as OP and I followed this, but it didn't help. Here's my `sudo microk8s.enable kubeflow Enabling dns... Enabling storage... Enabling dashboard... Enabling ingress... Enabling rbac... Enabling juju... Deploying Kubeflow... Kubeflow could not be enabled: ERROR microk8s: running: false Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit Failed to enable kubeflow` |
|
Hi. I'm very new to microk8s and kubernetes, but I'm afraid I'm getting a similar problem when trying to enable kubeflow, except with a slightly different error printout at the bottom. This happens when I try to enable:
I've made several attempts, and the charm that it could not add is different every time. For example, one time I tried to enable, the charm was "cs:~kubeflow-charmers/argo-controller-162". Furthermore, the 'Deploying Kubeflow' part takes about 10 minutes. Is there a verbose mode while running a microk8s enable action, so I can see what 'Deploying Kubeflow...' gets stuck on? Are there any commands in the enable.kubeflow action file I can run from the terminal to help diagnose this problem? Thanks for any help you can give, I'll also attach my Inspector file, which I created after this failure. EDIT: |
|
@zshenkle: For future reference, you can enable more verbose logging like this: As far as the nameserver issue you ran into, I'm not sure why you would have to edit those in. Did you put those in https://microk8s.io/docs/addon-dns cc @ktsakalozos ^^ |
Well if behind proxy, what should I set for DNS SERVER? |
@knkski thanks so much for that verbose logging tip! The machine I'm setting up microk8s on is a relatively new machine, and we apparently hadn't gotten around to adding all the necessary nameservers (IE: 8.8.8.8 and 8.8.4.4). And yes we effectively put those in |
|
I also met this issue like this Here my And the Any help would be greatly appreciated. |
|
@kawa23, can you attach the |
|
@kawa23 I have the exact same problem. Could it be the case that my Kubernetes version is not supported yet? It is 1.18, while Kubeflow is on alpha status with Microk8s, and 1.18 kubernetes version is not even mentioned on https://www.kubeflow.org/docs/started/k8s/overview/#minimum-system-requirements. |
|
@MathewStylianidis: It shouldn't be an issue with 1.18. Would you be able to run |
|
@knkski @markshuttle Sorry for late reply, I have installed kubeflow based on k8s cluster with multipass, and removed the microk8s, so I can't get |
|
I tried to install Everything works up to here, but fails with this error: Any idea what is causing it? |
|
Looks like a networking issue - it needs to be able to retrieve the charms from jujucharms.com. I think we should do some connectivity tests before we launch into the orchestration of kubeflow, so we catch and warn more appropriately if that is going to fail as it has here. |
But I can ping jujucharms.com from the same server. |
|
@atamahjoubfar could you share the output of: |
It fails at |
|
@atamahjoubfar: That looks like transient network issues. Does it work if you try it again? |
I tried again, and unfortunately, the problem persists. Is there any way to manually download the charms and enable |
|
@atamahjoubfar, @markshuttle: I've opened up #1296 to do a basic network connectivity check before enabling Kubeflow @atamahjoubfar: If you follow the installation instructions in the README here for deploying the Kubeflow bundle instead of via https://github.com/juju-solutions/bundle-kubeflow/ |
|
@kawa23, @MathewStylianidis: I believe the issue that you're facing is due to your hostname having capital letters, see this error message: mai 11 15:06:58 SESTVSDV0002 microk8s.daemon-kubelet[24507]: E0511 15:06:58.330800 24507 csi_plugin.go:271] Failed to initialize CSINodeInfo: error updating CSINode annotation: timed out waiting for the condition; caused by: csinodes.storage.k8s.io "sestvsdv0002" is forbidden: User "system:node:SESTVSDV0002" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope: can only access CSINode with the same name as the requesting node If you change your hostname to only have lowercase letters, does it work for you? |
|
@knkski @markshuttle: I tried to follow these instructions: I installed the However, from the same server, the following I can ping |
|
@atamahjoubfar: Can you paste the output from this command? |
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
log . {
class error
}
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . 8.8.8.8 8.8.4.4
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"Corefile":".:53 {\n errors\n health {\n lameduck 5s\n }\n ready\n log . {\n class error\n }\n kubernetes cluster.local in-addr.arpa ip6.arpa {\n pods insecure\n fallthrough in-addr.arpa ip6.arpa\n }\n prometheus :9153\n forward . 8.8.8.8 8.8.4.4\n cache 30\n loop\n reload\n loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}}
creationTimestamp: "2020-06-23T22:20:52Z"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
k8s-app: kube-dns
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:Corefile: {}
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:addonmanager.kubernetes.io/mode: {}
f:k8s-app: {}
manager: kubectl
operation: Update
time: "2020-06-23T22:20:52Z"
name: coredns
namespace: kube-system
resourceVersion: "1124"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 588deb7d-bf61-4096-a553-da7696eea724Does this help to identify the issue? Thank you. |
|
Got same error on Ubuntu 20.04 Desktop. I am not behind any firewall/proxy and can manually fetch https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-104/archive?channel=stable which downloads ambassador.zip (2.6 MB). I do not face this error on Unbutu 20.04 Server (in AWS) Tried with both
Output of Errors from coredns pod |
|
@knkski Thanks. Was actually referring to post from @atamahjoubfar on Jun 10th. i got the same error. |
|
I think all of the issues in this thread have been resolved. If not, feel free to open up a new issue |
and others
inspection-report-20200211_171636.tar.gz
Maybe this issue is same as #943 but I'm a microk8s newbie and cannot make a
correct judgement about it.
The text was updated successfully, but these errors were encountered: