Can't start tilt with multi node minikube clusters using docker driver

[code-merc]

Expected Behavior

Running tilt up should work with a multi node cluster

Current Behavior

tilt up executes, but the tilt web interface logs that the tiltfile could not be executed with this error

Failed to connect to Docker: Could not read docker env from minikube.
Did you forget to run `minikube start`?
X Exiting due to ENV_MULTINODE_CONFLICT: The docker-env command is incompatible with multi-node clusters. Use the 'registry' add-on: https://minikube.sigs.k8s.io/docs/handbook/registry/

Steps to Reproduce

1. Create a minikube cluster like this minikube start --driver=docker --nodes=2 --addons=default-storageclass,registry,storage-provisioner
2. Run tilt up
3. See problem above

Context

tilt doctor Output

$ tilt doctor
Tilt: v0.20.0, built 2021-04-30
System: linux-amd64
---
Docker (cluster)
- Host: Error: Could not read docker env from minikube.
Did you forget to run `minikube start`?
X Exiting due to ENV_MULTINODE_CONFLICT: The docker-env command is incompatible with multi-node clusters. Use the 'registry' add-on: https://minikube.sigs.k8s.io/docs/handbook/registry/

---
Docker (local)
- Host: [default]
- Version: 1.40
- Builder: 2
---
Kubernetes
- Env: minikube
- Context: minikube
- Cluster Name: minikube
- Namespace: default
- Container Runtime: docker
- Version: v1.20.2
- Cluster Local Registry: none
---

I think the issue is here, running minikube docker-env on a multi node cluster with the docker driver fails with

❌  Exiting due to ENV_MULTINODE_CONFLICT: The docker-env command is incompatible with multi-node clusters. Use the 'registry' add-on: https://minikube.sigs.k8s.io/docs/handbook/registry/

Enabling/disabling the registry addon doesn't make a difference either.

[nicks]

Thanks for the report!

Ya, if you just need a quick workaround, you can set up a registry as described here:
https://minikube.sigs.k8s.io/docs/handbook/registry/#enabling-insecure-registries

and then you can apply a configmap to tell tilt how to find it, like so:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    host: "localhost:5000"
EOF

(we should probably ask minikube to add this to their guide, since it's in the other cluster guides: https://kind.sigs.k8s.io/docs/user/local-registry/#create-a-cluster-and-registry)

We maintain a tool called ctlptl that tries to do some of this wiring for you https://github.com/tilt-dev/ctlptl but it doesn't have an api for multi-node clusters yet

[code-merc]

So we currently set up our registry using that documentation, only thing I haven't done is patch the containerd runtime but it seems like the registry plugin in minikube does that for you. Looks like the issue is with tilt is still trying to get info about the minikube cluster by running minikube docker-env regardless of the registry setup, which will always fail on a multi node cluster.

Currently what we're doing is running the minikube registry addon, forwarding to it with either with kubectl or

docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"

This works just fine if I set --nodes=1 in minikube start but once I change it to 2 tilt won't continue execution of the tiltfile.

Unfortunately we have a requirement to use cilium, and we were running into some odd issues using ctlptl, cilium, and a multi node kind cluster so we decided to use minikube instead and manually configure the registry.

[code-merc]

Also, I tried applying that configmap and I still run into the issue

[nicks]

are you able to use minikube --container-runtime=containerd? that will turn off the docker-env entirely in minikube, so tilt won't try to connect to it

(edit: if you're using a multi-node cluster, there's no real benefit to using docker as the container runtime over containerd - docker is just "containerd + an image builder you can connect to" but you can't use that builder anyway)

[code-merc]

Using

minikube start --cni=cilium --driver=docker --nodes=2 --container-runtime=containerd --addons=default-storageclass,registry,storage-provisioner

seems to let tilt start and push images to the registry, but pods are failing with

Failed to pull image "localhost:5000/example-html-image:tilt-ac72080bbe84db77": rpc error: code = Unknown desc = failed to pull and unpack image "localhost:5000/example-html-image:tilt-ac72080bbe84db77": failed to resolve reference "localhost:5000/example-html-image:tilt-ac72080bbe84db77": failed to do request: Head http://localhost:5000/v2/example-html-image/manifests/tilt-ac72080bbe84db77: dial tcp 127.0.0.1:5000: connect: connection refused

Using docker and a single node + cilium that setup works fine, so seems like containerd and cilium aren't playing nice unfortunately.

(edit: let me see if multinode docker has any issues with local registries, I didn't actually confirm that)

[code-merc]

Ok looks like there's that same issue with just docker, I'll try disabling cilium and see if there's still an issue

[code-merc]

Alright looks like cilium likes to absolutely destroy my local docker networking, not sure what that's about but unrelated to tilt. After disabling cilium and using containerd, minikube works fine with multiple nodes. I can close this unless you want to leave it open for docs/a warning message when minikube docker-env fails.

[nicks]

ya, let's leave it open for the ENV_MULTINODE_CONFLICT issue, which we should fix either way.

wiring up a registry definitely has some land mines 😬

[code-merc]

By the way this same issue comes up when running with a VM driver so minikube start --driver=kvm2 --nodes=2 --addons=default-storageclass,registry,storage-provisioner causes tilt to error the same as with docker.

[nicks]

this should be fixed in 0.20.7