minikube on Apple Silicon / M1 CPU (darwin-arm64) #9224

medyagh · 2020-09-11T21:00:31Z

Place holder to document all the issues

minikube binary for Darwin (amd64) works but no driver I installed.

med@meds-mac g1 % ./minikube-darwin-amd64 start
😄  minikube v1.13.0 on Darwin 11.0
👎  Unable to pick a default driver. Here is what was considered, in preference order:
    ▪ hyperkit: Not installed: exec: "hyperkit": executable file not found in $PATH
    ▪ parallels: Not installed: exec: "docker-machine-driver-parallels": executable file not found in $PATH
    ▪ podman: Not installed: exec: "podman": executable file not found in $PATH
    ▪ virtualbox: Not installed: unable to find VBoxManage in $PATH
    ▪ vmware: Not installed: exec: "docker-machine-driver-vmware": executable file not found in $PATH
    ▪ vmwarefusion: Not installed: vmrun path check: exec: "vmrun": executable file not found in $PATH
    ▪ docker: Not installed: exec: "docker": executable file not found in $PATH

❌  Exiting due to DRV_NOT_DETECTED: No possible driver was detected. Try specifying --driver, or see https://minikube.sigs.k8s.io/docs/start/

Docker can not be installed

Docker Desktop can not be installed on the CPU Incompatible CPU detected

hyper kit can not be installed

med@meds-mac g1 % brew install hyperkit
Error: dlopen(/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle, 0x0009): symbol '_ffi_prep_closure' not found, expected in flat namespace by '/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle' - /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fiddle.rb:2:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/linkage_checker.rb:6:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/formula.rb:20:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/formula_versions.rb:3:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/cmd/update-report.rb:3:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/utils.rb:85:in `require?'
/usr/local/Homebrew/Library/Homebrew/commands.rb:33:in `valid_internal_cmd?'
/usr/local/Homebrew/Library/Homebrew/brew.rb:85:in `'
Error: dlopen(/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle, 0x0009): symbol '_ffi_prep_closure' not found, expected in flat namespace by '/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle' - /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fiddle.rb:2:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/linkage_checker.rb:6:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/formula.rb:20:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/formula_installer.rb:4:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/cmd/install.rb:4:in `'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/utils.rb:85:in `require?'
/usr/local/Homebrew/Library/Homebrew/commands.rb:33:in `valid_internal_cmd?'
/usr/local/Homebrew/Library/Homebrew/brew.rb:85:in `'

The text was updated successfully, but these errors were encountered:

afbjorklund · 2020-09-12T08:21:06Z

It seems like VirtualBox is unlikely to be available: https://forums.virtualbox.org/viewtopic.php?f=8&t=98742

afbjorklund · 2020-09-12T08:26:28Z

Parallells Desktop ($80) has a prototype available: https://www.parallels.com/blogs/apple-silicon-wwdc/

afbjorklund · 2020-09-12T08:31:16Z

For the linux-arm64 platform (like Raspberry Pi 4), the available minikube options are libvirt (kvm2) and native (none).

We still have to add proper arm64 support to CI, though... Currently only amd64 is being tested. #9205 #6280

afbjorklund · 2020-09-12T10:15:04Z

Eventually we should also build a KIC base docker image for arm64. KIND is also working on this: kubernetes-sigs/kind#166

https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/

https://hub.docker.com/r/arm64v8/ubuntu/

Thanks to QEMU, you can do this on your regular Linux laptop already:

$ arch
x86_64
$ docker run -it arm64v8/ubuntu:focal
root@9937e658add6:/# arch
aarch64

(It uses binfmt_misc and /usr/bin/qemu-aarch64-static under the hood)

That way, once Docker Desktop is done with their VM (LinuxKit) it could run the kicbase arm64 image: docker/for-mac#4733

goafabric · 2020-12-06T12:34:26Z

Just to let you know,
Minikube already works on Apple Silicon M1 !.
Together with VFTool https://github.com/evansm7/vftool
And a Ubuntu 20 Iso Image, you just can install Docker via apt.
Than snatch Minikube ARM64 which is surprisingly there via

curl sudo curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-arm64
sudo curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/arm64/kubectl

Drawback you have to work with the None Driver
sudo sysctl fs.protected_regular=0
sudo minikube start --driver=none

Be sure to start Vftool with 2 CPUs, not less not more ...
And that's it.. just works

afbjorklund · 2020-12-06T14:11:57Z

Minikube already works on Apple Silicon M1 !.

This issue was actually about darwin (macOS), not so much about the actual hardware (arm64)

It is not so surprising with the arm64 support, and we will have support for more drivers soon...

See #9762 for more ARM links

Together with VFTool https://github.com/evansm7/vftool

Glad to hear that it works, if this has support for libvirt or something we could make it an alternative ?

Currently we are using Docker's HyperKit, and they have been struggling a bit with the ARM support

And no VirtualBox (it's x86 only)

Using QEMU works (on all platforms), but it has some issues with setting up the required networking.

The default user network "works", but minikube is built on a more regular networking model for the VM.

We could also use the "generic" driver, and then you can use any VM - just provide it with the SSH:

See #4733

medyagh · 2020-12-24T01:53:43Z

@goafabric Glad to see none driver working with vftool (nice discovery)

update:
for docker driver I installed the Preview Docker for M1 and it doesnt work for me docker/for-mac#5185

tstromberg · 2021-01-09T00:18:20Z

With #10115 - the Docker driver will at least work on macOS/arm64 - it will also pave the way for other hypervisors to work, though we'll also need to port the ISO to arm64 (#9228).

We'll also need to add this arch to our release scripts: #10116

sayboras · 2021-01-23T06:02:44Z

Thanks all for your awesome work, I just try to run minikube in apple silicon, it's working great with docker driver 💯

$ go version
go version go1.16beta1 darwin/arm64

$ minikube version
minikube version: v1.17.0
commit: 94a676a6717732052a7ed96ef00c6f1943af1497

$ kall
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-74ff55c5b-xlr2n            1/1     Running   0          68s
kube-system   pod/etcd-minikube                      1/1     Running   0          81s
kube-system   pod/kube-apiserver-minikube            1/1     Running   0          81s
kube-system   pod/kube-controller-manager-minikube   1/1     Running   0          81s
kube-system   pod/kube-proxy-2gfq5                   1/1     Running   0          68s
kube-system   pod/kube-scheduler-minikube            1/1     Running   0          81s
kube-system   pod/storage-provisioner                1/1     Running   2          75s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  85s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   82s

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   82s

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   1/1     1            1           83s

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-74ff55c5b   1         1         1       68s

smithaitufe · 2021-01-23T15:38:46Z

Thanks all for your awesome work, I just try to run minikube in apple silicon, it's working great with docker driver 💯

$ go version
go version go1.16beta1 darwin/arm64

$ minikube version
minikube version: v1.17.0
commit: 94a676a6717732052a7ed96ef00c6f1943af1497

$ kall
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-74ff55c5b-xlr2n            1/1     Running   0          68s
kube-system   pod/etcd-minikube                      1/1     Running   0          81s
kube-system   pod/kube-apiserver-minikube            1/1     Running   0          81s
kube-system   pod/kube-controller-manager-minikube   1/1     Running   0          81s
kube-system   pod/kube-proxy-2gfq5                   1/1     Running   0          68s
kube-system   pod/kube-scheduler-minikube            1/1     Running   0          81s
kube-system   pod/storage-provisioner                1/1     Running   2          75s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  85s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   82s

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   82s

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   1/1     1            1           83s

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-74ff55c5b   1         1         1       68s

How did you achieve it? Mind documenting here the process you followed?

I have the Apple MacBook Pro 13".

galishmann · 2021-01-23T17:53:09Z

➜  ~ minikube start --driver=docker
😄  minikube v1.17.0 on Darwin 11.1
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=3881MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.2 ...
💢  initialization failed, will try again: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV
stdout:

stderr:
SIGSEGV: segmentation violation
PC=0x0 m=0 sigcode=0

goroutine 1 [running]:
qemu: uncaught target signal 11 (Segmentation fault) - core dumped


💣  Error starting cluster: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV
stdout:

stderr:


😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

❌  Exiting due to GUEST_START: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV
stdout:

stderr:


😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new/choose

sayboras · 2021-01-24T00:21:52Z

How did you achieve it? Mind documenting here the process you followed?

Oh sorry, I forgot to mention the below points in my system:

Build minikube from source with go1.16beta version
Install docker mac preview for apple silicon, and make sure it's working. You might be running into some issues here and there, docker slack channel might have some useful info to tackle them.

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., unknown)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.3.5)

Server:
 Containers: 3
  Running: 1
  Paused: 0
  Stopped: 2
 Images: 6
 Server Version: 20.10.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.935GiB
 Name: docker-desktop
 ID: PYLF:XCJN:NO3S:L7UQ:AP3I:Z347:NWIH:UIRD:EMW2:YNE7:CMNR:AYQC
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
 
$ go version
go version go1.16beta1 darwin/arm64

$ make
go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/...
gofmt -s -w pkg/minikube/assets/assets.go
go-bindata -nomemcopy -o pkg/minikube/translate/translations.go -pkg translate translations/...
gofmt -s -w pkg/minikube/translate/translations.go
go build  -tags "go_getter_nos3 go_getter_nogcs" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.17.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.17.0 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="94a676a6717732052a7ed96ef00c6f1943af1497" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v4" -o out/minikube k8s.io/minikube/cmd/minikube

$ ./out/minikube version
minikube version: v1.17.0
commit: 94a676a6717732052a7ed96ef00c6f1943af1497

$ ./out/minikube start --driver=docker
😄  minikube v1.17.0 on Darwin 11.1 (arm64)
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=1981MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.2 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

kientran · 2021-01-24T07:30:48Z

Side note for those wanting to try this. The M1 build Go-lang formula is available in the M1 Homebrew preview with brew install go. While brew installs 1.15.7, it's actually 1.16beta1 so you don't need to clone and build it separately.

afbjorklund · 2021-01-24T08:49:41Z

Trying to find the upstream (moby) issue for the qemu segfault, since it seems to be a known issue with this Docker preview...

qemu: uncaught target signal 11 (Segmentation fault) - core dumped

But can mostly find user reports, that are all reporting crashes. Doesn't seem to be anything in https://github.com/moby/qemu

https://github.com/docker/for-mac/labels/area%2Fm1

choudharyanurag · 2021-01-26T04:50:02Z

@sayboras I did a make on the minikube code but it is not working for me.

MacBook-Pro:minikube anurag$ ./out/minikube start
😄 minikube v1.17.0 on Darwin 11.1
✨ Using the docker driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🏃 Updating the running docker "minikube" container ...
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.2 ...
💢 initialization failed, will try again: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV
stdout:

stderr:
qemu: uncaught target signal 11 (Segmentation fault) - core dumped

💣 Error starting cluster: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV
stdout:

stderr:
SIGSEGV:

😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
👉 https://github.com/kubernetes/minikube/issues/new/choose

❌ Exiting due to GUEST_START: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV
stdout:

stderr:
SIGSEGV:

😿 If the above advice does not help, please let us know:
👉 https://github.com/kubernetes/minikube/issues/new/choose

sasha7 · 2021-01-26T21:08:48Z

I've installed arm version of minikube via brew brew install minikube --build-from-source and latest docker preview beta 7,

❯ minikube start --driver=docker
😄  minikube v1.17.0 on Darwin 11.1 (arm64)
    ▪ MINIKUBE_ACTIVE_DOCKERD=minikube
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.20.2 preload ...
    > preloaded-images-k8s-v8-v1....: 514.92 MiB / 514.92 MiB  100.00% 9.47 MiB
🔥  Creating docker container (CPUs=2, Memory=1981MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.2 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

dashboard was loading as well:

❯ minikube dashboard
🔌  Enabling dashboard ...
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
🎉  Opening http://127.0.0.1:58958/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...

Besides doing that, I was able to deploy a simple golang hello world container to it and expose as a service, worked like a charm!

❯ k get all
NAME                           READY   STATUS    RESTARTS   AGE
pod/ms-test-7849ffc897-f8hcr   1/1     Running   0          4m25s
pod/ms-test-7849ffc897-ltjcr   1/1     Running   0          4m22s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP          16m
service/ms-test      NodePort    10.111.66.75   <none>        8080:31665/TCP   6m40s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ms-test   2/2     2            2           6m40s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/ms-test-5979675bfb   0         0         0       6m40s
replicaset.apps/ms-test-7849ffc897   2         2         2       4m25s

NAME                                          REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/ms-test   Deployment/ms-test   <unknown>/50%   2         5         2          6m40s

brhamill · 2021-01-30T20:37:44Z

@sasha7 - I tried running the "brew install minikube --build-from-source" and it couldn't get through the process.

Kept giving me the following error. Any thoughts?

tstromberg · 2021-02-01T16:41:48Z

We'll need to update our docs today, but as of last week, folks can install the latest release of minikube for Apple Silicon using:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-arm64
sudo install minikube-darwin-amd64 /usr/local/bin/minikube

brhamill · 2021-02-01T18:41:24Z

@tstromberg - I attempted to run the commands on my M1. When starting minikube I had the following error. Any ideas? Thanks!

tstromberg · 2021-02-01T19:03:34Z

@brhamill - I've not yet seen a SEGV. Can you please open a new issue? That seems like an unstable VM environment to me, so please check that you are on the latest Docker for M1 build.

choudharyanurag · 2021-02-02T08:34:40Z

same as @brhamill

@tstromberg : Probably I am on the latest build of the Docker for M1 preview :
Following is my docker info :

Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., unknown)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.3.5)

Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 20.10.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.104-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 1.935GiB
Name: docker-desktop
ID: MGZK:F6IX:3OU4:E2WG:AI74:AA5S:TVXR:M2MN:LWPL:5SER:N3ZA:KYQB
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

brhamill · 2021-02-03T06:18:50Z

@tstromberg - the commands you provided:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-arm64
sudo install minikube-darwin-amd64 /usr/local/bin/minikube

Should the second line be: sudo install minikube-darwin-arm64 /usr/local/bin/minikube instead? Since it's arm instead of amd?

I made that change and then attempted a "minikube start" I got a zsh permissions error. Am I doing something incorrect with my approach?

zsh: permission denied: minikube

brhamill · 2021-02-03T15:45:19Z

I was finally able to get minikube running well on my M1 Silicon.

This YT link lays it all out in a very straight-forward way. Hope it helps others...

https://www.youtube.com/watch?v=IjgrwYzVlMw

cheslijones · 2021-02-03T16:44:41Z

I was finally able to get minikube running well on my M1 Silicon.

This YT link lays it all out in a very straight-forward way. Hope it helps others...

https://www.youtube.com/watch?v=IjgrwYzVlMw

Thanks for sharing. I played with v1.17.1 last night and while I could get my cluster to spin up with skaffold I couldn't figure out how to access it as minikube addons enable ingress didn't work.

--vm-driver=docker seems to be the only supported driver. I've seen people recommend this driver in the past, but neither they nor the documentation actually explains how to use it so the end result has always been the same: my cluster spins up, but I can't access the running application from a browser--doesn't do any good (and port-forward is not a realistic alternative, which someone inevitably recommends doing).

Video seems to actually explain it. I'll give this a shot tonight.

choudharyanurag · 2021-02-03T16:52:18Z

I was finally able to get minikube running well on my M1 Silicon.
This YT link lays it all out in a very straight-forward way. Hope it helps others...
https://www.youtube.com/watch?v=IjgrwYzVlMw

Thanks for sharing. I played with v1.17.1 last night and while I could get my cluster to spin up with skaffold I couldn't figure out how to access it as minikube addons enable ingress didn't work.

--vm-driver=docker seems to be the only supported driver. I've seen people recommend this driver in the past, but neither they nor the documentation actually explains how to use it so the end result has always been the same: my cluster spins up, but I can't access the running application from a browser--doesn't do any good (and port-forward is not a realistic alternative, which someone inevitably recommends doing).

Video seems to actually explain it. I'll give this a shot tonight.

minikube tunnel

afbjorklund · 2021-02-03T16:53:07Z

The Docker Desktop networking model doesn't offer access to the VM, so that basically leaves port forwarding (unfortunately)

https://docs.docker.com/docker-for-mac/networking/

cheslijones · 2021-02-03T22:49:36Z

Well, did get it working a couple different ways:

The way outlined in the video which is mostly from here. Then had to do minikube service example-service --url to get the actual URL:

🏃  Starting tunnel for service example-service.
|-----------|-----------------|-------------|------------------------|
| NAMESPACE |      NAME       | TARGET PORT |          URL           |
|-----------|-----------------|-------------|------------------------|
| default   | example-service |             | http://127.0.0.1:51619 |
|-----------|-----------------|-------------|------------------------|

Also creating a LoadBalander based on the documentation here. After that, minikube tunnel did come to life as opposed to sitting there indefinitely with no console output. But, wasn't able to access the running application from browser. Had to do the same thing as the previous:

minikube service client-deployment-dev --url
🏃  Starting tunnel for service client-deployment-dev.
|-----------|-----------------------|-------------|------------------------|
| NAMESPACE |         NAME          | TARGET PORT |          URL           |
|-----------|-----------------------|-------------|------------------------|
| default   | client-deployment-dev |             | http://127.0.0.1:52989 |
|-----------|-----------------------|-------------|------------------------|

Checking the LoadBalancer, it is indefinitely pending on getting an EXTERNAL-IP:

NAME                            TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
client-cluster-ip-service-dev   ClusterIP      10.96.33.149   <none>        3000/TCP         35m
client-deployment-dev           LoadBalancer   10.104.53.56   <pending>     8080:30700/TCP   11m
example-service                 NodePort       10.101.7.132   <none>        3000:32764/TCP   27m
kubernetes                      ClusterIP      10.96.0.1      <none>        443/TCP          40m

Doesn't appear to be any issues in the describe:

Name:                     client-deployment-dev
Namespace:                default
Labels:                   app.kubernetes.io/managed-by=skaffold
                          skaffold.dev/run-id=d46b55ce-ca0f-4615-b9e2-22166fdf0b93
Annotations:              <none>
Selector:                 component=client,environment=development
Type:                     LoadBalancer
IP:                       10.104.53.56
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30700/TCP
Endpoints:                172.17.0.3:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

At any rate, with any of these, I'd have to make a service for each deployment--I have one / (client), /api, /admin. I can see that presenting issues similar to what I've faced using port-forward a call from / to the /api will append localhost:3000, for example, but /api is at localhost:5000, so then it says it can't find the API at localhost:3000/api.

Definitely a lot more straight forward with KVM2, Hyperkit, etc. drivers and using an ingress controller like ingress-nginx. Is there any time frame for Virtualization.Framework support, lol?

cheslijones · 2021-02-09T19:36:21Z

I did manage to get minikube tunnel and ingress-nginx to work together.

I just needed to enable ingress-nginx for Docker Desktop with:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/cloud/deploy.yaml

I started my cluster with skaffold.

When that was running I ran minikube tunnel and it found the ingress-nginx-controller saying:

❗  The service ingress-nginx-controller requires privileged ports to be exposed: [80 443]
🔑  sudo permission will be asked for it.
🏃  Starting tunnel for service ingress-nginx-controller.

Didn't have to expose any, or each, of my microservices and routing seems to be working fine between them where they refer to one another.

So using --driver=docker is basically one more step than hyperkit, kvm, etc.: you need to run minikube tunnel. I've had issues with doing that in the past, I think in WSL2, but seems to work with macOS, minikube 1.17.1, Docker M1 preview build, etc.

franciscosuca · 2021-02-16T08:06:26Z

I was finally able to get minikube running well on my M1 Silicon.

This YT link lays it all out in a very straight-forward way. Hope it helps others...

https://www.youtube.com/watch?v=IjgrwYzVlMw

This answer and the new Docker Desktop preview 3.1.0 (with K8s included) helped me to install it successfully.

Thanks!

tstromberg · 2021-02-22T19:43:40Z

Support for M1 launched, so I'm closing this in preference to individual issues being opened.

NitichaiSawangsai · 2021-07-02T14:16:36Z

You try this command

 arch -x86_64 brew install hyperkit & minikube start --vm=true

ayepRahman · 2021-07-03T18:29:51Z

You try this command

arch -x86_64 brew install hyperkit & minikube start --vm=true

Have not try this, but are you saying iam able to run minikube on arm64 with --docker=hyperkit without the needing of port forwarding?

slmingol · 2021-10-05T22:06:27Z

Confirming that Docker's official .dmg worked fine for me along w/ brew install minikube

https://docs.docker.com/desktop/mac/apple-silicon/

kamilgregorczyk · 2021-12-14T12:59:05Z

Is there a way of using minikube without the docker for mac app on M1? it's license became kind of repelling, hyperkit doesn't seem to work on M1.

chicks-net · 2022-03-24T21:57:57Z

Is there a way of using minikube without the docker for mac app on M1? it's license became kind of repelling, hyperkit doesn't seem to work on M1.

I've gotten minikube going with podman 4.0.2 on an M1 Mac. More details are in this stackoverflow answer.

I just filed #13855 to see if we can clean this up a bit more.

temikelani · 2022-05-05T15:46:51Z

use docker - install docker desktop for Mac M1 and run

make docker default driver
```
minikube config set driver docker
```
Start minikube
```
minikube start
```
Start Minikube without making docker default driver
```
minikube start --driver=docker 
```

if you wanna see logs

minikube start --driver=docker --alsologtostderr

source: https://medium.com/@seohee.sophie.kwon/how-to-run-a-minikube-on-apple-silicon-m1-8373c248d669

jnardiello · 2023-02-19T08:12:55Z

Virtualbox on apple silicon (M1/M2) is coming along and is now available on the developer preview version. Any chance minikube is going to support that? Have you ever discussed it or done any work on it?

Rajat-Dabade · 2023-03-03T08:50:07Z

Virtualbox on apple silicon (M1/M2) is coming along and is now available on the developer preview version. Any chance minikube is going to support that? Have you ever discussed it or done any work on it?

I have tried running it with a developer preview version of Virtualbox on M1 but hard luck.

minikube start --driver=virtualbox                                                                      ok  4s  18.14.2 node 
😄  minikube v1.29.0 on Darwin 12.6 (arm64)
✨  Using the virtualbox driver based on user configuration

❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'virtualbox' is not supported on darwin/arm64

lowang-bh · 2023-04-15T03:53:23Z

Virtualbox on apple silicon (M1/M2) is coming along and is now available on the developer preview version. Any chance minikube is going to support that? Have you ever discussed it or done any work on it?

I have tried running it with a developer preview version of Virtualbox on M1 but hard luck.
minikube start --driver=virtualbox                                                                      ok  4s  18.14.2 node 
😄  minikube v1.29.0 on Darwin 12.6 (arm64)
✨  Using the virtualbox driver based on user configuration

❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'virtualbox' is not supported on darwin/arm64

how about minikube start --driver=docker?

darkn3rd · 2023-07-18T19:55:16Z

And no VirtualBox (it's x86 only)

Actually, there's an arm64 virtualbox now: https://www.virtualbox.org/wiki/Download_Old_Builds_7_0

afbjorklund · 2023-07-18T20:16:51Z

The arm64 VirtualBox runs Intel emulation, not what you want for minikube

https://forums.virtualbox.org/viewtopic.php?t=107344

desylva · 2024-09-30T00:06:21Z

And no VirtualBox (it's x86 only)

Actually, there's an arm64 virtualbox now: https://www.virtualbox.org/wiki/Download_Old_Builds_7_0

❌ Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: precreate: Docker Machine can be used only with Parallels Desktop Pro or Business edition. You use: standard edition

Argh...

medyagh added kind/feature Categorizes issue or PR as related to a new feature. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Sep 11, 2020

afbjorklund changed the title ~~minikube on Apple Silicone~~ minikube on Apple Silicone (darwin-arm64) Sep 12, 2020

afbjorklund added the os/macos label Sep 12, 2020

This was referenced Sep 12, 2020

Build KIC base image for ARM #9227

Closed

Build ISO minikube image for ARM (aarch64) #9228

Closed

tstromberg mentioned this issue Nov 19, 2020

Publish macOS/arm64 binaries #9739

Closed

afbjorklund changed the title ~~minikube on Apple Silicone (darwin-arm64)~~ minikube on Apple Silicon / M1 CPU (darwin-arm64) Dec 6, 2020

tstromberg mentioned this issue Jan 9, 2021

Upgrade gopsutil & docker libs for macOS/arm64 compatibility #10115

Merged

kientran mentioned this issue Jan 24, 2021

Validation of kubernettes-101 episodes against Apple M1 build geerlingguy/kubernetes-101#20

Closed

tstromberg closed this as completed Feb 22, 2021

alexanderciuffreda mentioned this issue Oct 21, 2021

kubeflow deployment on M1 (darwin arm64) alexanderciuffreda/mlops#3

Open

iapicca mentioned this issue Oct 3, 2022

Error downloading kic artifacts #14664

Closed

minikube on Apple Silicon / M1 CPU (darwin-arm64) #9224

minikube on Apple Silicon / M1 CPU (darwin-arm64) #9224

Comments

medyagh commented Sep 11, 2020 • edited Loading

Docker can not be installed

hyper kit can not be installed

afbjorklund commented Sep 12, 2020

afbjorklund commented Sep 12, 2020

afbjorklund commented Sep 12, 2020 • edited Loading

afbjorklund commented Sep 12, 2020 • edited Loading

goafabric commented Dec 6, 2020 • edited Loading

afbjorklund commented Dec 6, 2020 • edited Loading

medyagh commented Dec 24, 2020 • edited Loading

tstromberg commented Jan 9, 2021

sayboras commented Jan 23, 2021

smithaitufe commented Jan 23, 2021

galishmann commented Jan 23, 2021

sayboras commented Jan 24, 2021 • edited Loading

kientran commented Jan 24, 2021

afbjorklund commented Jan 24, 2021 • edited Loading

choudharyanurag commented Jan 26, 2021

sasha7 commented Jan 26, 2021 • edited Loading

brhamill commented Jan 30, 2021 • edited Loading

tstromberg commented Feb 1, 2021 • edited Loading

brhamill commented Feb 1, 2021

tstromberg commented Feb 1, 2021

choudharyanurag commented Feb 2, 2021

brhamill commented Feb 3, 2021 • edited Loading

brhamill commented Feb 3, 2021

cheslijones commented Feb 3, 2021

choudharyanurag commented Feb 3, 2021

afbjorklund commented Feb 3, 2021

cheslijones commented Feb 3, 2021 • edited Loading

cheslijones commented Feb 9, 2021 • edited Loading

franciscosuca commented Feb 16, 2021 • edited Loading

tstromberg commented Feb 22, 2021

NitichaiSawangsai commented Jul 2, 2021 • edited Loading

ayepRahman commented Jul 3, 2021

slmingol commented Oct 5, 2021

kamilgregorczyk commented Dec 14, 2021

chicks-net commented Mar 24, 2022

temikelani commented May 5, 2022 • edited Loading

jnardiello commented Feb 19, 2023

Rajat-Dabade commented Mar 3, 2023

lowang-bh commented Apr 15, 2023

darkn3rd commented Jul 18, 2023

afbjorklund commented Jul 18, 2023

desylva commented Sep 30, 2024

medyagh commented Sep 11, 2020 •

edited

Loading

afbjorklund commented Sep 12, 2020 •

edited

Loading

afbjorklund commented Sep 12, 2020 •

edited

Loading

goafabric commented Dec 6, 2020 •

edited

Loading

afbjorklund commented Dec 6, 2020 •

edited

Loading

medyagh commented Dec 24, 2020 •

edited

Loading

sayboras commented Jan 24, 2021 •

edited

Loading

afbjorklund commented Jan 24, 2021 •

edited

Loading

sasha7 commented Jan 26, 2021 •

edited

Loading

brhamill commented Jan 30, 2021 •

edited

Loading

tstromberg commented Feb 1, 2021 •

edited

Loading

brhamill commented Feb 3, 2021 •

edited

Loading

cheslijones commented Feb 3, 2021 •

edited

Loading

cheslijones commented Feb 9, 2021 •

edited

Loading

franciscosuca commented Feb 16, 2021 •

edited

Loading

NitichaiSawangsai commented Jul 2, 2021 •

edited

Loading

temikelani commented May 5, 2022 •

edited

Loading