autoset --preload=false when storage driver is not overlay

[afbjorklund]

So if your docker daemon runs anything else, it is broken after the minikube start.

$ docker info | grep Driver
 Storage Driver: aufs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.

$ minikube start --driver none

	[WARNING SystemVerification]: error verifying Docker info: "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

I0412 09:38:01.674741   16553 exec_runner.go:78] Completed: sudo tar -I lz4 -C /var -xvf /preloaded.tar.lz4: (2.874924393s)
I0412 09:38:01.674767   16553 exec_runner.go:114] rm: /preloaded.tar.lz4
I0412 09:38:01.742410   16553 exec_runner.go:49] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0412 09:38:01.752632   16553 exec_runner.go:91] found /var/lib/docker/image/overlay2/repositories.json, removing ...
I0412 09:38:01.752681   16553 exec_runner.go:98] cp: memory --> /var/lib/docker/image/overlay2/repositories.json (3417 bytes)
I0412 09:38:01.752764   16553 exec_runner.go:49] Run: sudo systemctl daemon-reload
I0412 09:38:01.867623   16553 exec_runner.go:49] Run: sudo systemctl restart docker
I0412 09:38:02.357525   16553 kubeadm.go:634] prelaoding failed, will try to load cached images: sudo systemctl restart docker: exit status 1
stdout:

stderr:
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

The systemd status is a silly docker.service: Failed with result 'exit-code'.
But the journal has the real error: failed to start daemon: error initializing graphdriver: /var/lib/docker contains several valid graphdrivers: aufs, overlay2; Please cleanup or explicitly choose storage driver (-s <DRIVER>)

We should check the actual driver used, in case something else replaces "overlay2" in the future.

---

To restore the functionality, one needs to first sudo rm -r /var/lib/docker/overlay2 (and restart)

Note that we do use the storage driver when writing the tarball. Just not when actually reading it.

And of course, it's not flexible there either:

// TarballName returns name of the tarball
func TarballName(k8sVersion, containerRuntime string) string {
        return fmt.Sprintf("preloaded-images-k8s-%s-%s-%s-overlay2-%s.tar.lz4", PreloadVersion, k8sVersion, containerRuntime, runtime.GOARCH)
}

There is a driver check verifyDockerStorage, but it doesn't seem to actually be used for start.

The right thing to do is probably to just skip the preload, and let docker load the images as usual ?

[medyagh]

ing to do is probably to just skip the preload, and let docker load the images as usual ?

when we generate the preload we include the storage name in the file name,
so the right thing is,

check the sotrage driver of the user,
then try to get the preload image for https:///URLTOBUCKET/k8s-version-runtime-storagedriver.tar.gz

if it exists then proceed if not, then dont proceed.

(that way in the futrure if we generate the preload image for those drivers we can use them

[afbjorklund]

For this to work, the storage driver needs to be added to the API (after the runtime)
func PreloadExists(k8sVersion, containerRuntime, storageDriver string) bool

[afbjorklund]

The main goal wasn't to provide tarballs for deprecated or specialized storage drivers.
It was more to avoid minikube destroying the docker service, when trying to preload.

[afbjorklund]

This was for using the none driver.

For the docker driver, we should still use "overlay2" since that is what the Ubuntu 19.10 is using.
i.e. those images are for the docker-in-docker, not for the host docker (that could run another one)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[liclac]

I've been running into some problems with this on a laptop running on ZFS, where minikube start fails to start Docker:

$ minikube start
[...]
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

💡  Suggestion: Remove the incompatible --docker-opt flag if one was provided
⁉️   Related issue: https://github.com/kubernetes/minikube/issues/7070

$ minikube logs
[...]
Jul 19 14:11:08 minikube dockerd[277]: time="2020-07-19T14:11:08.787309773Z" level=error msg="'overlay2' is not supported over zfs" storage-driver=overlay2
Jul 19 14:11:08 minikube dockerd[277]: time="2020-07-19T14:11:08.787355903Z" level=error msg="[graphdriver] prior storage driver overlay2 failed: backing file system is unsupported for this graph driver"
Jul 19 14:11:08 minikube dockerd[277]: failed to start daemon: error initializing graphdriver: backing file system is unsupported for this graph driver
[...]

While explicitly passing --docker-opt storage-driver=zfs prints an even more confounding error:

$ minikube logs
[...]
Jul 19 14:20:48 minikube dockerd[280]: failed to start daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?)
[...]

This sent me down a bit of a wild goose chase, before finally realising that the problem was the preload, and that this works:

$ minikube start --preload=false

Docker's documentation on storage drivers has a table of supported backing filesystems for each, and I think we should look to detect if we're about to run into this problem, and disable preloading by default if we are.

The easiest way to do this might be to just ask the host's dockerd which one it's using, and disable preloading if it answers zfs or btrfs.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten