Builds don't work if the docker socket isn't in /var/run/docker.sock location (using lima instead of Docker Desktop)

[agxs]

Actual behavior

It looks like running skaffold run fails when the docker socket isn't in the default location of /var/run/docker.sock. Even when setting the options in the skaffold.yaml to use useDockerCLI: true the build still fails.

Expected behavior

That the docker socket file location is can be anywhere when the options in the skaffold.yaml file are set to use the docker cli and/or provide a way of overriding the expected socket location that skaffold uses.

Information

• Skaffold version: v1.35.2
• Operating system: MacOS 11.6.1
• Installed via: Homebrew

I'm using limactl version 0.8.2 instead of Docker Desktop, installed via Homebrew, along with the docker cli also installed via Homebrew. VM was created using the sample docker.yaml at https://github.com/lima-vm/lima/blob/master/examples/docker.yaml

Steps to reproduce the behavior

1. limactl start https://raw.githubusercontent.com/lima-vm/lima/master/examples/docker.yaml
2. Run the suggested commands to configure a docker context in the message output from limactl
3. skaffold run
4. Receive the following error:

getting imageID for <image id-dirty>: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

Normal docker commands work fine, just commands via skaffold have the issue.

I currently have a workaround where I manually create a symlink from /var/run/docker.sock to the socket that lima creates in $HOME/.lima/<vm>/sock/docker.sock

[814HiManny]

I have the same issue. Docker context is not honored by skaffold.

[cooperbenson-qz]

I'm also experiencing the same issue with colima. Would love to see Docker context support added 😁

[carsondwilber]

Also facing this issue trying to use Podman in place of Docker. On Mac, using a symlink to any Podman socket doesn't work either. There is ongoing discussion in the Kubernetes Slack server, channel #skaffold related to this issue.

#5739 and #5773 are both open issues that could easily provide an alternative via Minikube, but this has been deprioritized for unexplained reasons. The Minikube team appears to have done their part for this and is waiting for the Skaffold team to do the same as noted here.

Hope these breadcrumbs help others seeking a solution.

[tejal29]

Thanks you for the issue. We would appreciate any help from the community on this issue.

[PertsevRoman]

Hi, I succeeded to run Docker build when specified
DOCKER_HOST=unix:///home/dir/.colima/docker.sock

[jearangoo]

i have the same issue when using docker desktop for linux in ubuntu, as the documentation says there is another context that uses another endpoint different from unix:///var/run/docker.sock, the context for docker desktop for linux is unix:///home/<user>/.docker/desktop/docker.sock

https://docs.docker.com/desktop/linux/install/#context

[Guillaume-Mayer]

I think I have the same issue, I'm trying to push an image to a registry.
Docker-cli is installed but there is no Docker engine available.
I thought it was the point of the useDockerCLI: true setting.

Skaffold v1.36.0

time="2022-05-31T01:01:01Z" level=warning msg="failed to list images: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" subtask=gitlab-ci-java task=Build
failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
exit status 1. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

[leslieluyu]

Hi, guys. I think I'm having a related problem.
currently, we're using contained instead of docker. So we're using nerdctl as a docker-like CLI tools.
After installing buildkit, now we can build images as before.
But how can I let skaffold using nerdctl to build images?
the error message shows that skafflod needs the docker

I've tried several methods including create a soft link in /usr/local/sbin. But still no good.
Is there any way to solve this problem? Thanks anyway.

[MarlonGamez]

@agxs it looks like there are a couple places in our code where we're not respecting the user's use of useCLI. Would be great to comb through these and make sure we're using docker cli where appropriate when specified. I'm not sure our team has the bandwidth to look at this/fix this soon, but PRs are always welcome

[MarlonGamez]

@leslieluyu would you be able to open an issue and attach logs from running with -v=debug?

[drewp]

Similar workaround for podman users:

1. Be running podman system service --time=0
2. DOCKER_HOST=unix:///run/user/${UID}/podman/podman.sock skaffold run

[jdoylei]

@MarlonGamez, it sounds like there's not currently a workaround for podman users not using the podman system service (and podman.sock and DOCKER_HOST) as @drewp mentioned, since there is still some code not respecting useCLI, right?

E.g. I see from our skaffold build (v1.39.2) attempting to use podman:

build [xxx.yyy.zzz/archrepo/archrepo-ui] failed: getting imageID for xxx.yyy.zzz/archrepo/archrepo-ui:versions_container_initial-4-g937a427-dirty: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/xxx.yyy.zzz/archrepo/archrepo-ui:versions_container_initial-4-g937a427-dirty/json": dial unix /var/run/docker.sock: connect: permission denied. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

[gsquared94]

The way to fix this is to set DOCKER_HOST correctly or link the correct socket to the default socket path.
See here for Colima and #7078 (comment) for podman.
Opened #8190 to improve skaffold's error message on this.

[jdoylei]

@gsquared94, I understand that the original issue description here was about build tools that have a docker socket but place it elsewhere, and that the DOCKER_HOST workaround applies there.

But the issue description also references "useDockerCLI: true". I think @MarlonGamez 's comment earlier mentioned that the useDockerCLI setting was a little incomplete, contributing to this problem. It sounds like build tools that do not have a docker socket at all (like using podman but without the podman system service) will still not work because of this, right?

I know there may not be bandwidth to complete the cleanup of useDockerCLI. But could an issue be created to track that it's incomplete? Right now it kind of seems that useDockerCLI doesn't do anything from the user's point of view, does it? Since skaffold is still going to use the Docker Engine API anyway via the required docker socket. And/or a documentation update clarifying the state of useDockerCLI?

Thanks for your time!

[afbjorklund]

We are making an effort to provide a socket for use with minikube docker-env:

• Create a prototype of direct image build on containerd and cri-o kubernetes/minikube#15389

Mostly since projects like skaffold have been so slow to adopt minikube image...

• consider using minikube image build #5739

---

The tricky part with DOCKER_HOST for ssh:, is that it also needs to be set for a non-interactive shell...

https://docs.docker.com/engine/security/rootless/#expose-docker-api-socket-through-ssh

Thus it needs to go at the very top of the ~/.bashrc (on the remote!), before it returns.

# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

It also need a "docker" program to run docker system dial-stdio, but that can be fixed.

For instance you can do a small shell wrapper with socat, or some similar workaround.