Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to access docker api on a mac #11397

Closed
tibcoplord opened this issue Sep 1, 2021 · 95 comments · Fixed by #13075
Closed

How to access docker api on a mac #11397

tibcoplord opened this issue Sep 1, 2021 · 95 comments · Fixed by #13075
Assignees
Labels
kind/feature Categorizes issue or PR as related to a new feature. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. machine macos MacOS (OSX) related

Comments

@tibcoplord
Copy link

tibcoplord commented Sep 1, 2021

/kind feature

Description

I've been following https://github.com/containers/podman/blob/main/docs/tutorials/mac_experimental.md on my mac, specifically -

podman machine init --cpus 6 --disk-size 200
podman machine start
podman build ...
podman run ...

Actually works pretty well :-) However next I want to use a docker client ( specifically the fabric8 maven plugin to build images, but we can test with docker command ), but I see -

$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I read at https://github.com/containers/podman/blob/main/docs/source/markdown/podman-system-service.1.md that there is an option to provide the API service, but this fails with -

$ podman system service
Error: unrecognized command `podman system service`
Try 'podman system --help' for more information.

Steps to reproduce the issue:

  1. On a mac with podman machine running, run podman system service

Describe the results you received:

Error: unrecognized command podman system service

Describe the results you expected:

Able to connect to the socket from a docker client (outside the machine)

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Client:
Version:      3.3.0
API Version:  3.3.0
Go Version:   go1.16.6
Built:        Fri Aug 20 18:47:23 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.0
API Version:  3.3.0
Go Version:   go1.16.6
Built:        Fri Aug 20 20:36:14 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 6
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.12-200.fc34.x86_64
  linkmode: dynamic
  memFree: 2224381952
  memTotal: 25198764032
  ociRuntime:
    name: crun
    package: crun-0.21-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.21
      commit: c4c3cdf2ce408ed44a9e027c618473e6485c635b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 4h 50m 50.8s (Approximately 0.17 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 18
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.0
  Built: 1629488174
  BuiltTime: Fri Aug 20 19:36:14 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.0

Additional environment details (AWS, VirtualBox, physical, etc.):

mac client with podman machine running.

I did wonder if the technique I need is to somehow tunnel the socket running inside the podman machine to outside ... but I couldn't find this in the docs.

@openshift-ci openshift-ci bot added the kind/feature Categorizes issue or PR as related to a new feature. label Sep 1, 2021
@mheon
Copy link
Member

mheon commented Sep 1, 2021

@baude @jwhonce PTAL

@rhatdan
Copy link
Member

rhatdan commented Sep 1, 2021

I think you need to specify DOCKER_HOST environment variable for this, but this is where my knowledge ends.

@matejvasek
Copy link
Contributor

matejvasek commented Sep 1, 2021

As Dan mentioned export DOCKER_HOST="/run/user/1000/podman/podman.sock" should do the trick. It works for instance with docker CLI and official Go client library.

@matejvasek
Copy link
Contributor

However it still my happen that some application wrongly assume that socket is always /var/run/docker.sock.

@tibcoplord
Copy link
Author

On the mac I see -

$ podman ps
CONTAINER ID  IMAGE                    COMMAND               CREATED      STATUS          PORTS       NAMES
5b8fccfa9106  localhost/jenkins:1.0.0  ./start-jenkins.s...  5 hours ago  Up 5 hours ago              jenkins
$ DOCKER_HOST="/run/user/1000/podman/podman.sock" docker ps
Cannot connect to the Docker daemon at tcp://localhost:2375/run/user/1000/podman/podman.sock. Is the docker daemon running?

Indeed, /run/user/1000/podman/podman.sock doesn't exist on the mac, only inside the podman machine.

@matejvasek
Copy link
Contributor

matejvasek commented Sep 1, 2021

oh my bad, you need to specify protocol too export DOCKER_HOST="unix:///run/user/1000/podman/podman.sock".

@tibcoplord
Copy link
Author

$ DOCKER_HOST="unix:///run/user/1000/podman/podman.sock" docker ps
Cannot connect to the Docker daemon at unix:///run/user/1000/podman/podman.sock. Is the docker daemon running?

I'm not seeing how the socket would work if its inside the podman machine.

@baude
Copy link
Member

baude commented Sep 1, 2021

it can be made to work i think, the problem is a little more sophisticated than the network connection I am afraid. i'm looking into casually, but the first issue is the presence of moby on the os.

@tibcoplord
Copy link
Author

Many thanks :-)

@matejvasek
Copy link
Contributor

@baude isn't podman on mac actually podman-remote?

@afbjorklund
Copy link
Contributor

afbjorklund commented Sep 1, 2021

We ended up symlinking inside the VM, since the location of the unix socket is hardcoded (when using ssh: address) :

export DOCKER_HOST=$CONTAINER_HOST

ssh host connection is not valid: extra path after the host: "/run/podman/podman.sock"

So had to remove the /run/podman/podman.sock from the URL, and then link /var/run/docker.sock to it (in the VM)


To work in CoreOS, you first have to systemctl stop docker.socket. And then remove the old docker.sock.

$ podman machine ssh
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 34.20210821.1.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

[core@localhost ~]$ sudo systemctl stop docker.socket
[core@localhost ~]$ sudo rm /var/run/docker.sock
[core@localhost ~]$ sudo ln -s /run/podman/podman.sock /var/run/docker.sock
[core@localhost ~]$ sudo docker version
Client:
 Version:           20.10.7
 API version:       1.40
 Go version:        go1.16.5
 Git commit:        f0df350
 Built:             Fri Jul 16 12:30:15 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: linux/amd64/fedora-34
 Podman Engine:
  Version:          3.3.0
  APIVersion:       3.3.0
  Arch:             amd64
  BuildTime:        2021-08-20T19:36:14Z
  Experimental:     true
  GitCommit:        
  GoVersion:        go1.16.6
  KernelVersion:    5.13.12-200.fc34.x86_64
  MinAPIVersion:    3.1.0
  Os:               linux
 Engine:
  Version:          3.3.0
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.16.6
  Git commit:       
  Built:            Fri Aug 20 19:36:14 2021
  OS/Arch:          linux/amd64
  Experimental:     true
[core@localhost ~]$ exit

Now it will work with the URL from podman system connection ls, as long as the path is removed.

$ podman system connection ls
Name                         Identity                                  URI
podman-machine-default*      /home/anders/.ssh/podman-machine-default  ssh://core@localhost:45271/run/user/1000/podman/podman.sock
podman-machine-default-root  /home/anders/.ssh/podman-machine-default  ssh://root@localhost:45271/run/podman/podman.sock
CONTAINER_HOST=ssh://root@localhost:45271/run/podman/podman.sock podman --remote version
DOCKER_HOST=ssh://root@localhost:45271 docker version

And similar for the rootless socket. (located in the VM at /run/user/1000/podman/podman.sock)

EDIT: You also need to ssh-add the identity, since there is no CONTAINER_SSHKEY in Docker...

@matejvasek
Copy link
Contributor

ssh host connection is not valid: extra path after the host: "/run/podman/podman.sock"

yeah docker doesn't support unix socket over ssh it seems, but I can be worked around using ssh tunnel

@afbjorklund
Copy link
Contributor

afbjorklund commented Sep 1, 2021

It does support the socket, it just hardcodes the location. So I guess it doesn't support giving a path.

And apparently CoreOS still ships with Docker, so it will actually start the real dockerd instead...

@matejvasek
Copy link
Contributor

what I mean you can have something like ssh://user@host as DOCKER_HOST but not something like ssh://use@host/some/path.sock

@matejvasek
Copy link
Contributor

matejvasek commented Sep 1, 2021

ssh -nNT -L/tmp/podman.sock:/run/user/1000/podman/podman.sock ssh://user@host
or
ssh -nNT -L/run/user/1000/podman/podman.sock:/tmp/podman.sock ssh://user@host
not sure which one, probably the first one, I always mix it up.
Then you can set export DOCKER_HOST=unix:///tmp/podman.sock

@matejvasek
Copy link
Contributor

The command above basically tunnels remote unix socket to you local machine.

@matejvasek
Copy link
Contributor

what I mean you can have something like ssh://user@host as DOCKER_HOST but not something like ssh://use@host/some/path.sock

At least this seems to be true for standard Go client library.

podman is better at this since CONTAINER_HOST can be ssh://use@host/some/path.sock

@matejvasek
Copy link
Contributor

I played with this when I tried to connect to podman running in CRC OpenShift VM.

@matejvasek
Copy link
Contributor

@tibcoplord

I'm not seeing how the socket would work if its inside the podman machine.

Ah I missed that it's VM via ssh. Then you probably need to workaround this using ssh tunnel.

@matejvasek
Copy link
Contributor

@afbjorklund symlink solution looks better that using ssh tunnel, I myself used tunnel in past because my VM didn't allow me create symlinks.

@afbjorklund
Copy link
Contributor

afbjorklund commented Sep 1, 2021

@afbjorklund symlink solution looks better that using ssh tunnel, I myself used tunnel in past because my VM didn't allow me create symlinks.

We had to use the tunnel hack for containerd/buildkitd, they are even worse than Docker when it comes to remoting.

https://minikube.sigs.k8s.io/docs/handbook/pushing/

For Podman it is mostly that the name of binary changes, once we get past the whole Varlink business in ancient history.

(and maybe --remote won't be needed anymore in 4.0)

@tibcoplord
Copy link
Author

Ah ha :-) So following the sym link approach I got to -

$ export DOCKER_HOST=ssh://root@localhost:57586 
$ docker version
Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.40
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:20 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: linux/amd64/fedora-34
 Podman Engine:
  Version:          3.3.0
  APIVersion:       3.3.0
  Arch:             amd64
  BuildTime:        2021-08-20T19:36:14Z
  Experimental:     true
  GitCommit:        
  GoVersion:        go1.16.6
  KernelVersion:    5.13.12-200.fc34.x86_64
  MinAPIVersion:    3.1.0
  Os:               linux
 Engine:
  Version:          3.3.0
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.16.6
  Git commit:       
  Built:            Fri Aug 20 19:36:14 2021
  OS/Arch:          linux/amd64
  Experimental:     true

But then in my next use-case, fabric8 maven plugin -

[INFO] --- fabric8-maven-plugin:4.4.1:build (build container image) @ test-datasink ---
[INFO] F8: Running in Kubernetes mode
[INFO] F8: Building Container image with Docker in Kubernetes mode
[ERROR] F8: Cannot create docker access object  [ssh protocol is not supported]

I'll try the tunnel approach.

@matejvasek
Copy link
Contributor

matejvasek commented Sep 1, 2021

ssh protocol is not supported

seems like flaw of the java lib

@afbjorklund
Copy link
Contributor

afbjorklund commented Sep 1, 2021

@tibcoplord

Well, if docker client doesn't support ssh and podman server doesn't support tcp - I guess you are left to tunneling unix over ssh.


I think Docker has some way of presenting a unix socket from the VM at /var/run/docker.sock, but not sure how it works:

https://web.archive.org/web/20190623234611/https://docs.docker.com/docker-for-mac/docker-toolbox/

Some kind of virtual sockets.

https://github.com/linuxkit/virtsock

@matejvasek
Copy link
Contributor

I believe podman can be set to use plain TCP.

@matejvasek
Copy link
Contributor

matejvasek commented Sep 1, 2021

e.g. podman system service --time=0 tcp:0.0.0.0:1234 --log-level=debug

@matejvasek
Copy link
Contributor

matejvasek commented Sep 1, 2021

but I don't know how the podman service process is started in VM, it my require changes to some systemd config files maybe?

@matejvasek
Copy link
Contributor

and even if it didn't support TCP you still could use socat 😄 , not much better that tunneling

@afbjorklund what problems did you encountered with tunneling?

@afbjorklund
Copy link
Contributor

afbjorklund commented Sep 1, 2021

@afbjorklund what problems did you encountered with tunneling?

It's a bunch of arcane commands, instead of an URL in a variable.

So it's hard to explain to the end users, that just want to "run Docker".

It probably doesn't work on Windows either, but then again what does...

@rhatdan
Copy link
Member

rhatdan commented Oct 31, 2021

@ashley-cui ?

@github-actions
Copy link

github-actions bot commented Dec 1, 2021

A friendly reminder that this issue had no activity for 30 days.

@drpebcak
Copy link

drpebcak commented Dec 1, 2021

This would be really nice to have!

@rhatdan rhatdan added 4.0 macos MacOS (OSX) related and removed stale-issue labels Dec 1, 2021
@rhatdan
Copy link
Member

rhatdan commented Dec 1, 2021

Agreed.

@github-actions
Copy link

github-actions bot commented Jan 1, 2022

A friendly reminder that this issue had no activity for 30 days.

@github-actions
Copy link

github-actions bot commented Feb 2, 2022

A friendly reminder that this issue had no activity for 30 days.

@dangthanh360
Copy link

dangthanh360 commented Jul 8, 2022

thanks @tibcoplord, it works well on my MacOS BigSur.

Well, with -

$ ssh -nNT -L/tmp/podman.sock:/run/user/1000/podman/podman.sock -i ~/.ssh/podman-machine-default ssh://core@localhost:64724 &
$ export DOCKER_HOST=unix:///tmp/podman.sock

The fabric8 build worked :)

[INFO] --- fabric8-maven-plugin:4.4.1:build (build container image) @ sbrt-base ---
[INFO] F8: Running in Kubernetes mode
[INFO] F8: Building Container image with Docker in Kubernetes mode
ba2c2d4a4d0c: Download complete 
7a497d63e726: Download complete 
[INFO] F8: Pulled docker.io/library/almalinux:8.4 in 12 seconds 
...

Need to find the best set of commands for the two options.

I use this command to get the port number
PODMAN_PORT=$(podman system connection ls --format="{{.URI}}" | grep "core" | grep -o -E '[0-9]+' | head -n 1)

Then run these commands to create tunnel over SSH

ssh -i ~/.ssh/podman-machine-default -nNT -L '/tmp/podman.sock:/run/user/1000/podman/podman.sock' ssh://core@localhost:$PODMAN_PORT
export DOCKER_HOST=unix:///tmp/podman.sock 

@dangthanh360
Copy link

thanks @n0rig, the simplest way as you mentioned:

For each shell, let's export the podman based SSH socket for the docker cli to use
export DOCKER_HOST="cat ~/.config/containers/containers.conf | grep -ioE "ssh://root@localhost:[0-9]+""

I use
export DOCKER_HOST="`cat ~/.config/containers/containers.conf | grep -ioE "ssh://core@localhost:[0-9]+"`"

@debu999
Copy link

debu999 commented Nov 20, 2022

please note the following actually helped to make it work. Thanks for your help. You all rock.

user is per system so is port

  1. alias docker=podman or sudo ln -sfv /usr/local/bin/podman /usr/local/bin/docker
  2. podman system connection ls -- <-- This is to get port and user details in my case its user=501 and port=49523
    Name URI Identity Default
    podman-machine-default ssh://core@localhost:49523/run/user/501/podman/podman.sock /Users/wwww/.ssh/podman-machine-default true
    podman-machine-default-root ssh://root@localhost:49523/run/podman/podman.sock /Users/wwww/.ssh/podman-machine-default false
  3. ssh -fnNT -L/tmp/podman.sock:/run/user/501/podman/podman.sock -i ~/.ssh/podman-machine-default ssh://core@localhost:49523 -o StreamLocalBindUnlink=yes
  4. export DOCKER_HOST='unix:///tmp/podman.sock'

Key Takeaway Commands to execute to make it work

  • ssh -fnNT -L/tmp/podman.sock:/run/user/ZZZ/podman/podman.sock -i ~/.ssh/podman-machine-default ssh://core@localhost:YYYYY -o StreamLocalBindUnlink=yes
  • export DOCKER_HOST='unix:///tmp/podman.sock'

@davidhsingyuchen
Copy link

davidhsingyuchen commented Nov 28, 2022

The socket address is actually printed in the output of podman machine start:

You can still connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:

	export DOCKER_HOST='unix:///Users/davidhyc/.local/share/containers/podman/machine/podman-machine-default/podman.sock'

Maybe we just need to make that info more accessible (e.g., printed by some command)? system connection ls is printing something else:

~ podman system connection ls
Name                         URI                                                         Identity                                     Default
podman-machine-default       ssh://core@localhost:63594/run/user/504/podman/podman.sock  /Users/davidhyc/.ssh/podman-machine-default  true
podman-machine-default-root  ssh://root@localhost:63594/run/podman/podman.sock           /Users/davidhyc/.ssh/podman-machine-default  false

@rhatdan
Copy link
Member

rhatdan commented Nov 30, 2022

Would you expect that same line to print out again there, if you have a podman machine running?

@davidhsingyuchen
Copy link

Would you expect that same line to print out again there, if you have a podman machine running?

Thanks for the response. I think that would work. Or anywhere under the system subcommand that makes sense .

@rhatdan
Copy link
Member

rhatdan commented Dec 3, 2022

Please open a new issue on this.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 8, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 8, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/feature Categorizes issue or PR as related to a new feature. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. machine macos MacOS (OSX) related
Projects
None yet
Development

Successfully merging a pull request may close this issue.