Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support podman as alternative to docker #336

Closed
candlerb opened this issue Jul 29, 2021 · 15 comments
Closed

support podman as alternative to docker #336

candlerb opened this issue Jul 29, 2021 · 15 comments
Labels
podman Issues regarding podman.

Comments

@candlerb
Copy link
Contributor

I wondered if you had considered supporting podman for running containers, as an alternative backend to docker?

Using podman (and buildah for creating container images) lets you get rid of the docker daemon entirely.

There is a go API. This does involve running a podman service: but unlike docker, the service runs as your own userid, with no elevated privileges, and can be started and stopped by the user themselves.

@candlerb
Copy link
Contributor Author

UPDATE: it looks like very little effort should be required to work with podman 3+. It is compatible with the docker API, even to the point where you can run native "docker compose". That works as follows:

  • Start a global podman service: systemctl enable --now podman.socket
  • For RHEL/CentOS/Arch there's a package "podman-docker" which installs symlinks such that /var/run/docker.sock points to the podman socket.
  • For Debian/Ubuntu you need to set DOCKER_HOST=unix:///run/podman/podman.sock: see https://thesynack.com/posts/docker-compose-podman/

If you're not using docker compose, you don't need the global podman service anyway. Start a user service instead:

$ systemctl start --user podman.socket

This listens on /run/user/1000/podman/podman.sock (for example).

However, the DOCKER_HOST environment variable doesn't appear to be completely honoured by testcontainers-go. If I run a test without it, I get a failure as expected:

$ go test .
--- FAIL: TestNginxLatestReturn (0.00s)
    main_test.go:25: failed to create container: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8cfb77]

goroutine 19 [running]:
testing.tRunner.func1.2(0x9445c0, 0xd44680)
	/usr/local/go/src/testing/testing.go:1143 +0x332
testing.tRunner.func1(0xc000082a80)
	/usr/local/go/src/testing/testing.go:1146 +0x4b6
panic(0x9445c0, 0xd44680)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
example.TestNginxLatestReturn(0xc000082a80)
	/home/nsrc/tmp/testcontainers/main_test.go:27 +0x257
testing.tRunner(0xc000082a80, 0x9f8348)
	/usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1238 +0x2b3
FAIL	example	0.007s
FAIL

(Aside: the ugly nil pointer dereference is because the sample code uses t.Error() when it should use t.Fatal())

With DOCKER_HOST set, it gets further, but fails later:

$ DOCKER_HOST=unix:///run/user/1000/podman/podman.sock go test .
--- FAIL: TestNginxLatestReturn (2.40s)
    main_test.go:25: failed to create container: creating reaper failed: Error response from daemon: container create: statfs /var/run/docker.sock: no suy
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8cfb77]

goroutine 19 [running]:
testing.tRunner.func1.2(0x9445c0, 0xd44680)
	/usr/local/go/src/testing/testing.go:1143 +0x332
testing.tRunner.func1(0xc000082a80)
	/usr/local/go/src/testing/testing.go:1146 +0x4b6
panic(0x9445c0, 0xd44680)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
example.TestNginxLatestReturn(0xc000082a80)
	/home/nsrc/tmp/testcontainers/main_test.go:27 +0x257
testing.tRunner(0xc000082a80, 0x9f8348)
	/usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1238 +0x2b3
FAIL	example	2.404s
FAIL

It appears that NewReaper has a hard-coded path to /var/run/docker.sock

@candlerb
Copy link
Contributor Author

Here is a POC patch for that problem:

diff --git a/reaper.go b/reaper.go
index be9818f..708b39a 100644
--- a/reaper.go
+++ b/reaper.go
@@ -7,6 +7,7 @@ import (
        "github.com/docker/go-connections/nat"
        "github.com/testcontainers/testcontainers-go/wait"
        "net"
+       "os"
        "strings"
        "sync"
        "time"
@@ -55,6 +56,11 @@ func NewReaper(ctx context.Context, sessionID string, provider ReaperProvider, r

        listeningPort := nat.Port("8080/tcp")

+       sock := "/var/run/docker.sock"
+       docker_host := os.Getenv("DOCKER_HOST")
+       if docker_host[0:8] == "unix:///" {
+               sock = docker_host[7:]
+       }
        req := ContainerRequest{
                Image:        reaperImage(reaperImageName),
                ExposedPorts: []string{string(listeningPort)},
@@ -64,7 +70,7 @@ func NewReaper(ctx context.Context, sessionID string, provider ReaperProvider, r
                },
                SkipReaper: true,
                BindMounts: map[string]string{
-                       "/var/run/docker.sock": "/var/run/docker.sock",
+                       sock: "/var/run/docker.sock",
                },
                AutoRemove: true,
                WaitingFor: wait.ForListeningPort(listeningPort),

However, I'm not providing this as a PR because it's still not running under my environment.

Note that I'm using Ubuntu 18.04 on the base host. The podman package (3.0.1 from the kubic repo) uses a systemd feature Type=exec which was introduced in systemd 240, but Ubuntu 18.04 has systemd 237. That's OK; I can work around this by running podman in a separate terminal instead of under systemd.

# first terminal
$ podman system service --time=0 unix:///tmp/podman.sock

# second terminal
$ DOCKER_HOST=unix:///tmp/podman.sock go test .

But there is still an error coming from podman or crun: "must provide at least one stream to attach to"

$ podman system service --time=0 unix:///tmp/podman.sock
ERRO[0004] error attaching to container 0f3f81628e7f1d97b3a1172970ca89998eb0d46f6065c896ae59b347f4aa1688 exec session e2cc62240c65de72b24e4882867b056fc24fb0de4017a3a74411d767ec678060: must provide at least one stream to attach to: invalid argument
$ DOCKER_HOST=unix:///tmp/podman.sock go test .
2021/07/29 10:42:58 Starting container id: 0f3f81628e7f image: quay.io/testcontainers/ryuk:0.2.3
2021/07/29 10:42:59 Waiting for container id 0f3f81628e7f image: quay.io/testcontainers/ryuk:0.2.3
--- FAIL: TestNginxLatestReturn (0.32s)
    main_test.go:25: failed to create container: creating reaper failed: could not start container: host port waiting failed: Error response from daemon: must provide at least one stream to attach to: invalid argument
FAIL
FAIL	example	0.323s
FAIL

@candlerb
Copy link
Contributor Author

strace shows the HTTP exchanges to podman as:

[pid 24161] read(9, "HEAD /_ping HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\n\r\n", 4096) = 70
[pid 24161] write(9, "HTTP/1.1 200 OK\r\nApi-Version: 1.40\r\nBuilder-Version: \r\nBuildkit-Version: \r\nCache-Control: no-cache\r\nDocker-Experimental: true\r\nLibpod-Api-Version: 3.0.0\r\nLibpod-Buildah-Version: 1.19.4\r\nPragma: no-cache\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\n\r\n", 273 <unfinished ...>
[pid 24152] <... read resumed> "GET /v1.40/networks HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\n\r\n", 4096) = 78
[pid 24152] write(9, "HTTP/1.1 200 OK\r\nApi-Version: 1.40\r\nContent-Type: application/json\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nContent-Length: 479\r\n\r\n[{\"Name\":\"reaper_default\",\"Id\":\"ebfa7be6426bee33e85e620ee17643cab01caee87998f586ff228340fcb18778\",\"Created\":\"2021-07-29T10:11:47.519629477+01:00\",\"Scope\":\"local\",\"Driver\":\"bridge\",\"EnableIPv6\":false,\"IPAM\":{\"Driver\":\"default\",\"Options\":null,\"Config\":[{\"Subnet\":\"10.88.2.0/24\",\"Gateway\":\"10.88.2.1\"}]},\"Internal\":false,\"Attachab"..., 664) = 664
[pid 24155] <... read resumed> "GET /v1.40/networks HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\n\r\n", 4096) = 78
[pid 24155] write(9, "HTTP/1.1 200 OK\r\nApi-Version: 1.40\r\nContent-Type: application/json\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nContent-Length: 479\r\n\r\n[{\"Name\":\"reaper_default\",\"Id\":\"ebfa7be6426bee33e85e620ee17643cab01caee87998f586ff228340fcb18778\",\"Created\":\"2021-07-29T10:11:47.519629477+01:00\",\"Scope\":\"local\",\"Driver\":\"bridge\",\"EnableIPv6\":false,\"IPAM\":{\"Driver\":\"default\",\"Options\":null,\"Config\":[{\"Subnet\":\"10.88.2.0/24\",\"Gateway\":\"10.88.2.1\"}]},\"Internal\":false,\"Attachab"..., 664) = 664
[pid 24158] <... read resumed> "GET /v1.40/images/quay.io/testcontainers/ryuk:0.2.3/json HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\n\r\n", 4096) = 115
[pid 24152] write(9, "HTTP/1.1 200 OK\r\nApi-Version: 1.40\r\nContent-Type: application/json\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nContent-Length: 971\r\n\r\n{\"Id\":\"sha256:64849fd2d4645159f8c839015f7754dac1207181ca65cf2a10f08597ac5baa3e\",\"RepoTags\":[\"quay.io/testcontainers/ryuk:0.2.3\"],\"RepoDigests\":[\"quay.io/testcontainers/ryuk@sha256:bb5a635cac4bd96c93cc476969ce11dc56436238ec7cd028d0524462f4739dd9\"],\"Parent\":\"\",\"Comment\":\"\",\"Created\":\"2019-02-04T15:39:36.450706328Z\",\"Container\":"..., 1156 <unfinished ...>
[pid 24158] read(9, "GET /v1.40/networks/reaper_default?verbose=true HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\n\r\n", 4096) = 106
[pid 24158] write(9, "HTTP/1.1 200 OK\r\nApi-Version: 1.40\r\nContent-Type: application/json\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nContent-Length: 477\r\n\r\n{\"Name\":\"reaper_default\",\"Id\":\"ebfa7be6426bee33e85e620ee17643cab01caee87998f586ff228340fcb18778\",\"Created\":\"2021-07-29T10:11:47.519629477+01:00\",\"Scope\":\"local\",\"Driver\":\"bridge\",\"EnableIPv6\":false,\"IPAM\":{\"Driver\":\"default\",\"Options\":null,\"Config\":[{\"Subnet\":\"10.88.2.0/24\",\"Gateway\":\"10.88.2.1\"}]},\"Internal\":false,\"Attachabl"..., 662) = 662
[pid 24152] read(9, "POST /v1.40/containers/create HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 2099\r\nContent-Type: application/json\r\n\r\n{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"8080/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[],\"Cmd\":null,\"Image\":\"quay.io/testcontainers/ryuk:0.2.3\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{\"org.testcontainers.golang\":\"true\",\"org.testcontai"..., 4096) = 2241
[pid 24156] write(9, "HTTP/1.1 201 Created\r\nApi-Version: 1.40\r\nContent-Type: application/json\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nContent-Length: 88\r\n\r\n{\"Id\":\"f28f733472cee2ed9b65a334b916179a68528e6ee80701cb684b64f94343c570\",\"Warnings\":[]}\n", 277 <unfinished ...>
[pid 24154] read(9, "POST /v1.40/containers/f28f733472cee2ed9b65a334b916179a68528e6ee80701cb684b64f94343c570/start HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 0\r\nContent-Type: text/plain\r\n\r\n", 4096) = 197
[pid 24154] write(9, "HTTP/1.1 204 No Content\r\nApi-Version: 1.40\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\n\r\n", 140 <unfinished ...>
[pid 24161] <... read resumed> "GET /v1.40/containers/f28f733472cee2ed9b65a334b916179a68528e6ee80701cb684b64f94343c570/json HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\n\r\n", 4096) = 150
[pid 24376] write(3, "GET /_ping HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\n\r\n", 69 <unfinished ...>
[pid 24154] <... read resumed> "GET /_ping HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\n\r\n", 4096) = 69
[pid 24154] write(13, "HTTP/1.1 200 OK\r\nApi-Version: 1.40\r\nBuilder-Version: \r\nBuildkit-Version: \r\nCache-Control: no-cache\r\nDocker-Experimental: true\r\nLibpod-Api-Version: 3.0.0\r\nLibpod-Buildah-Version: 1.19.4\r\nPragma: no-cache\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\n\r\nOK", 335 <unfinished ...>
[pid 24375] <... read resumed> "HTTP/1.1 200 OK\r\nApi-Version: 1.40\r\nBuilder-Version: \r\nBuildkit-Version: \r\nCache-Control: no-cache\r\nDocker-Experimental: true\r\nLibpod-Api-Version: 3.0.0\r\nLibpod-Buildah-Version: 1.19.4\r\nPragma: no-cache\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\n\r\nOK", 4096) = 335
[pid 24161] write(9, "HTTP/1.1 200 OK\r\nApi-Version: 1.40\r\nContent-Type: application/json\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nTransfer-Encoding: chunked\r\n\r\n800\r\n{\"Id\":\"f28f733472cee2ed9b65a334b916179a68528e6ee80701cb684b64f94343c570\",\"Created\":\"2021-07-29T11:11:27.213337212+01:00\",\"Path\":\"/app\",\"Args\":[\"/app\"],\"State\":{\"Status\":\"running\",\"Running\":true,\"Paused\":false,\"Restarting\":false,\"OOMKilled\":false,\"Dead\":false,\"Pid\":24371,\"ExitCode\":0,\"Error\":\"\",\"StartedAt\":\"2021-07"..., 3960) = 3960
[pid 24151] <... read resumed> "GET /v1.40/containers/f28f733472cee2ed9b65a334b916179a68528e6ee80701cb684b64f94343c570/json HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\n\r\n", 4096) = 150
[pid 24151] write(9, "HTTP/1.1 200 OK\r\nApi-Version: 1.40\r\nContent-Type: application/json\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nTransfer-Encoding: chunked\r\n\r\n800\r\n{\"Id\":\"f28f733472cee2ed9b65a334b916179a68528e6ee80701cb684b64f94343c570\",\"Created\":\"2021-07-29T11:11:27.213337212+01:00\",\"Path\":\"/app\",\"Args\":[\"/app\"],\"State\":{\"Status\":\"running\",\"Running\":true,\"Paused\":false,\"Restarting\":false,\"OOMKilled\":false,\"Dead\":false,\"Pid\":24371,\"ExitCode\":0,\"Error\":\"\",\"StartedAt\":\"2021-07"..., 3960) = 3960
[pid 24151] <... read resumed> "POST /v1.40/containers/f28f733472cee2ed9b65a334b916179a68528e6ee80701cb684b64f94343c570/exec HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 392\r\nContent-Type: application/json\r\n\r\n{\"User\":\"\",\"Privileged\":false,\"Tty\":false,\"AttachStdin\":false,\"AttachStderr\":false,\"AttachStdout\":false,\"Detach\":false,\"DetachKeys\":\"\",\"Env\":null,\"WorkingDir\":\"\",\"Cmd\":[\"/bin/sh\",\"-c\",\"true \\u0026\\u0026 (\\n\\t\\t\\t\\t\\tcat /proc/net/tcp* | awk '{print $2}' | grep -i :1f90 ||\\n\\t\\t\\t\\t\\tnc -vz -w 1 localhost 80"..., 4096) = 596
[pid 24151] write(9, "HTTP/1.1 201 Created\r\nApi-Version: 1.40\r\nContent-Type: application/json\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nContent-Length: 74\r\n\r\n{\"Id\":\"d31d0e561a6199af8abd5ae3f0e5569b41eff7c1db7dad77c656bc7e618c0913\"}\n", 263) = 263
[pid 24151] read(9, "POST /v1.40/exec/d31d0e561a6199af8abd5ae3f0e5569b41eff7c1db7dad77c656bc7e618c0913/start HTTP/1.1\r\nHost: docker\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 29\r\nContent-Type: application/json\r\n\r\n{\"Detach\":false,\"Tty\":false}\n", 4096) = 227
[pid 24151] write(9, "HTTP/1.1 500 Internal Server Error\r\nApi-Version: 1.40\r\nContent-Type: application/json\r\nLibpod-Api-Version: 3.0.0\r\nServer: Libpod/3.0.0 (linux)\r\nDate: Thu, 29 Jul 2021 10:11:27 GMT\r\nContent-Length: 120\r\n\r\n{\"cause\":\"invalid argument\",\"message\":\"must provide at least one stream to attach to: invalid argument\",\"response\":500}\n", 324) = 324

@rnorth
Copy link
Member

rnorth commented Aug 3, 2021

I'll put my 2 cents in: testcontainers/testcontainers-java#2088 describes the direction we should take. TLDR: Testcontainers should continue to use the docker API, and Podman should implement that API.

I think the Podman team have done a lot of great work in Podman's support for the Docker API, but evidently this must provide at least one stream to attach to error indicates a gap. I'd suggest that you report this to the Podman team and then, when it's resolved in Podman, Testcontainers should be able to work seamlessly.

@fpozzobon
Copy link

Hi, any update on this thread?
It seems that if we replace /var/run/docker.sock with DOCKER_HOST environment variable, this should make Podman compatible.

@paralin
Copy link

paralin commented Mar 23, 2022

This is the only place I found that "must provide at least one stream to attach to" error mentioned.

I fixed it by setting the ErrorStream and OutputStream variables on ContainerRunOptions:

		pentities.ContainerRunOptions{
			Rm:   true,
			Spec: specGen,

			ErrorStream:  os.Stdout,
			OutputStream: os.Stdout,
		},

Maybe this helps.

@prskr
Copy link
Contributor

prskr commented Apr 11, 2022

I'm working on a PR that also includes the change @paralin mentioned and a few more things e.g. to also run all tests with Podman. There's another PR that already covers the support of the DOCKER_HOST environment variable.
As soon as we managed to merge both PRs Podman should be well supported including support for ryuk and docker-compose.

I cannot say how long it will take to stabilize everything but I hope soon ™️ 😄

@tommyalatalo
Copy link

Any progress on this?

1 similar comment
@tommyalatalo
Copy link

Any progress on this?

@mdelapenya
Copy link
Member

Any progress on this?

Hey @altosys it was merged in #414. Not released yet though, although it will come soon

@tommyalatalo
Copy link

tommyalatalo commented Aug 13, 2022

Any progress on this?

Hey @altosys it was merged in #414. Not released yet though, although it will come soon

Yeah I was looking for documentation on how to use testcontainers-go with podman and didn't find any, so I assumed it wasn't quite ready. Do you have a time frame that is more specific than "soon"? It's very hard for me to tell what that means since the discussion here has been quiet since April.

@mdelapenya
Copy link
Member

I'm currently on summer PTO, end of month could be an accurate estimation

@prskr
Copy link
Contributor

prskr commented Aug 13, 2022

I'll try to also add some docs till then - end of August sounds reasonable to write at least some hints/examples

@mdelapenya
Copy link
Member

After the release of v0.14.0, tc-go experimentally supports Podman, including some docs (https://golang.testcontainers.org/features/using_podman/). So I'd like to send big kudos to @baez90 for this massive contribution.

OTOH, I'd like to stand with what @rnorth mentioned in #336 (comment) too: if Podman supports Docker APIs, then it should work with tc-go, which uses Docker APIs.

Said that, I think this issue can be closed, and if there is anything else missing, I'd suggest opening separate issues for each missing topic.

Big thanks to you all for participating in the community!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
podman Issues regarding podman.
Projects
None yet
Development

No branches or pull requests

8 participants