Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't connect to Ryuk when using when running in container and using Docker-Socket-Mounting on Docker Desktop #4395

Closed
jrussellsmyth opened this issue Aug 25, 2021 · 16 comments · Fixed by #6282

Comments

@jrussellsmyth
Copy link

When running testcontainer based test within another container (vscode devcontainer)

Tests that executed flawlessly on 1.15.3 now fail attempting to start the compose configuration

java.lang.IllegalStateException: Could not connect to Ryuk at 172.17.0.1:58364
at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:227) ~[testcontainers-1.16.0.jar:?]
at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:204) ~[testcontainers-1.16.0.jar:?]
at org.testcontainers.containers.DockerComposeContainer.(DockerComposeContainer.java:135) ~[testcontainers-1.16.0.jar:?]
at org.testcontainers.containers.DockerComposeContainer.(DockerComposeContainer.java:119) ~[testcontainers-1.16.0.jar:?]

We did attempt disabling Ryak, but that just results in the same error attempting to connect to the first container.

Docker Desktop 3.6.0
macOS Mojave version 10.14.6
intel mac

@jrussellsmyth
Copy link
Author

jrussellsmyth commented Aug 25, 2021

Created a minimal test case for this here demonstrating expectations with 1.15.3 and failure with 1.16.0
https://github.com/jrussellsmyth/testcontainer-problem

@kiview
Copy link
Member

kiview commented Aug 26, 2021

Hi @jrussellsmyth,
thanks a lot for reporting this issue and providing us with a reproducer.

This problem also occurs on Windows in WSL2. In addition, it also happens with GenericContainer. Therefore, I assume this has nothing to do with DockerComposeContainer or Docker for Mac and it seems there is a regression in 1.16.0 for the Docker Wormhole Pattern (i.e., socket mounting) in general.

@Nayshins
Copy link

Nayshins commented Nov 4, 2021

Is there a timeline for this to be fixed? I'm stuck in a bit of a bind here of needing to be on 1.16.+ for M1 support, but this issue is blocking me from using this version with docker for mac.

If anyone has a suggestion for a workaround that would be greatly appreciated!

@kiview
Copy link
Member

kiview commented Nov 4, 2021

No ETA yet, see the PR #4396 and the possible implications from upstream Docker here:
docker/for-mac#5588 (comment)

What is your use case for running the tests inside of Docker Compose in the first place? Maybe if we think about this, we can come up with a workaround or alternative setup.

@jonathan-meier
Copy link

Our use case is a development environment inside a docker container using the VS Code Remote Containers extension running on a local instance of Docker Desktop for Windows (WSL2). We have tests using testcontainers that need to be runnable in that environment. This works fine with testcontainers up to version 1.15.3, but we can't update to version 1.16.0 due to this issue.

Any advice on how we might work around this issue until PR #4396 is merged or a more general solution is available?

@kiview
Copy link
Member

kiview commented Nov 30, 2021

Hi @jonathan-meier, as you can see from the comment in the upstream issue, there might exist different sockets that need to be mounted for the docker-wormhole pattern to work again. However, I did not yet figure out how to make it work on Docker for Desktop with WSL2.

As a temporary workaround, you could try out to consume the PR as a Jitpack dependency:
https://www.testcontainers.org/jitpack_dependencies/

It would also be interesting to know, if this mitigates the issue for your setup as well.

@ggili
Copy link

ggili commented Dec 1, 2021

hi all

I came across to this issue which I think its related to mine (opened last week). Doe I am. using also a mac , but it happens as well on linux CentOS
#4685
Does the PR on docker is providing a fix for both versions? (linux and mac)?

@jonathan-meier
Copy link

Hi @kiview, thanks a lot for your answer. I saw that there might be a workaround using a different socket, however, I'm not sure if we could use it, because our configuration needs to be platform independent, i.e. work on Docker Desktop with WSL2 as well as Docker on Linux.

Using jitpack as you suggest, I can confirm that PR #4396 solves the problem for our setup, so we would love to see it make its way into the next released version.

@kiview
Copy link
Member

kiview commented Dec 3, 2021

@ggili It is very unlikely that this issue appears on Linux as well since it is related to the VM setup of Docker Desktop.

@jonathan-meier Thanks for confirming that the PR solves the issue for you, we will look into merging it.

@ggili
Copy link

ggili commented Dec 3, 2021

@kiview
Yes indeed. I was too fast on conclusion. On mac that might be a solution . Was investigating our gitlabrunner issue further on my mac.

However I just tried on our centos gitlabrunner. Indeed -p 1234:80 or -p 80 doesn't matter. The containers cannot talk to eachother via the gateway. Where on Mac they can.

`docker run -dp 1234:80 docker/getting-started

"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "423fde3a0e3896a69512aa3c8fa2016a1d5e2c976707b423e8accc075e3c17e7",
"EndpointID": "d46fd61a22e4fcceb9c6229405d5db747248981d8f56de4f1ad112c3adff3197",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.3",
`

`docker run -it maven:3.3-jdk-8 /bin/bash

Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "423fde3a0e3896a69512aa3c8fa2016a1d5e2c976707b423e8accc075e3c17e7",
"EndpointID": "ecdd25978e06d056ff62a51fc82910bd34cf638439aa3feafaed975b3460c067",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.5",
`

curl 172.17.0.1:1234 - works well from host, but not from maven:3.3-jdk-8 container. Only via curl 172.17.0.5:8080

I will try digging a bit later, but for me seems that is also a regression on docker. Or there might be something on our Centos which blows things up.

@kiview
Copy link
Member

kiview commented Dec 3, 2021

Since I remember a regression on Fedora from some time ago, it might be about the firewalld configuration on CentOS.

@jonathan-meier
Copy link

I just saw that version 1.16.3 was released, but sadly without PR #4396. I understand that there is some risk involved with merging PR #4396 as is, though what would you think about hiding the workaround behind an opt-in configuration flag?

@kiview kiview pinned this issue Jun 17, 2022
@BoraxTheClean
Copy link

On Docker Desktop for Mac (M1) 4.9.0, I was able to work around this issue by mounting the docker socket into the container in this fashion: -v /var/run/docker.sock.raw:/var/run/docker.sock

@bozo-ivetic-cif
Copy link

On Docker Desktop for Mac (M1) 4.9.0, I was able to work around this issue by mounting the docker socket into the container in this fashion: -v /var/run/docker.sock.raw:/var/run/docker.sock

I run into the same problem and this workaround did the trick. macOS 13, testcontainers:1.17.5 and Docker Desktop 4.13.1 (90346)

@kiview kiview changed the title 1.16.0 Breaks compose based testcontainers on docker desktop for mac Can't connect to Ryuk when using when running in container and using Docker-Socket-Mounting on Docker Desktop Nov 30, 2022
@kiview
Copy link
Member

kiview commented Dec 2, 2022

I can confirm that -v /var/run/docker.sock.raw:/var/run/docker.sock also work in WSL2 now. Therefore, I recommend to extend the docs with this information and then close this issue.

@jxblum
Copy link

jxblum commented May 12, 2023

/var/run/docker.sock.raw is not even a file on macOS 13.3.1 with Docker Desktop 4.19.0 (106363) using Testcontainers 1.18.1:

$ ls -la /var/run/docker.sock.raw
ls: /var/run/docker.sock.raw: No such file or directory

I still see:

Caused by: java.lang.IllegalStateException: Could not connect to Ryuk at 172.17.0.1:59391
	at org.testcontainers.utility.RyukResourceReaper.maybeStart(RyukResourceReaper.java:132)
	at org.testcontainers.utility.RyukResourceReaper.init(RyukResourceReaper.java:42)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:224)
	at org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:104)
	at com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:109)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:321)
	at example.app.crm.config.TestcontainersCassandraConfiguration.cassandraContainer(TestcontainersCassandraConfiguration.java:71)
...
..
.

I am attempting to run tests using Testcontainers to bootstrap a Apache Cassandra 3.11.15 container with:

$ docker run -it --mount type=bind,source="$(pwd)",target=/source-from-github -v /var/run/docker.sock:/var/run/docker.sock -v /usr/local/bin/docker:/usr/local/bin/docker eclipse-temurin:17.0.6_10-jdk-focal /bin/bash

Essentially, bootstrapping the Cassandra container with:

	private static final DockerImageName CASSANDRA_DOCKER_IMAGE_NAME = DockerImageName.parse("cassandra:3.11.15");

	private static final String LOCAL_DATACENTER_NAME = "datacenter1";

	private static final int CASSANDRA_DEFAULT_PORT = 9042;

	@Bean("CassandraContainer")
	GenericContainer<?> cassandraContainer(Environment environment) {

		GenericContainer<?> cassandraContainer = newEnvironmentOptimizedCassandraContainer();

71:		cassandraContainer.start();

		return withCassandraServer(cassandraContainer, environment);
	}

	private @NonNull GenericContainer<?> newCassandraContainer() {

		return new CassandraContainer<>(CASSANDRA_DOCKER_IMAGE_NAME)
			.withInitScript(CASSANDRA_SCHEMA_CQL)
			//.withInitScript(CASSANDRA_INIT_CQL)
			.withExposedPorts(CASSANDRA_DEFAULT_PORT)
			.withReuse(true);
	}

	private @NonNull GenericContainer<?> newEnvironmentOptimizedCassandraContainer() {

		return newCassandraContainer()
			.withEnv("CASSANDRA_SNITCH", "SimpleSnitch")
			.withEnv("HEAP_NEWSIZE", "128M")
			.withEnv("MAX_HEAP_SIZE", "1024M")
			.withEnv("JVM_OPTS", "-Dcassandra.skip_wait_for_gossip_to_settle=0 -Dcassandra.initial_token=0");
	}

	private @NonNull CqlSession newCqlSession(@NonNull GenericContainer<?> cassandraContainer) {

		return CqlSession.builder()
			.addContactPoint(resolveContactPoint(cassandraContainer))
			.withLocalDatacenter(LOCAL_DATACENTER_NAME)
			.build();
	}

	private @NonNull InetSocketAddress resolveContactPoint(@NonNull GenericContainer<?> cassandraContainer) {
		return new InetSocketAddress(cassandraContainer.getHost(),
			cassandraContainer.getMappedPort(CASSANDRA_DEFAULT_PORT));
	}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment