Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Can not connect to Ryuk at 172.17.0.1:61798: Connection refused #5857

Closed
adamjshook opened this issue Sep 21, 2022 · 10 comments
Closed

Comments

@adamjshook
Copy link

adamjshook commented Sep 21, 2022

Module

Core

Testcontainers version

1.17.3

Using the latest Testcontainers version?

Yes

Host OS

Mac OS X

Host Arch

x86_64

Docker version

Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:20 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Testcontainers is unable to connect to Ryuk after it starts. I would expect it to be able to connect and launch the additional containers.

Note I am using running tests using Testcontainers inside a Docker container on Mac OS. The Docker container runs a Maven build to run the tests. This same image to run the tests is used on the CI server which uses Linux (don't know which flavor) and the build succeeds.

The command is essentially:

docker run -it --rm -u 0:0 -v /var/run/docker.sock:/var/run/docker.sock:ro -P -v ~/.m2/repository:/root/.m2/repository -v /Users/adamjshook/project:/Users/adamjshook/project -w /Users/adamjshook/project internal/ubuntu18-openjdk11-maven /bin/bash -c "mvn install"

Relevant log output

[main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
[main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
[main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.8
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 16011 MB
[main] WARN org.testcontainers.utility.ConfigurationFileImageNameSubstitutor - Image name testcontainers/ryuk:0.3.3 was substituted by configuration to internal/ryuk:latest. This approach is deprecated and will be removed in the future
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Using internal/ryuk:latest as a substitute image for testcontainers/ryuk:0.3.3 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
[main] INFO docker[internal/ryuk:latest] - Creating container for image: internal/ryuk:latest
[main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: internal/ryuk:latest, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
[main] INFO docker[internal/ryuk:latest] - Container internal/ryuk:latest is starting: 086f10a9ab6496f1cf4cab982890a69fbc09f86548a2e8bdd468ddbdb8416d5f
[main] INFO docker[internal/ryuk:latest] - Container internal/ryuk:latest started in PT1.337369S
[testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at 172.17.0.1:61798
java.net.ConnectException: Connection refused (Connection refused)
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.base/java.net.Socket.connect(Socket.java:609)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
        at java.base/java.lang.Thread.run(Thread.java:829)

Additional Information

Setting TESTCONTAINERS_RYUK_DISABLED=true allows my test cases to pass. I do not experience this on previous version 1.15.3; the tests run as expected. I hit this issue starting in 1.16.0 and is still present in 1.16.3.

@hariohmprasath
Copy link
Contributor

Hi @adamjshook,
Just curious, seems like you are using image name substitution feature to use internal/ryuk:latest image instead of testcontainers/ryuk:0.3.3, can you try directly using testcontainers/ryuk:0.3.3 and see whether you are seeing this error?

@adamjshook
Copy link
Author

@hariohmprasath

Yes, we are using that feature. I changed it to use testcontainers/ryuk:0.3.3 and I still see the error on 1.17.3 and 1.16.0. Works on 1.15.3.

@hariohmprasath
Copy link
Contributor

Hi @adamjshook,
Thanks for sharing the details, looks like ryuk container didn't start properly which would have resulted in a socket connection failure:

Here a sample code that would create the ryuk container and connect to it using Socket object, you can run this code in a test case and it would be easy to debug the failure from here:

Note: Make sure to set this environment variable (TESTCONTAINERS_RYUK_DISABLED=true) before running the unit test orelse we would run in container conflict exception.

@Test
    public void testGetValue() {

        final GenericContainer<?> ryukContainer = new GenericContainer<>("testcontainers/ryuk:0.3.4")
            .withExposedPorts(8080)
            .withCreateContainerCmdModifier(cmd -> {
                cmd.withName("testcontainers-ryuk-" + DockerClientFactory.SESSION_ID);
                cmd.withHostConfig(
                    cmd
                        .getHostConfig()
                        .withAutoRemove(true)
                        .withPrivileged(TestcontainersConfiguration.getInstance().isRyukPrivileged())
                        .withBinds(
                            new Bind(
                                DockerClientFactory.instance().getRemoteDockerUnixSocketPath(),
                                new Volume("/var/run/docker.sock")
                            )
                        )
                );
            })
            .waitingFor(Wait.forLogMessage(".*Started.*", 1));

        ryukContainer.start();

        String host = ryukContainer.getHost();
        Integer ryukPort = ryukContainer.getFirstMappedPort();
        try (Socket clientSocket = new Socket()) {
            clientSocket.connect(new InetSocketAddress(host, ryukPort), 5 * 1000);
        } catch (Exception e) {
            throw new IllegalStateException("Could not connect to Ryuk at " + host + ":" + ryukPort, e);
        }
}

@adamjshook
Copy link
Author

adamjshook commented Sep 26, 2022

I ran the provided code (with some additional logging) and in both cases the Ryuk container starts and the client attempts to connect at 172.17.0.1 but it fails on version 1.17.3.

Logs from the test case:

1.17.3

[main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
[main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
[main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
[main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.8
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 16011 MB
[main] INFO org.testcontainers.DockerClientFactory - Checking the system...
[main] INFO org.testcontainers.DockerClientFactory - ?? Docker server version should be at least 1.6.0
[main] WARN org.testcontainers.utility.ConfigurationFileImageNameSubstitutor - Image name testcontainers/ryuk:0.3.4 was substituted by configuration to testcontainers/ryuk:0.3.3. This approach is deprecated and will be removed in the future
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Using testcontainers/ryuk:0.3.3 as a substitute image for testcontainers/ryuk:0.3.4 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
[main] INFO docker[testcontainers/ryuk:0.3.3] - Creating container for image: testcontainers/ryuk:0.3.3
[main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.3, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
[main] INFO docker[testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 is starting: 7ca3969a762cde2c61d0c16a9f9488ba1eae88fdc849ae1ed0bed8d7e0f42441
[docker-java-stream--391206907] INFO ryuk - STDERR: 2022/09/26 14:01:44 Pinging Docker...
[docker-java-stream--391206907] INFO ryuk - STDERR: 2022/09/26 14:01:44 Docker daemon is available!
[docker-java-stream--391206907] INFO ryuk - STDERR: 2022/09/26 14:01:44 Starting on port 8080...
[docker-java-stream--391206907] INFO ryuk - STDERR: 2022/09/26 14:01:44 Started!
[main] INFO docker[testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 started in PT1.458051S
[main] INFO host - Connecting to Ryuk at 172.17.0.1:51042
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.107 s <<< FAILURE! - in com.foo.TestRyuk
[ERROR] testGetValue  Time elapsed: 6.061 s  <<< ERROR!
java.lang.IllegalStateException: Could not connect to Ryuk at 172.17.0.1:51042
        at com.foo.TestRyuk.testGetValue(TestRyuk.java:48)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at com.foo.TestRyuk.testGetValue(TestRyuk.java:46)

1.15.3

[main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
[main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/apache-maven/bin, /java/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin])
[main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
[main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
[main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.8
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 16011 MB
[main] INFO org.testcontainers.DockerClientFactory - Checking the system...
[main] INFO org.testcontainers.DockerClientFactory - ?? Docker server version should be at least 1.6.0
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
[main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: alpine:3.5, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
[main] INFO org.testcontainers.DockerClientFactory - ?? Docker environment should have more than 2GB free disk space
[main] WARN org.testcontainers.utility.ConfigurationFileImageNameSubstitutor - Image name testcontainers/ryuk:0.3.4 was substituted by configuration to testcontainers/ryuk:0.3.3. This approach is deprecated and will be removed in the future
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Using testcontainers/ryuk:0.3.3 as a substitute image for testcontainers/ryuk:0.3.4 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
[main] INFO docker[testcontainers/ryuk:0.3.3] - Creating container for image: testcontainers/ryuk:0.3.3
[main] INFO docker[testcontainers/ryuk:0.3.3] - Starting container with ID: 52d67782ec5e99098b1411cf6a8e22e597ed51a95605fafa9e59a019832a4e0c
[main] INFO docker[testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 is starting: 52d67782ec5e99098b1411cf6a8e22e597ed51a95605fafa9e59a019832a4e0c
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Pinging Docker...
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Docker daemon is available!
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Starting on port 8080...
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Started!
[main] INFO docker[testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 started in PT0.947427S
[main] INFO host - Connecting to Ryuk at 172.17.0.1:55003
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 New client connected: 172.17.0.1:55040
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 EOF
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Client disconnected: 172.17.0.1:55040
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Received the first connection
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.614 s - in com.foo.TestRyuk

@kiview
Copy link
Member

kiview commented Sep 26, 2022

Testcontainers inside a Docker container on Mac OS

I think you are running into this upstream issue: #4395
Please try out the workaround specified here and let us know if it helps.

@hariohmprasath Thanks a lot for your supportt in triaging this issue 🙇

@kiview kiview self-assigned this Sep 26, 2022
@adamjshook
Copy link
Author

let us know if it helps.

I tried the workaround of mounting /var/run/docker.sock.raw:/var/run/docker.sock instead of /var/run/docker.sock:/var/run/docker.sock and I got the same Connection refused result.

@kiview
Copy link
Member

kiview commented Sep 28, 2022

@adamjshook Unfortunately I don't know what else we can do here, it probably requires more detailed debugging of the Docker environment. Did you verify that /var/run/docker.sock.raw exists and that you can interact with it?

Can you please share your finding, that the workaround does not work for you in #4395? I will close this issue as a duplicate.

@kiview kiview closed this as not planned Won't fix, can't repro, duplicate, stale Sep 28, 2022
@bigunyak
Copy link

bigunyak commented Dec 2, 2023

I have the same issue with fresh colima installation and testcontainers-1.17.5.
I'm not sure how to try the workaround you suggested @kiview with a different socket mounting?

@eddumelendez
Copy link
Member

@bigunyak take a look at our documentation. Also, Testcontainers Desktop with embedded runtime works OOTB with Testcontainers

@bigunyak
Copy link

bigunyak commented Dec 3, 2023

@eddumelendez , starting colima with --network-address and setting TESTCONTAINERS_HOST_OVERRIDE like it now mentioned in the docs, removed all warnings about Ryuk that I had. Thank you very much! ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants