feat: support Ryuk for the compose module #2485

mdelapenya · 2024-04-16T14:31:10Z

feat: add testcontainers labels to compose containers
feat: support reaper for compose

What does this PR do?

This PR enables the usage of Ryuk in the compose module, following the same approach than for a regular container:

check if Ryuk is enabled at the config
start the reaper, and store it in the compose struct
connect to the reaper getting a termination signal for each container service, and delegate to container lifecycle the termination of that signal
pass all the required testcontainers labels to each service in a compose file, so that they receive the same behaviors than the generic containers: be killed by Ryuk and aggregate test executions with the sessionID.
detect the networks of the compose project, adding the testcontainers labels to each of them. Each network will receive a termination signal from ryuk.

We had to rewrite a bit all the tests for compose, as they were using lots os fixed ports that could lead to errors while testing the library. Instead, we are converting them into YAML templates and getting a random free port from the OS to render the template into a temporary compose file (created by the testing.T instance). The service names has been modified in the template too, to distinguish the usage from the local compose to the api compose.

Finally, we noticed that is important to fine tune the Ryuk reconnection timeout on CI, default is 10s: with compose, for the tests in the module, I had to update it to 30s; otherwise Ryuk could die before getting the new compose container. I have not see the same behaviour in the generic container approach.

Why is it important?

Will allow not forcing the call to compose.Down, as Ryuk will take care of the compose containers.

Related issues

How to test this PR

Added tests demonstrating that:

the labels are added to all services in a compose file
with the reaper enabled, there is no need to call compose.Down
with the reaper disabled, we need to call compose.Down in the case we want to clean the state

Follow-ups

A proper abstraction on how to bootstrap the reaper would be needed, so that the compose module can call it without using a deprecated API. BTW I noticed the NewReaper func is not deprecated because the // Deprecated comment does not go first in the function comments 🤯

netlify · 2024-04-16T14:31:29Z

✅ Deploy Preview for testcontainers-go ready!

Name	Link
🔨 Latest commit	`f702fb8`
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-go/deploys/6626348935e1f900086bcc3b
😎 Deploy Preview	https://deploy-preview-2485--testcontainers-go.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

* main: fix(postgres): Fix the non-default dbname error (testcontainers#2489)

… connects to it

the first time wait is called, the context is cancelled

It will allow the tests to be more deterministic, as there could be service containers started from the local test suite with the same name as in the API test suite.

This reverts commit 336760c.

modules/compose/compose_api.go

* feat: add testcontainers labels to compose containers * feat: support reaper for compose * chore: increase ryuk reconnection timeout on CI * chore: cache containers on UP * chore: more tuning for compose * chore: more consistent assertion * chore: the compose stack asks for the reaper, but each container then connects to it * chore: use different error groups the first time wait is called, the context is cancelled * chore: the lookup method include cache checks * chore: update tests to make them deterministic * chore: rename local compose testss * chore: support returning the dynamic port in the helper function * chore: try with default reconnection timeout * feat: support removing networks from compose * chore: support naming test services with local and api It will allow the tests to be more deterministic, as there could be service containers started from the local test suite with the same name as in the API test suite. * Revert "chore: try with default reconnection timeout" This reverts commit 336760c. * fix: typo

* main: fix: don't retry on permanent APIClient errors (testcontainers#2506) feat: support overriding the default recreate options for compose (testcontainers#2511) feat: support passing io.Reader for compose files when creating a compose instance (testcontainers#2509) chore: add funding button for testcontainers (testcontainers#2510) feat: support Ryuk for the compose module (testcontainers#2485) chore(deps): bump golang.org/x/net in modules (minio, gcloud, weaviate, compose, qdrant, couchbase, k3s, milvus, mockserver, pulsar, kafka) (testcontainers#2505) fix: fallback to URL-path when parsing auth config URL without scheme (testcontainers#2488) fix(postgres): Fix the non-default dbname error (testcontainers#2489) feat: Bump default postgres version (testcontainers#2481) support Dolt (testcontainers#2177) chore: create TLS certs in a consistent manner (testcontainers#2478) chore(deps): bump idna from 3.6 to 3.7 (testcontainers#2480) Elasticsearch disable CA retrieval when ssl is disabled (testcontainers#2475) fix: handle dockerignore exclusions properly (testcontainers#2476)

* chore: start a foundational package for interacting with Docker networks * feat: add an SSH tunnel forwarding a host port to a container * fix: rename struct * chore: pass the original context to the exposeHostPorts function * chore: start tunnel using context * chore: push goroutines to the method where they are used * fix: proper eval of first network * fix: handle dockerignore exclusions properly (#2476) * chore: only include the dockerignore if it contains ignore files * fix: the inclusions must be relative to the context * docs: document the dockerignore feature * chore: only include the dockerignore file if it exists * Elasticsearch disable CA retrieval when ssl is disabled (#2475) * skip search for CACert if ssl has been turned off * add tests with and without ssl enabled * add all config keys that disable CA gen, restrict check to version 8 * rename test to match content * chore(deps): bump idna from 3.6 to 3.7 (#2480) Bumps [idna](https://github.com/kjd/idna) from 3.6 to 3.7. - [Release notes](https://github.com/kjd/idna/releases) - [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst) - [Commits](kjd/idna@v3.6...v3.7) --- updated-dependencies: - dependency-name: idna dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: create TLS certs in a consistent manner (#2478) * fix: remove suspicious filepath.Join * chore: fix lint * fix: handle error * chore: reverse assertion for lint * feat: support generating TLS certificates on the fly * chore: apply to cockroachdb * chore: support saving the cert and priv key files to disk * chore: apply to rabbitmq * chore: simplify * chore: use in redpanda module * chore: lint * chore: set validFrom internally * fix: properly use the new API in redpanda * docs: document the TLS helpers * chore: simplify WithParent to accept the struct directly * chore: use tlscert package instead * fix: use non-deprecated API * docs: update * docs: fix examples * chore: use released version of tlscert * fix: add common name for the node cert * support Dolt (#2177) * /modules/dolt: wip, kinda working * /modules/dolt: get tests passing * /{.github,.vscode,docs,mkdocs,modules,sonar-project}: use modulegen tool * /modules/dolt/{dolt.go,examples_test.go}: run linter * /modules/dolt/{dolt.go,examples_test.go}: add methods for cloning * /{docs, modules}: add with creds file * /{docs,modules}: pr feedback, cleanup * /modules/dolt/examples_test.go: remove panics, lint * chore: run mod tidy * chore: include MustConnectionString method * chore: do not use named returns * chore: perform initialisation before the container has started --------- Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com> * feat: Bump default postgres version (#2481) * Bump default postgres version * Bump to use latest pg * Bump version from non-ancient version --------- Co-authored-by: bstrausser <bstrausser@locusrobotics.com> * fix(postgres): Fix the non-default dbname error (#2489) * Fix the non-default dbname error The linked issue described in great detail an issue where we assumed everyone would use the default database user, whose home DB defaults to the postgres database. When that was not the case, the snapshots would fail silently as the user would not connect to the right database to take the commands. This PR fixes the issue by adding the dbname by default in the command, and adds a test to validate this works as intended. In addition, it also adds some logic to handle any error that does not cause the exec command to fail, such as database access failures. Run the added test to test this works as intended. Closes #2474 * Document the postgres dbname issue in the docs * fix: fallback to URL-path when parsing auth config URL without scheme (#2488) * chore(deps): bump golang.org/x/net in modules (minio, gcloud, weaviate, compose, qdrant, couchbase, k3s, milvus, mockserver, pulsar, kafka) (#2505) * chore(deps): bump golang.org/x/net in /modules/kafka Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/pulsar Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/mockserver Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/milvus Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net from 0.19.0 to 0.23.0 in /modules/k3s Bumps [golang.org/x/net](https://github.com/golang/net) from 0.19.0 to 0.23.0. - [Commits](golang/net@v0.19.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/couchbase Bumps [golang.org/x/net](https://github.com/golang/net) from 0.20.0 to 0.23.0. - [Commits](golang/net@v0.20.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/qdrant Bumps [golang.org/x/net](https://github.com/golang/net) from 0.20.0 to 0.23.0. - [Commits](golang/net@v0.20.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/compose Bumps [golang.org/x/net](https://github.com/golang/net) from 0.20.0 to 0.23.0. - [Commits](golang/net@v0.20.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/weaviate Bumps [golang.org/x/net](https://github.com/golang/net) from 0.20.0 to 0.23.0. - [Commits](golang/net@v0.20.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/gcloud Bumps [golang.org/x/net](https://github.com/golang/net) from 0.21.0 to 0.23.0. - [Commits](golang/net@v0.21.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/minio Bumps [golang.org/x/net](https://github.com/golang/net) from 0.21.0 to 0.23.0. - [Commits](golang/net@v0.21.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: support Ryuk for the compose module (#2485) * feat: add testcontainers labels to compose containers * feat: support reaper for compose * chore: increase ryuk reconnection timeout on CI * chore: cache containers on UP * chore: more tuning for compose * chore: more consistent assertion * chore: the compose stack asks for the reaper, but each container then connects to it * chore: use different error groups the first time wait is called, the context is cancelled * chore: the lookup method include cache checks * chore: update tests to make them deterministic * chore: rename local compose testss * chore: support returning the dynamic port in the helper function * chore: try with default reconnection timeout * feat: support removing networks from compose * chore: support naming test services with local and api It will allow the tests to be more deterministic, as there could be service containers started from the local test suite with the same name as in the API test suite. * Revert "chore: try with default reconnection timeout" This reverts commit 336760c. * fix: typo * chore: add funding button for testcontainers (#2510) * feat: support passing io.Reader for compose files when creating a compose instance (#2509) * feat: support passing io.Reader when creating a compose instance * docs: change title * feat: support overriding the default recreate options for compose (#2511) * feat: support overriding the default recreate options for compose * chore: validate recreation values * fix: don't retry on permanent APIClient errors (#2506) * fix: don't retry on permanent APIClient errors * fix: add more tests for un-retryable scenarios * chore: run mod tidy * chore: implement the port-forwarding correctly * chore: use new sshd image * chore: simplify channel creation to avoid allocations * fix: do not leak goroutines Detected with go.uber.org/goleak * chore: expose host internal constant * fix: update variables --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Laurent Saint-Félix <laurent.saintfelix@elastic.co> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dustin Brown <dustin@dolthub.com> Co-authored-by: Barrett Strausser <bearrito@users.noreply.github.com> Co-authored-by: bstrausser <bstrausser@locusrobotics.com> Co-authored-by: Guillaume St-Pierre <guillaume.stpierre@docker.com> Co-authored-by: Patrick Jahn <33724206+p-jahn@users.noreply.github.com>

* main: (34 commits) break: return error from Customize request option (testcontainers#2267) fix: wrong copy paste (testcontainers#2515) docs: add documentation for Exec method (testcontainers#2451) docs: document the SSHd tunnel (testcontainers#2514) fix: enhance host configuration port binding (testcontainers#2512) feat: forward host ports to a container using an SSH tunnel (testcontainers#2471) Update follow_logs.md with adding missing package (testcontainers#2513) fix: don't retry on permanent APIClient errors (testcontainers#2506) feat: support overriding the default recreate options for compose (testcontainers#2511) feat: support passing io.Reader for compose files when creating a compose instance (testcontainers#2509) chore: add funding button for testcontainers (testcontainers#2510) feat: support Ryuk for the compose module (testcontainers#2485) chore(deps): bump golang.org/x/net in modules (minio, gcloud, weaviate, compose, qdrant, couchbase, k3s, milvus, mockserver, pulsar, kafka) (testcontainers#2505) fix: fallback to URL-path when parsing auth config URL without scheme (testcontainers#2488) fix(postgres): Fix the non-default dbname error (testcontainers#2489) feat: Bump default postgres version (testcontainers#2481) support Dolt (testcontainers#2177) chore: create TLS certs in a consistent manner (testcontainers#2478) chore(deps): bump idna from 3.6 to 3.7 (testcontainers#2480) Elasticsearch disable CA retrieval when ssl is disabled (testcontainers#2475) ...

mdelapenya added 2 commits April 16, 2024 12:36

feat: add testcontainers labels to compose containers

dd25b63

feat: support reaper for compose

c6546ff

mdelapenya requested a review from a team as a code owner April 16, 2024 14:31

mdelapenya added the enhancement New feature or request label Apr 16, 2024

mdelapenya self-assigned this Apr 16, 2024

mdelapenya requested a review from kiview April 16, 2024 14:31

mdelapenya added 16 commits April 18, 2024 17:48

chore: increase ryuk reconnection timeout on CI

5897162

Merge branch 'main' into compose-reaper

d4fff51

* main: fix(postgres): Fix the non-default dbname error (testcontainers#2489)

chore: cache containers on UP

f29d297

chore: more tuning for compose

3d50c38

chore: more consistent assertion

a2bc318

chore: the compose stack asks for the reaper, but each container then…

86eb7fd

… connects to it

chore: use different error groups

79bef4b

the first time wait is called, the context is cancelled

chore: the lookup method include cache checks

df587aa

chore: update tests to make them deterministic

c0d1cf9

chore: rename local compose testss

7aad6e1

chore: support returning the dynamic port in the helper function

05e78f6

Merge branch 'main' into compose-reaper

0e0dcd4

chore: try with default reconnection timeout

336760c

feat: support removing networks from compose

51539e1

chore: support naming test services with local and api

6c00d6c

It will allow the tests to be more deterministic, as there could be service containers started from the local test suite with the same name as in the API test suite.

Revert "chore: try with default reconnection timeout"

b40e061

This reverts commit 336760c.

mdelapenya commented Apr 22, 2024

View reviewed changes

modules/compose/compose_api.go Outdated Show resolved Hide resolved

fix: typo

f702fb8

mdelapenya merged commit 66f2721 into testcontainers:main Apr 22, 2024
104 of 105 checks passed

mdelapenya deleted the compose-reaper branch April 22, 2024 10:40

mdelapenya mentioned this pull request May 3, 2024

fix(compose): avoid race conditions when caching services #2528

Merged

mdelapenya mentioned this pull request Jun 7, 2024

[Bug]: compose.dockerCompose.Up errors out with "failed to connect to reaper: dial tcp [::1]:49485: connect: connection refused: Connecting to Ryuk on localhost:49485 failed", although containers are up & running #2563

Closed

mdelapenya mentioned this pull request Jul 2, 2024

docs: document ryuk timeouts for compose #2620

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support Ryuk for the compose module #2485

feat: support Ryuk for the compose module #2485

mdelapenya commented Apr 16, 2024 •

edited

Loading

netlify bot commented Apr 16, 2024 •

edited

Loading

feat: support Ryuk for the compose module #2485

feat: support Ryuk for the compose module #2485

Conversation

mdelapenya commented Apr 16, 2024 • edited Loading

What does this PR do?

Why is it important?

Related issues

How to test this PR

Follow-ups

netlify bot commented Apr 16, 2024 • edited Loading

✅ Deploy Preview for testcontainers-go ready!

mdelapenya commented Apr 16, 2024 •

edited

Loading

netlify bot commented Apr 16, 2024 •

edited

Loading