Add testcontainers support in the Remote adapter

[hantsy]

TestContainers¹ becomes popular in these days.

When deploying to a running server using remote adapter, we have to prepare a server for it. Nowdays most of the application can be running in docker container.

Add support to combine @TestsContainers and @Container, and Arquillian @Deployment, and make sure application server is started and ready for remote deployment.

Footnotes

1. testcontainers ↩

[bartoszmajsak]

We do have Arquillian Cube which provides similar capabilities (but it's dusty) - just to point out. No opinion on how to move forward :)

[hantsy]

but there is no activity in Arquillian Cube for years.

[phillipross]

@bartoszmajsak "dusty" is generous 😁

@hantsy have you done or seen any projects producing arquillian container extensions to use testcontainers as an arquillian container? I've done it before writing very specific code, but it would be nice to see a more general framework-level way of doing it.

[bartoszmajsak]

Ok so if I understand it right there's interest from the community to create integration with TestContainers, but there's no interest from the same community to help keep Cube up-to-date?

[phillipross]

@bartoszmajsak It depends on what your notion of "the community" might be.

There is a large community of people already using TestContainers project directly from their test frameworks, without using arquillian or arquillian-cube. This is the community that powers the demand that keeps the momentum behind the testcontainers project.

There is a separate group of people that have some interest in utilizing testcontainers along with arquillian. This is probably the group represented by @hantsy. Given the amount of overlap between arquillian-cube and testcontainers, and the staleness of cube, it would be reasonable to say that this separate group of people would prefer testcontainers over cube, and probably not see any compelling value in trying to put in the work to modernize cube.

My comment about characterizing cube as "dusty" as generous was meant to be lighthearted more than critical, but docker, compose, k8s, and the fast pace of "innovation" in the microservices and orchestration world in general have unfortunately left a big gap in what cube offers and the approaches it takes. Maybe I'm mistaken, but I feel the gap that I'm referring to is too wide for the smaller community to be able to address when testcontainers is already there and not standing still.

[bartoszmajsak]

It depends on what your notion of "the community" might be.

I specifically meant ppl interested in this feature for Arquillian. It wasn't my intention to be harsh with my previous comment, my only point is that both things will require effort and I am wondering what's best.

I feel the gap that I'm referring to is too wide for the smaller community to be able to address when testcontainers is already there and not standing still.

What's the gap we are seeing here? Would be good to identify so we can make a good decision.

[phillipross]

I specifically meant ppl interested in this feature for Arquillian. It wasn't my intention to be harsh with my previous comment, my only point is that both things will require effort and I am wondering what's best.

Right, and I realize now that I actually didn't qualify the group of people as well as I could. To try to re-characterize this group of people, and @hantsy can correct me if I'm inaccurate about this, it would be the group of people who've already adopted TestContainers and would actually like to continue to use that as test orchestration but also use arquillian to deploy to the orchestrated containers and communicate with them.

What's the gap we are seeing here? Would be good to identify so we can make a good decision.

In my mind, the gap is actually a collection of many things that would take some time to enumerate and analyze well.

Off the top of my head, Cube makes references to docker machine and boot2docker which puts into perspective how legacy it it currently is. The other thing is that it tries to mimic docker compose, and a very old docker compose at that. It would probably make sense for Cube to rearchitected to use more of a delegation-type model that delegates to an orchestration framework for managing containers... docker compose being the baseline for this.

The other major challenge is that arquillian and cube event and lifecycle models don't perfectly align with modern container use cases, so some serious thought would need to go into how to handle these mismatches, the possibility of creating ways of bridging or adapting, etc. I apologize for not being able to state any concrete/specific details at the moment, but it would require me to go back to codebases and analyze things at that level.

[hantsy]

@phillipross I have tried to user testcontainers(@Testcontainers and @Container) and Aquillian together, it did not work as expected. I found some scripts on Github to create a LoadableExtension to ensure testcontainers docker service is running before deploying, but it is not a generic framework.

[hantsy]

@bartoszmajsak I am NOT sure which is a better solution. For me, I have used Testcontainers frequently in Spring projects.
In an integration testing env, I would like it supports dependent services, such as database, key-value db(redis), message broker etc. and application servers(required by Aquillian adapters) through a simple configuration.

[phillipross]

@phillipross I have tried to user testcontainers(@Testcontainers and @Container) and Aquillian together, it did not work as expected. I found some scripts on Github to create a LoadableExtension to ensure testcontainers docker service is running before deploying, but it is not a generic framework.

Gotcha, that's pretty much what I discovered too. I had to write a loadable extension with some hardcoded descriptors and I also had to make some modifications to the arquillian container to properly handle portmapping. It works but it's not too flexible. It'd probably take a lot of work to turn it into a framework.

[bartoszmajsak]

Thanks for the valuable discussion folks! So if I'm getting this right Arquillian Cube is not flexible nor provides a model suitable for your needs. Its event flow does not fit your use cases and thus you are looking for a slim integration with TestContainers to use both tools in your integration tests?

[bartoszmajsak]

@hantsy Can you share those gists with LoadableExtension?

[kiview]

I'd also be happy to chime in if any input or advice from the Testcontainers side of things is helpful 🙂
Do I understand it correctly that the integration of Arquillian with Testcontainers would allow doing @SpringBootTest style in-process transparent-box tests for Jakarta applications? In the past, I only used Jakarta in conjunction with Testcontainers for out-of-process opaque-box testing, where the Jakarta app itself is launched as a container.

[iocanel]

@bartoszmajsak from my perspective there is definitely demand of a framework that would allow you users to test their apps in the context of a container / container orchestration technologies. However, I feel that most people favor simplicity over anything else and often resort to poor-man in-house solutions that are tailor-made to their needs vs picking up a tool like arquillian-cube. This is my take on why the need is not reflected in the community around cube. Docker/Kubernetes becoming a comodity and junit5 lowering the entry barrier to extension authoring, further pushed people to in house solutions. If we are to revive arquillian-cube I feel we need a good story around junit5 (TBH the situation between arquillian and junit5 is a bit blurry) and possibly also a good trim on the provided features.

[hantsy]

@hantsy Can you share those gists with LoadableExtension?

Personally, I just tried testcontainers @Testcontainers/@Container in Arquillian tests before, it did not work as expected.
At that moment, using Google I found this sample to integrate Testcontainers and Arquillian manually. https://github.com/kaiwinter/testcontainers-examples/blob/master/wildfly-mariadb/src/test/java/com/github/kaiwinter/testsupport/arquillian/WildflyMariaDBDockerExtension.java

I am not sure if possible to create a simple extension to check the docker service is available before performing deployments, thus make sure the testscontainers(@Testcontainers/@Container) and arquillian work seamlessly.

[phillipross]

@kiview I'm not sure if this answers your question or not... but arquillian has the its own notion of a "container" which doesn't help see things any clearer. The word "container" is heavily overloaded at this point 😅

I think a fair way to say it is that arquillian's notion of container is a wrapper around some other component runtimes that are also called containers. Examples of these might be servlet containers (such as Tomcat), JavaEE/Jakarta containers (such as glassfish, payara, wilfdly), or even more specialized containers such as CDI or EJB containers (Jboss WELD, Apache OpenWebBeans, Apache OpenEJB).

Arquillian as a framework implements an event-based lifecycle system which allows the various wrappers to manage the lifecycle of these container runtimes to start, stop, deploy/undeploy apps to the containers, etc. This was a very novel idea for doing functional testing back before the advent of OCI containers and orchestrators 😃

But to answer your question... some of these wrappers have the ability to instantiate and manage the containers inside the same java VM where the tests run, and other containers invoke separate processes outside the java VM and then deploy the apps to be tested over a network connection. In the case of utilizing testcontainers, this would pretty much fall into the later category where arquillian would be deploying to remote docker containers. Hope that helps clear things up!

[kiview]

Thanks for the detailed explanation @phillipross. Let me know if there is anything we can do from the Testcontainers side of things to support any efforts in this direction. TBH, most of the time Testcontainers would be used to spin-up other dependencies (e.g. databases) and not necessarily the container into which the application is deployed (although this is of course also possible).

But if I understand it correctly, the Remote adapter use case would mean using Testcontainers to spin up a Jakarta container, so Arquillian can deploy the application into it, correct?

What about the other use case of spinning up dependencies, is this completely unrelated to this issue?

[hantsy]

@kiview Yeah, all Jakarta EE compatible application server, such as Payara, WildFly, Open Liberty provide docker images.

When using a remote adapter, we have to prepare a running application server before running the tests. The test itself will package the test classes(defined by @Deployment) into an archive(jar or war, ear) and deploy into the running server, then run the test against the deployed test application.

What we want is simplify the progress, and let testcontainers to prepare the running server(and maybe other dependent services, such as database, redis, mq broker, etc).

Currently if we use @Testcontainers/@Container and Arquillian @Deployment, Arquillian engine do not know the existence of testcontainers container.

[phillipross]

Thanks for the detailed explanation @phillipross. Let me know if there is anything we can do from the Testcontainers side of things to support any efforts in this direction. TBH, most of the time Testcontainers would be used to spin-up other dependencies (e.g. databases) and not necessarily the container into which the application is deployed (although this is of course also possible).

Thanks for the offer of assistance!

But if I understand it correctly, the Remote adapter use case would mean using Testcontainers to spin up a Jakarta container, so Arquillian can deploy the application into it, correct?

This is absolutely correct, and probably the primary use case

What about the other use case of spinning up dependencies, is this completely unrelated to this issue?

This is also the case, that all containers would ideally be orchestrate-able from the test. I've not done it yet with testcontainers, but it seems like the ability for testcontainers to use docker compose is helpful here

[kiview]

Testcontainers can be used as a complete substitute for Docker Compose. While it can also delegate to Docker Compose, using TC directly gives more flexibility in addition to a more robust setup.

Please let me know if there is any format in which we can discuss or kick off ways for integration if this community wants to start working on this. I have a bit of experience with Jakarta as well, so I would love to see if testing for the Jakarta community can become more convenient.

[hantsy]

@bartoszmajsak @kiview

The newest Arquillian 1.7.0.Alpha11 provides an automatic deployment SPI and @BeforeDeployment make it possible to start the the testcotnaienrs container before starting deployment.

I am trying to create an example to combine the AutomaticDeployment service and testcontainers @Container. It did not work as expected.

@ExtendWith(ArquillianExtension.class)
@Testcontainers
public class GreetingResourceTest {
    private final static Logger LOGGER = Logger.getLogger(GreetingResourceTest.class.getName());

    @Container
    static GenericContainer wildfly = new GenericContainer<>(
            DockerImageName
                    .parse("quay.io/wildfly/wildfly")
                    .asCompatibleSubstituteFor("jboss/wildfly")
    )
            .withExposedPorts(8080, 9990)
            .withCreateContainerCmdModifier(cmd -> cmd.withCmd("/opt/jboss/wildfly/bin/add-user.sh admin Admin@123 --silent").exec())
            .withCommand("/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0");

    @BeforeDeployment
    public static Archive<?> beforeDeployment(Archive<?> archive) {
        Wait.forListeningPort().waitUntilReady(wildfly);
        LOGGER.log(Level.INFO, "deployment files: {}", archive.toString(true));
        return archive;
    }

    @ArquillianResource
    private URL base;

    private Client client;

    @BeforeEach
    public void setup() {
        LOGGER.info("call BeforeEach");
        this.client = ClientBuilder.newClient();
        //removed the Jackson json provider registry, due to OpenLiberty 21.0.0.1 switched to use Resteasy.
    }

    @AfterEach
    public void teardown() {
        LOGGER.info("call AfterEach");
        if (this.client != null) {
            this.client.close();
        }
    }

    @Test
    @DisplayName("Given a name:`JakartaEE` should return `Say Hello to JakartaEE`")
    public void should_create_greeting() throws MalformedURLException {
        LOGGER.log(Level.INFO, " client: {0}, baseURL: {1}", new Object[]{client, base});
        final var greetingTarget = this.client.target(new URL(this.base, "api/greeting/JakartaEE").toExternalForm());
        try (final Response greetingGetResponse = greetingTarget.request()
                .accept(MediaType.APPLICATION_JSON)
                .get()) {
            assertThat(greetingGetResponse.getStatus()).isEqualTo(200);
            assertThat(greetingGetResponse.readEntity(GreetingMessage.class).getMessage()).startsWith("Say Hello to JakartaEE");
        }
    }
}

Register the GreetingResourceDeployment via service loader.

public class GreetingResourceDeployment implements AutomaticDeployment {
    @Override
    public DeploymentConfiguration generateDeploymentScenario(TestClass testClass) {
        var war = ShrinkWrap.create(WebArchive.class)
                .addClass(GreetingMessage.class)
                .addClass(GreetingService.class)
                .addClass(GreetingResource.class)
                .addClass(JaxrsActivator.class)
                // Enable CDI (Optional since Java EE 7.0)
                .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml");

        return new DeploymentContentBuilder(war)
                .withDeployment().withTestable(false)
                .build()
                .get();
    }
}

But I am not sure how to connect the registered AutoamticDeployment to the current test class, if there are multi deployments registered.

In the above test, the deployment progress is not executed.

The complete project: https://github.com/hantsy/arquillian-autodeployment-example

[kiview]

On a first look, there are some issue with this container definition:

new GenericContainer<>(
            DockerImageName
                    .parse("quay.io/wildfly/wildfly")
                    .asCompatibleSubstituteFor("jboss/wildfly")
    )
            .withExposedPorts(8080, 9990)
            .withCreateContainerCmdModifier(cmd -> cmd.withCmd("/opt/jboss/wildfly/bin/add-user.sh admin Admin@123 --silent").exec())
            .withCommand("/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0");

.asCompatibleSubstituteFor("jboss/wildfly") is not needed.

            .withCreateContainerCmdModifier(cmd -> cmd.withCmd("/opt/jboss/wildfly/bin/add-user.sh admin Admin@123 --silent").exec())
            .withCommand("/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0");

The effect of this is that you first set the Docker CMD to /opt/jboss/wildfly/bin/add-user.sh admin Admin@123 --silent and then set it to "/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0. I think what you want is wrap the execution of those 2 commands into an entrypoint script withing the container.

[hantsy]

@kiview I have updated the command to https://github.com/hantsy/arquillian-autodeployment-example/blob/master/src/test/java/com/example/it/GreetingResourceTest.java#L43-L47.

I think the Docker startup is no problem, but I can not find any log of preparing deployment archive and deploying progress.

And how to connect the current test to the specific AutomaticDeployment if there are multi automatic deployment defined.

[poikilotherm]

Folks that subscribed to this issue might be interested in what I did with @hantsy's demo code here: hantsy/arquillian-testcontainers-example#1

It actually works! Testcontainers and Arquillian hand-in-hand! 🤝 👪

[kiview]

From a Testcontainers perspective, this looks already pretty good and also allows dynamic port mappings and remote Docker daemons (making it a very portable config).

The PoC currently uses System Properties as a integration point with Arquillian, which are always somewhat problematic if you want to avoid test pollution and keep things as idempotent as possible:
https://github.com/hantsy/arquillian-autodeployment-example/pull/1/files#diff-3d902f43d9aeb1f4cca1afd4c9a024991ef10597ecd31afaa6756d82d03d76c6R17-R20

What would be the ideal approach to configure Arquillian programmatically?

[hantsy]

My original codes used the AutomaticDeployment API(a new feature introduced in the latest Arquillian 1.7), but it seems it did not work as expected. I did not find any log of preparing the deployment archive.

And I used a Wait.forListeningPort().waitUntilReady(wildfly); in the @BeforeDeployment to ensure the container is ready before preparing deployment.

[hantsy]

What would be the ideal approach to configure Arquillian programmatically?

I think Arquillian could consider the Spring Boot 3.1 way, add a built-in way to detect the running application services from Testscontainers.

see: https://spring.io/blog/2023/06/23/improved-testcontainers-support-in-spring-boot-3-1

[kifj]

I took the afore mentioned test drive from WildflyMariaDBDockerExtension.java as base and put it into my own playground https://github.com/kifj/stomp-test/ and in the package java/x1/arquillian

This looks pretty promising, as I managed to separate the TestContainers definition from Arquillian. One could put these classes into an own library (a bit more general or make your conventions about the definition in arquillian.xml).

[jcputney]

The link in the comment above should be updated to https://github.com/kifj/stomp-test/tree/wildfly-32/src/test/java/x1/arquillian

[kiview]

@kifj Anything we can do from the Testcontainers side to support you with it? Having it as its own library while incubating the feature (or if there are other reasons why I can't be added to Arquillian directly) sound like a good idea to me.

[kifj]

@kiview if there is enough interest to turn it into a own module of testcontainers I could try to create a new module, but surely would need some support and time

[hantsy]

Spring Boot provides a @ServiceConnection on container declaration to support all popular docker images for the services that have been integrated in boot staters, from database, eg, postgres, mysql, couchbase, redis, etc, to message brokers, eg. RabbitMQ, Apache Kafka, etc. With the help of @ServiceConnection no need to configure the connection settings, check a simple example here: https://github.com/hantsy/spring6-sandbox/blob/master/boot-rabbit/src/test/java/com/example/demo/AmqpIntegrationTests.java.

Arquillian also can provides similar features to erase the connection config with remote container adapter(that running in Docker by testcontainers).

[hantsy]

@kiview Firstly, it is better to create testcontainers compatible GlassfishContainer, WildFlyContainer, OpenLibertyContainer to replace the generic container and get more customization.

[kifj]

arquillian-testcontainers.tar.gz

Hi guys,

I've extracted the code what would be in a testcontainers module, and also there's a WildflyContainer class for testing.
I've absolutely no clue about gradle, so if somebody can work out what this should look like - I would be thankful.

If there is support I can fork and try to make a PR - but I assume without a proper build configuration in gradle it would be rejected without further inspection

[hantsy]

@kifj It looks great.
Not sure if it is better to create a standalone module project for it.

And also consider the best practice from Spring Boot testcontainers integration, Quarkus DevServer, Micronaut Dev Resource.

For a remote adapter, if it does not configure connection info(host/port or connection url, username, password, etc), and detect a related testcotnainers container in the classpath, start related container, configure the connection automatically.

If the remote adapter has complete configuration of connection info, ignore the testcontainers container in the classpath.

[starksm64]

Note, we have created a separate repo to which we are adding the work that has been done internally.
https://github.com/arquillian/arquillian-testcontainers

@kifj you can look at enhancing that once the initial commit has been done.

[kifj]

Propose to close this issue with reference to https://github.com/arquillian/arquillian-testcontainers where actual work is done

[jamezp]

Seems reasonable to me. If anyone feels it needs to be reopened, please let me know.