Load Testcontainers parent first and start the containers with the Platform CL as the TCCL #42560

gsmet · 2024-08-14T21:52:12Z

Note

This PR is part of a coordinated effort to address the OOM reported in #42471:

This is the last piece of the puzzle to make #42471 actually pass.

TBH, I'm not exactly excited about adding testcontainers always but for now that's the only solution I could find.

There are two aspects to it:

Start the containers with the TCCL set to the Platform CL to avoid having the TCCL of the threads being a QuarkusClassLoader
Load Testcontainers base classes parent first: some are long lived so we need to avoid loading them with the QuarkusClassLoader

The problem with the latter is that we then need to make testcontainers a dependency of quarkus-test-common so that it appears in the test classpath.

There might be a better way to do it in the future with the work @holly-cummins has started but that's how it is for now (and I'm not entirely sure Holly's work would be enough to solve this as we would basically need to figure out that testcontainers is around and add it to a base class loader that would be shared across all the tests - it might be possible but I haven't looked at Holly's work yet).

The problem with Testcontainers is that there are some long lived threads that are kept around and our class loaders leak in two ways:

by being the CL of some classes that are kept around
by being the TCCL of some Testcontainers thread (Testcontainers itself, the Ryuk reaper thread and a ducttape thread)

Thus why this work is in two parts.

Class loader of a long lived class:

TCCL of thread:

gsmet · 2024-08-14T21:53:50Z

@aloubyansky this is what I was talking about.

core/deployment/pom.xml

core/runtime/pom.xml

aloubyansky · 2024-08-15T08:55:11Z

quarkus-core-deployment classes are a part of the bootstrap process, so they are on the classpath that launches the application, unlike the rest of the deployment dependencies. Actually, quarkus-core-deployment is also available for loading, along with other deployment dependencies, by the QuarkusClassLoader. It will depend on the classloader delegation whether that will happen.

To avoid long lived threads being attached to child class loaders. It was done at some point but was reverted because of the ducttape dependency not being parent first. I think it's worth another try by including this dependency.

While we probably want to be a lot more clever about it, it's the easiest way to have it in the test classpath and thus being allowed to load it parent first.

Testcontainers start several long lived threads (Ryuk reaper, ducttape), and if the TCCL is one of the QuarkusClassLoader, we end up leaking the class loaders. Using the Platform CL makes sure we don't leak any QuarkusClassLoader via these threads. The observability containers have not been adjusted as they are a bit of a weird beast and not worth having a look in the scope of this experiment.

gsmet · 2024-08-17T09:42:41Z

FWIW, while it fixes the problem in the OP's case and somehow works in some cases (a PostgreSQL container works fine), this PR comes with all sorts of issues.

It includes the following issues but there are others:

MongoDB

2024-08-16T20:30:22.9762024Z Caused by: java.lang.IllegalAccessError: class org.testcontainers.containers.MongoDBContainer$MongoDBContainerDef cannot access its superclass org.testcontainers.containers.ContainerDef (org.testcontainers.containers.MongoDBContainer$MongoDBContainerDef is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @51508029; org.testcontainers.containers.ContainerDef is in unnamed module of loader 'app')
2024-08-16T20:30:22.9766481Z 	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
2024-08-16T20:30:22.9767668Z 	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
2024-08-16T20:30:22.9771695Z 	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:533)
2024-08-16T20:30:22.9773679Z 	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:491)
2024-08-16T20:30:22.9775470Z 	at io.quarkus.mongodb.deployment.DevServicesMongoProcessor.lambda$startMongo$0(DevServicesMongoProcessor.java:184)
2024-08-16T20:30:22.9776915Z 	at java.base/java.util.Optional.orElseGet(Optional.java:364)
2024-08-16T20:30:22.9778302Z 	at io.quarkus.mongodb.deployment.DevServicesMongoProcessor.startMongo(DevServicesMongoProcessor.java:209)
2024-08-16T20:30:22.9780123Z 	at io.quarkus.mongodb.deployment.DevServicesMongoProcessor.startMongo(DevServicesMongoProcessor.java:113)

IBM Db2

2024-08-16T18:21:48.1732198Z Caused by: java.lang.IllegalStateException: The image icr.io/db2_community/db2:11.5.9.0 requires you to accept a license agreement. Please place a file at the root of the classpath named container-license-acceptance.txt, e.g. at src/test/resources/container-license-acceptance.txt. This file should contain the line:
2024-08-16T18:21:48.1735318Z   icr.io/db2_community/db2:11.5.9.0
2024-08-16T18:21:48.1736670Z 	at org.testcontainers.utility.LicenseAcceptance.assertLicenseAccepted(LicenseAcceptance.java:30)
2024-08-16T18:21:48.1738487Z 	at org.testcontainers.containers.Db2Container.configure(Db2Container.java:83)
2024-08-16T18:21:48.1740496Z 	at io.quarkus.devservices.db2.deployment.DB2DevServicesProcessor$QuarkusDb2Container.configure(DB2DevServicesProcessor.java:98)

Microsoft SQL Server

2024-08-16T18:22:21.6021940Z Caused by: java.lang.IllegalStateException: The image mcr.microsoft.com/mssql/server:2022-latest requires you to accept a license agreement. Please place a file at the root of the classpath named container-license-acceptance.txt, e.g. at src/test/resources/container-license-acceptance.txt. This file should contain the line:
2024-08-16T18:22:21.6025146Z   mcr.microsoft.com/mssql/server:2022-latest
2024-08-16T18:22:21.6026699Z 	at org.testcontainers.utility.LicenseAcceptance.assertLicenseAccepted(LicenseAcceptance.java:30)
2024-08-16T18:22:21.6028452Z 	at org.testcontainers.containers.MSSQLServerContainer.configure(MSSQLServerContainer.java:77)
2024-08-16T18:22:21.6030600Z 	at io.quarkus.devservices.mssql.deployment.MSSQLDevServicesProcessor$QuarkusMSSQLServerContainer.configure(MSSQLDevServicesProcessor.java:100)

gsmet · 2024-09-12T15:29:18Z

I marked this PR ready for review to prepare for a talk (and get a CI report). DON'T MERGE IT.

quarkus-bot · 2024-09-12T21:17:00Z

Status for workflow `Quarkus CI`

This is the status report for running Quarkus CI on commit c75ded8.

Warning

Unable to include the stracktraces as the report was too long. See annotations below for the details.

Failing Jobs

Status	Name	Step	Failures	Logs	Raw logs	Build scan
✖	JVM Tests - JDK 17	`Build`	Failures	Logs	Raw logs	🔍
✖	JVM Tests - JDK 21	`Build`	Failures	Logs	Raw logs	🔍
✖	Gradle Tests - JDK 17	`Build`	Failures	Logs	Raw logs	🔍
✖	Gradle Tests - JDK 17 Windows	`Build`	Failures	Logs	Raw logs	🚧
✖	MicroProfile TCKs Tests	`Verify`	Failures	Logs	Raw logs	🔍
✖	Native Tests - Data4	`Build`	Failures	Logs	Raw logs	🔍
✖	Native Tests - Security2	`Build`	Failures	Logs	Raw logs	🚧
✖	Native Tests - Security3	`Build`	Failures	Logs	Raw logs	🚧

Full information is available in the Build summary check run.
You can consult the Develocity build scans.

Failures

⚙️ JVM Tests - JDK 17 #

- Failing: extensions/jdbc/jdbc-db2/deployment extensions/jdbc/jdbc-mssql/deployment extensions/keycloak-admin-rest-client/deployment and 8 more
! Skipped: extensions/keycloak-authorization/deployment extensions/oidc-client-filter/deployment extensions/oidc-client-graphql/deployment and 17 more

📦 extensions/jdbc/jdbc-db2/deployment

✖ io.quarkus.jdbc.db2.deployment.DevServicesDB2DatasourceTestCase. - History - More details - Source on GitHub

📦 extensions/jdbc/jdbc-mssql/deployment

✖ io.quarkus.jdbc.mssql.deployment.DevServicesMsSQLDatasourceTestCase. - History - More details - Source on GitHub

📦 extensions/keycloak-admin-rest-client/deployment

✖ io.quarkus.keycloak.admin.client.reactive.KeycloakAdminClientInjectionDevServicesTest.testGetRoles - History - More details - Source on GitHub

📦 extensions/keycloak-admin-resteasy-client/deployment

✖ io.quarkus.keycloak.adminclient.deployment.KeycloakAdminClientInjectionDevServicesTest.testGetRoles - History - More details - Source on GitHub

📦 extensions/oidc-client/deployment

✖ io.quarkus.oidc.client.OidcClientPasswordGrantSecretIsMissingTestCase. line 36 - History - More details - Source on GitHub

✖ io.quarkus.oidc.client.OidcClientTooManyJwtCredentialKeyPropsTestCase. line 36 - History - More details - Source on GitHub

📦 extensions/oidc-db-token-state-manager/deployment

✖ io.quarkus.oidc.db.token.state.manager.HibernateOrmPgDbTokenStateManagerTest. - History - More details - Source on GitHub

✖ io.quarkus.oidc.db.token.state.manager.PostgresDbTokenStateManagerTest. - History - More details - Source on GitHub

📦 extensions/reactive-mssql-client/deployment

✖ io.quarkus.reactive.mssql.client.DevServicesMsSQLDatasourceTestCase. - History - More details - Source on GitHub

📦 integration-tests/mongodb-devservices

✖ io.quarkus.it.mongodb.BookResourceTest. - History - More details - Source on GitHub

✖ io.quarkus.it.mongodb.BookResourceWithParameterInjectionTest. - History - More details - Source on GitHub

✖ io.quarkus.it.mongodb.OtherProfileBookResourceTest.testBlockingClient - History - More details - Source on GitHub

📦 integration-tests/oidc-client-registration

✖ io.quarkus.it.keycloak.OidcClientRegistrationTest.testRegisteredClientDynamicTenant - History - More details - Source on GitHub

📦 integration-tests/oidc-mtls

✖ io.quarkus.it.oidc.OidcMtlsTest.testGetIdentityNames - History - More details - Source on GitHub

📦 integration-tests/oidc-wiremock

✖ io.quarkus.it.keycloak.AnnotationBasedTenantTest.testClassLevelAnnotation line 84 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.AnnotationBasedTenantTest.testClassicHttpSecurityPolicyWithRbac line 209 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.AnnotationBasedTenantTest.testJaxRsAndClassicHttpSecurityPolicyNoRbac line 236 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.AnnotationBasedTenantTest.testJaxRsAndClassicHttpSecurityPolicyWithRbac line 276 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.AnnotationBasedTenantTest.testJaxRsHttpSecurityPolicyNoRbac line 152 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.AnnotationBasedTenantTest.testJaxRsHttpSecurityPolicyWithRbac line 181 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.AnnotationBasedTenantTest.testJaxRsIdentityAugmentation line 308 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.AnnotationBasedTenantTest.testMethodLevelAnnotation line 113 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.AnnotationBasedTenantTest.testPolicyAppliedBeforeTenantAnnotationMatched line 356 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.BearerTokenOidcRecoveredTest.testOidcRecoveredWithDiscovery line 36 - History - More details - Source on GitHub

✖ io.quarkus.it.keycloak.BearerTokenOidcRecoveredTest.testOidcRecoveredWithNoDiscovery line 61 - History - More details - Source on GitHub

⚙️ JVM Tests - JDK 21 #

- Failing: extensions/jdbc/jdbc-db2/deployment extensions/jdbc/jdbc-mssql/deployment extensions/keycloak-admin-rest-client/deployment and 8 more
! Skipped: extensions/keycloak-authorization/deployment extensions/oidc-client-filter/deployment extensions/oidc-client-graphql/deployment and 17 more