Java generator from CRD: class not found in quarkus when enabling extraAnnotations #6738

spike83 · 2024-12-17T12:44:11Z

Describe the bug

When generating Java classes from CRD within Quarkus I get ClassNotFound Exceptions.

I was playing around with annotation processing settings but nothing helped.

Every time I enable the extraAnnotations I get exceptions when executing. It is also taking way longer to generate and I even generates the class io/fabric8/kubernetes/api/model/ObjectMetaBuilder that was not found in the target folder.

Fabric8 Kubernetes Client version

7.0.0, 6.13.14

Steps to reproduce

POM:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>test</groupId>
    <artifactId>test-component</artifactId>
    <version>1.0.0-SNAPSHOT</version>

    <properties>
        <compiler-plugin.version>3.13.0</compiler-plugin.version>
        <maven.compiler.release>21</maven.compiler.release>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <quarkus.platform.artifact-id>quarkus-bom</quarkus.platform.artifact-id>
        <quarkus.platform.group-id>io.quarkus.platform</quarkus.platform.group-id>
        <quarkus.platform.version>3.17.3</quarkus.platform.version>
        <kubernetes-client.version>7.0.0</kubernetes-client.version>
        <sundrio.version>0.200.0</sundrio.version>
        <lombok.version>1.18.36</lombok.version>

        <skipITs>true</skipITs>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>${quarkus.platform.group-id}</groupId>
                <artifactId>${quarkus.platform.artifact-id}</artifactId>
                <version>${quarkus.platform.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-kubernetes-client</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-arc</artifactId>
        </dependency>
         <!-- extraAnnotations requires these additional dependencies -->
        <dependency>
          <groupId>io.sundr</groupId>
          <artifactId>builder-annotations</artifactId>
          <version>${sundrio.version}</version>
          <scope>provided</scope>
        </dependency>
        <dependency>
          <groupId>org.projectlombok</groupId>
          <artifactId>lombok</artifactId>
          <version>${lombok.version}</version>
          <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.fabric8</groupId>
            <artifactId>generator-annotations</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-junit5</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>${quarkus.platform.group-id}</groupId>
                <artifactId>quarkus-maven-plugin</artifactId>
                <version>${quarkus.platform.version}</version>
                <extensions>true</extensions>
                <executions>
                    <execution>
                        <goals>
                            <goal>build</goal>
                            <goal>generate-code</goal>
                            <goal>generate-code-tests</goal>
                            <goal>native-image-agent</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${compiler-plugin.version}</version>
                <configuration>
                    <parameters>true</parameters>
                    <annotationProcessorPaths>
                        <path>
                            <groupId>org.projectlombok</groupId>
                            <artifactId>lombok</artifactId>
                            <version>${lombok.version}</version>
                        </path>
                        <path>
                            <groupId>io.sundr</groupId>
                            <artifactId>builder-annotations</artifactId>
                            <version>${sundrio.version}</version>
                        </path>
                        <path>
                            <groupId>io.fabric8</groupId>
                            <artifactId>generator-annotations</artifactId>
                            <version>${kubernetes-client.version}</version>
                        </path>
                    </annotationProcessorPaths>
                </configuration>
            </plugin>
            <plugin>
                <groupId>io.fabric8</groupId>
                <artifactId>java-generator-maven-plugin</artifactId>
                <version>${kubernetes-client.version}</version>
                <configuration>
                    <source>${project.basedir}/test-crd.yaml</source>
                    <alwaysPreserveUnknown>true</alwaysPreserveUnknown>
                    <extraAnnotations>true</extraAnnotations>
                    <generatedAnnotations>true</generatedAnnotations>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>generate</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            
        </plugins>
    </build>
</project>

test-crd.yaml: I took one of the PrinterColmn from the tests. But it doesn't matter.

src/test/java/MyTest.java:

import org.junit.jupiter.api.Test;

import io.fabric8.kubernetes.client.KubernetesClient;
import io.quarkus.test.junit.QuarkusTest;
import jakarta.inject.Inject;

@QuarkusTest
public class MyTest {
    @Inject
    private KubernetesClient kubernetesClient;
    
    @Test
    public void test() {
        System.out.println("pod list:");
        var pods = this.kubernetesClient.pods().list();
        pods.getItems().forEach(pod -> {
            System.out.println(pod.getMetadata().getName());
        });
    }
}

mvn clean test

This causes

13:34:40,619 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 18.49 s <<< FAILURE! -- in MyTest
13:34:40,620 [ERROR] MyTest.test -- Time elapsed: 1.204 s <<< ERROR!
io.fabric8.kubernetes.client.KubernetesClientException: io/fabric8/kubernetes/api/model/ObjectMetaBuilder
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:509)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:451)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:419)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:98)
        at MyTest.test(MyTest.java:15)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:967)
        at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:817)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: java.lang.NoClassDefFoundError: io/fabric8/kubernetes/api/model/ObjectMetaBuilder
        at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
        at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3578)
        at java.base/java.lang.Class.getDeclaredMethods(Class.java:2676)
        at com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1317)
        at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
        at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)

And it seems the class was even generated

Expected behavior

I would think that I can generate the code and run the kubernetes client afterwards.

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

1.25.3@latest

Environment

Linux

Fabric8 Kubernetes Client Logs

No response

Additional context

No response

The text was updated successfully, but these errors were encountered:

andreaTP · 2024-12-17T13:22:29Z

Hi @spike83 thanks for your interest!

My gut feeling is that there is an incompatibility with the sundrio version, but need to verify it.

Would you be able to share a test-crd.yaml? (possibly a minimal one)
Or a reproducer project that we can play with?

andreaTP · 2024-12-17T13:30:23Z

Oh, wait, you are trying to use kubernetes-client at version 7.0.0 with Quarkus that doesn't support it just yet ref.

I doubt this configuration will work cc. @manusa

spike83 · 2024-12-17T13:50:03Z

I tried to use 6.13.4 and get the same error.
Caused by: java.lang.ClassNotFoundException: io.fabric8.kubernetes.api.model.ObjectMetaBuilder

andreaTP · 2024-12-17T15:42:02Z

Ok, thanks for the additional info, would you mind providing the CRD content?
Or a full reproducer 🙏

spike83 · 2024-12-18T05:20:59Z

Sure, there is the reproducer:
k8s-crd-gen-quarkus.zip

run mvn test will produce the error.

andreaTP · 2024-12-18T10:32:57Z

@spike83 thanks, unfortunately I cannot reproduce the error using the zip you provided.

The error message I obtain is the expectd one:

io.fabric8.kubernetes.client.KubernetesClientException: The timeout of 10000 ms has been exceeded when getting a connection to my.cluster:6443
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:509)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:451)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:419)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:98)
        at MyTest.test(MyTest.java:15)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:967)
        at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:817)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: io.vertx.core.impl.NoStackTraceTimeoutException: The timeout of 10000 ms has been exceeded when getting a connection to my.cluster:6443

We need to bisect further the differences:

java version - I tested with 23 - 21
Maven version - I'm using 3.9.9

Or have a reproducible reproducer in, for example, a GH Action.

The only thing I'm noticing is that the sundrio version is not aligned with the one used by the k8s client.
It should be: https://github.com/fabric8io/kubernetes-client/blob/6ee190c3a36ea6ae5ed7513942e2cabef177a0eb/pom.xml#L129C21-L129C23

instead of 0.200.0, but I can't reproduce the issue with either versions.

spike83 · 2024-12-18T11:33:18Z

I understand that there is no Kubernetes cluster at your address. Could you point to one, as you can see in the test it only tries to list the pods? I guess it's happening when deserializing the actual response. I tried to put a dummy address and get no error.

k8s-crd-gen-quarkus-v2.zip

Here I also have another reproducer which tries to create a pod first and then tries to list the pods. This can be run against the local dev-servcie with

KUBECONFIG=./none.yaml mvn test. This causes another exception of the same nature.

This is what I get here with the second reproducer:

024-12-18 12:28:17,574 INFO  [tc.reg.k8s.io/.31.0] (build-9) Creating container for image: registry.k8s.io/kube-apiserver:v1.31.0
2024-12-18 12:28:17,585 INFO  [tc.tes.11.0] (build-9) Creating container for image: testcontainers/ryuk:0.11.0
2024-12-18 12:28:17,670 INFO  [tc.tes.11.0] (build-9) Container testcontainers/ryuk:0.11.0 is starting: b29291f98b598f34853a664992a9101a7495e26083e3559f944ee0ef9d97d96a
2024-12-18 12:28:17,991 INFO  [tc.tes.11.0] (build-9) Container testcontainers/ryuk:0.11.0 started in PT0.405975703S
2024-12-18 12:28:18,106 INFO  [tc.reg.k8s.io/.31.0] (build-9) Container registry.k8s.io/kube-apiserver:v1.31.0 is starting: c69b2a8d1924461042224533ce6f4fa505c6e31c5e9737a0e39fb3518b38cf4f
2024-12-18 12:28:18,501 INFO  [tc.reg.k8s.io/.5.12-0] (build-9) Creating container for image: registry.k8s.io/etcd:3.5.12-0
2024-12-18 12:28:18,548 INFO  [tc.reg.k8s.io/.5.12-0] (build-9) Container registry.k8s.io/etcd:3.5.12-0 is starting: 0177f8f296fb8f2eb137b595cec73426bebc7ec2c45f517dc99065ed47a94fd5
2024-12-18 12:28:18,702 INFO  [tc.reg.k8s.io/.5.12-0] (build-9) Container registry.k8s.io/etcd:3.5.12-0 started in PT0.200892865S
2024-12-18 12:28:18,702 INFO  [com.daj.kin.ApiServerContainer] (build-9) Waiting for API server...
2024-12-18 12:28:20,588 INFO  [tc.reg.k8s.io/.31.0] (build-9) Container registry.k8s.io/kube-apiserver:v1.31.0 started in PT3.014359281S
Dez. 18, 2024 12:28:20 PM io.quarkus.kubernetes.client.deployment.DevServicesKubernetesProcessor setupKubernetesDevService
INFO: Dev Services for Kubernetes started. Other Quarkus applications in dev mode will find the cluster automatically.
Dez. 18, 2024 12:28:21 PM io.quarkus.bootstrap.runner.Timing printStartupTime
INFO: test-component 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.17.3) started in 10.776s. Listening on: http://localhost:8081
Dez. 18, 2024 12:28:21 PM io.quarkus.bootstrap.runner.Timing printStartupTime
INFO: Profile test activated.
Dez. 18, 2024 12:28:21 PM io.quarkus.bootstrap.runner.Timing printStartupTime
INFO: Installed features: [cdi, kubernetes-client, rest, smallrye-context-propagation, vertx]
12:28:22,187 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 14.86 s <<< FAILURE! -- in MyTest
12:28:22,188 [ERROR] MyTest.test -- Time elapsed: 0.033 s <<< ERROR!
java.lang.NoClassDefFoundError: io/fabric8/kubernetes/api/model/ObjectMetaFluent
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:558)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:517)
	at io.fabric8.kubernetes.api.model.PodFluent.withNewMetadata(PodFluent.java:88)
	at MyTest.test(MyTest.java:20)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:967)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:817)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: java.lang.ClassNotFoundException: io.fabric8.kubernetes.api.model.ObjectMetaFluent
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:542)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:517)
	... 11 more

Dez. 18, 2024 12:28:22 PM io.quarkus.bootstrap.runner.Timing printStopTime
INFO: test-component stopped in 0.040s
Dez. 18, 2024 12:28:22 PM io.quarkus.devservices.common.ContainerShutdownCloseable close
INFO: Dev Services for kubernetes-client shut down.
12:28:22,584 [INFO]
12:28:22,585 [INFO] Results:
12:28:22,585 [INFO]
12:28:22,585 [ERROR] Errors:
12:28:22,585 [ERROR]   MyTest.test:20 » NoClassDefFound io/fabric8/kubernetes/api/model/ObjectMetaFluent

andreaTP · 2024-12-18T12:26:46Z

I got the reproducer to work and minimized a little further here.

@manusa would you mind taking a look?
My memory regarding this flow starts to be rusty ...

manusa · 2024-12-19T14:41:17Z

I can reproduce the issue with the provided example.

@andreaTP, have we tried the Java Generator with Quarkus before? do you know if there's an integration test somewhere?

Also, maybe the problem might ring a bell to @metacosm

The problem seems to be with the Quarkus test runner and the loading of the generated classes. I'd assume that this won't happen with a regular JUnit test.

andreaTP · 2024-12-19T15:07:51Z

have we tried the Java Generator with Quarkus before?

I used the Java generator with the QOSDK a few times.

do you know if there's an integration test somewhere?

I don't think we have specific tests, the Java Generator is emitting POJOs, most of the hiccups we found are related to lombok or sundrio interactions when "extraAnnotations" is enabled.

The problem seems to be with the Quarkus test runner and the loading of the generated classes. I'd assume that this won't happen with a regular JUnit test.

Should we report the bug in the Quarkus repo?
We can likely minimize it further to avoid using the java generator at all.

manusa · 2024-12-19T15:29:01Z

Should we report the bug in the Quarkus repo?
We can likely minimize it further to avoid using the java generator at all.

This does seem like a Quarkus bug.
We'll probably need to take care of this ourselves, but from what I can see the problem needs fixing in Quarkus.

andreaTP changed the title ~~CRD generator: class not found in quarkus when enabling extraAnnotations~~ Java generator from CRD: class not found in quarkus when enabling extraAnnotations Dec 17, 2024

andreaTP added component/java-generator bug labels Dec 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Java generator from CRD: class not found in quarkus when enabling extraAnnotations #6738

Java generator from CRD: class not found in quarkus when enabling extraAnnotations #6738

spike83 commented Dec 17, 2024 •

edited by manusa

Loading

andreaTP commented Dec 17, 2024

andreaTP commented Dec 17, 2024

spike83 commented Dec 17, 2024

andreaTP commented Dec 17, 2024

spike83 commented Dec 18, 2024

andreaTP commented Dec 18, 2024

spike83 commented Dec 18, 2024 •

edited

Loading

andreaTP commented Dec 18, 2024

manusa commented Dec 19, 2024 •

edited

Loading

andreaTP commented Dec 19, 2024

manusa commented Dec 19, 2024

Java generator from CRD: class not found in quarkus when enabling extraAnnotations #6738

Java generator from CRD: class not found in quarkus when enabling extraAnnotations #6738

Comments

spike83 commented Dec 17, 2024 • edited by manusa Loading

Describe the bug

Fabric8 Kubernetes Client version

Steps to reproduce

Expected behavior

Runtime

Kubernetes API Server version

Environment

Fabric8 Kubernetes Client Logs

Additional context

andreaTP commented Dec 17, 2024

andreaTP commented Dec 17, 2024

spike83 commented Dec 17, 2024

andreaTP commented Dec 17, 2024

spike83 commented Dec 18, 2024

andreaTP commented Dec 18, 2024

spike83 commented Dec 18, 2024 • edited Loading

andreaTP commented Dec 18, 2024

manusa commented Dec 19, 2024 • edited Loading

andreaTP commented Dec 19, 2024

manusa commented Dec 19, 2024

spike83 commented Dec 17, 2024 •

edited by manusa

Loading

spike83 commented Dec 18, 2024 •

edited

Loading

manusa commented Dec 19, 2024 •

edited

Loading