Failed to build in native-image mode

[oldshen]

Environment

• Dubbo version: 3.3.0-beta.1
• Java version: 17

Steps to reproduce this issue

If there is a BigInteger type in the defined interface return class or parameter type, an exception will be thrown when native-image is compiled, prompting java.lang.StackOverflowError
After investigation, it was caused by the imprecise implementation of the registerSerializationType method in AotUtils. The specific code:

   Arrays.stream(registerType.getDeclaredFields()).forEach((field -> registerSerializationType(field.getType(), hints)));

If there is a Field of its own type in the registerType type, it will fall into an infinite loop, for example:

class A implements Serializable{
    public static A empty=new A();
}

Based on the above analysis, we look at the BigInteger source code, which has similar code as follows:

     public static final BigInteger ZERO = new BigInteger(new int[0], 0);

Therefore, AotUtils will throw an exception when encountering BigInteger;

In the same way, if there are two classes that are each other's Field, a java.lang.StackOverflowError exception will also be thrown, for example:

class A implements Serializable{
     private B b;
}

class B implements Serializable{
     private A a;
}

Solution

In order to solve the infinite loop caused by similar circular dependencies, a variable of type Set<Class<?>> is added for deduplication judgment. The modified AotUtils code is as follows

public class AotUtils {

     private AotUtils() {

     }

     public static void registerSerializationForService(Class<?> serviceType, RuntimeHints hints) {
         Set<Class<?>> classSet = new HashSet<>();
         Arrays.stream(serviceType.getMethods()).forEach((method) -> {

             Arrays.stream(method.getParameterTypes())
                     .filter(x -> !x.equals(serviceType))
                     .forEach((parameterType) -> registerSerializationType(classSet, parameterType, hints));

             registerSerializationType(classSet, method.getReturnType(), hints);
         });
         classSet.clear();
     }

     private static void registerSerializationType(Set<Class<?>> classSet, Class<?> registerType, RuntimeHints hints) {
         if (classSet.contains(registerType)) {
             return;
         }
         classSet.add(registerType);
         if (isPrimitive(registerType)) {
             hints.serialization().registerType(TypeReference.of(ClassUtils.getBoxedClass(registerType)));
         } else {
             if (Serializable.class.isAssignableFrom(registerType)) {

                 hints.serialization().registerType(TypeReference.of(registerType));

                 Arrays.stream(registerType.getDeclaredFields())
                         .filter(x -> !x.equals(registerType))
                         .forEach((field -> registerSerializationType(classSet, field.getType(), hints)));
                 if (registerType.getSuperclass() != null) {
                     registerSerializationType(classSet, registerType.getSuperclass(), hints);
                 }
             }
         }

     }

     private static boolean isPrimitive(Class<?> cls) {
         return cls.isPrimitive() || cls == Boolean.class || cls == Byte.class
                 || cls == Character.class || cls == Short.class || cls == Integer.class
                 || cls == Long.class || cls == Float.class || cls == Double.class
                 || cls == String.class || cls == Date.class || cls == Class.class
                 || cls.isEnum() || cls.getName().equals("java.lang.Record");
     }
}

---

Environment

• Dubbo version: 3.3.0-beta.1
• Java version: 17

Steps to reproduce this issue

如果定义的接口返回类或参数类型中存在BigInteger类型，在native-image编译时，会抛出异常，提示java.lang.StackOverflowError
经排查是AotUtils中registerSerializationType方法的实现不严谨造成的，具体代码：

  Arrays.stream(registerType.getDeclaredFields()).forEach((field -> registerSerializationType(field.getType(), hints)));

如果registerType 类型中存在以自己为类型的Field，就会陷入死循环，比如：

class A implements Serializable{
   public static A empty=new A();
}

根据上面分析，我们查看BigInteger源码，其存在类似的如下代码：

    public static final BigInteger ZERO = new BigInteger(new int[0], 0);

因此AotUtils遇到BigInteger时会抛出异常；

同理，如果存在两个类，互为对方的Field，也会抛出java.lang.StackOverflowError异常，例如：

class A implements Serializable{
    private B b;
}

class B implements Serializable{
    private A a;
}

解决办法

为解决类似循环依赖造成的死循环，增加一个Set<Class<?>>类型的变量，用于去重判断，修改后的AotUtils代码如下

public class AotUtils {

    private AotUtils() {

    }

    public static void registerSerializationForService(Class<?> serviceType, RuntimeHints hints) {
        Set<Class<?>> classSet = new HashSet<>();
        Arrays.stream(serviceType.getMethods()).forEach((method) -> {

            Arrays.stream(method.getParameterTypes())
                    .filter(x -> !x.equals(serviceType))
                    .forEach((parameterType) -> registerSerializationType(classSet, parameterType, hints));

            registerSerializationType(classSet, method.getReturnType(), hints);
        });
        classSet.clear();
    }

    private static void registerSerializationType(Set<Class<?>> classSet, Class<?> registerType, RuntimeHints hints) {
        if (classSet.contains(registerType)) {
            return;
        }
        classSet.add(registerType);
        if (isPrimitive(registerType)) {
            hints.serialization().registerType(TypeReference.of(ClassUtils.getBoxedClass(registerType)));
        } else {
            if (Serializable.class.isAssignableFrom(registerType)) {

                hints.serialization().registerType(TypeReference.of(registerType));

                Arrays.stream(registerType.getDeclaredFields())
                        .filter(x -> !x.equals(registerType))
                        .forEach((field -> registerSerializationType(classSet, field.getType(), hints)));
                if (registerType.getSuperclass() != null) {
                    registerSerializationType(classSet, registerType.getSuperclass(), hints);
                }
            }
        }

    }

    private static boolean isPrimitive(Class<?> cls) {
        return cls.isPrimitive() || cls == Boolean.class || cls == Byte.class
                || cls == Character.class || cls == Short.class || cls == Integer.class
                || cls == Long.class || cls == Float.class || cls == Double.class
                || cls == String.class || cls == Date.class || cls == Class.class
                || cls.isEnum() || cls.getName().equals("java.lang.Record");
    }
}