Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for JDK17 #885

Open
dkroehan opened this issue Feb 18, 2022 · 23 comments
Open

Support for JDK17 #885

dkroehan opened this issue Feb 18, 2022 · 23 comments

Comments

@dkroehan
Copy link

Describe the bug
It seems that Kryo is currently not compatible with JDK17. When calling the default constructor it fails with:

java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @61322f9d
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:188)
	at java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:181)
	at com.esotericsoftware.kryo.unsafe.UnsafeUtil.<clinit>(UnsafeUtil.java:100)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:467)
	at com.esotericsoftware.kryo.util.Util.<clinit>(Util.java:47)
	at com.esotericsoftware.kryo.serializers.OptionalSerializers.addDefaultSerializers(OptionalSerializers.java:38)
	at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:230)
	at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:168)

To Reproduce
Kryo kryo = new Kryo();

Environment:

  • JDK Version: OpenJDK 17.0.2 (Corretto-17.0.2.8.1)
  • Kryo Version: 5.3.0

Additional context
In my case I am also registering some standard classes.
It seems to work for classes like:

  • java.util.Locale
  • java.time.Instant
  • java.util.HashMap

But it fails for java.util.regex.Pattern due to:

java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.String java.util.regex.Pattern.pattern accessible: module java.base does not "opens java.util.regex" to unnamed module @185a6e9

What would be the proper way to fix this?

@theigl
Copy link
Collaborator

theigl commented Feb 18, 2022

Your specific case can be solved by using a custom RegexSerializer that doesn't rely on reflection.

Alternatively, you can add JVM arguments for opening up the modules in question. See this comment.

I think it makes sense to include the RegexSerializer and UUIDSerializer directly in Kryo, so those two classes can be serialized on JDK 17 without opening any modules.

@theigl
Copy link
Collaborator

theigl commented Feb 18, 2022

I cannot reproduce your first issue. I just downloaded and ran my tests on Corretto to make sure it isn't a JVM specific problem.

@dkroehan
Copy link
Author

Thank you for the quick response.

I did set a breakpoint at AccessibleObject#checkCanSetAccessible at line 354 where an InaccessibleObjectException is created for debugging purposes.
This is where I found: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @61322f9d

This is caused by the Util static initializer block at: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/util/Util.java#L47
This calls UnsafeUtil with the static initializer at: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/unsafe/UnsafeUtil.java#L99

The InaccessibleObjectException is caught in line 101, which causes the directByteBufferConstructor to be null. I didn't pay attention that the Exception is actually not bubbling up.

Now I'm wondering if this is expected behaviour or a failure with java 17.

@dkroehan
Copy link
Author

dkroehan commented Feb 21, 2022

I tried the RegexSerializer using:

kryo.register(Pattern.class, new RegexSerializer());

It seems to work.

FYI: UnmodifiableCollectionsSerializer from kryo-serializers also lacks compatibility. There is already an open issue, which I just updated:
#magro/kryo-serializers#131 (comment)

@theigl
Copy link
Collaborator

theigl commented Feb 21, 2022

Now I'm wondering if this is expected behaviour or a failure with java 17.

This is expected behavior and has no effect unless you want to use UnsafeByteBuffers for Input/Output. If you want to use those, you have to open the module up for reflection.

FYI: UnmodifiableCollectionsSerializer from kryo-serializers also lacks compatibility. There is already an open issue, which I just updated:

AFAIK it is impossible to write a non-reflective version of a serializer for UnmodifiableCollection that covers all potential cases. So these serializers will probably never make it into Kryo and you will have to add add-opens arguments if you need to serialize them.

@theigl
Copy link
Collaborator

theigl commented Apr 27, 2022

Kryo is currently missing built-in, safe serializers for the following JDK classes:

  • java.net.URI
  • java.util.UUID
  • java.util.regex.Pattern
  • java.util.concurrent.atomic.AtomicBoolean
  • java.util.concurrent.atomic.AtomicInteger
  • java.util.concurrent.atomic.AtomicLong
  • java.util.concurrent.atomic.AtomicReference

These serializers are trivial to implement, but cannot be registered in Kryo 5 as default serializers because it would break backwards compatibility. I will add them to DefaultSerializers so users can register them, but they will only be registered by default in Kryo 6.

@jdorleans
Copy link

jdorleans commented Jun 17, 2022

I'm trying to upgrade from Java 11 to 17 running Kryo 5 with Unsafe I/O, but am getting a weird NoClassDefFoundError for UnsafeUtil while writing data with writeClassAndObject:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.esotericsoftware.kryo.kryo5.unsafe.UnsafeUtil
	at com.esotericsoftware.kryo.kryo5.unsafe.UnsafeOutput.writeByte(UnsafeOutput.java:93)
	at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeName(DefaultClassResolver.java:123)
	at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:114)
	at com.esotericsoftware.kryo.kryo5.Kryo.writeClass(Kryo.java:613)
	at com.esotericsoftware.kryo.kryo5.Kryo.writeClassAndObject(Kryo.java:708)

I also tested moving away from Unsafe* to use Input/Output and it works fine. Now I'm wondering how do I get Unsafe to work again.

@theigl
Copy link
Collaborator

theigl commented Jun 23, 2022

@jdorleans: On JDK 17, you currently need to open up sun.nio.ch like this:

--add-opens=java.base/sun.nio.ch=ALL-UNNAMED

Otherwise initialization of UnsafeUtil fails because it cannot run DirectBuffer.class.getMethod("cleaner"). We should allow usage of unsafe without requiring access to DirectBuffer. I'll investigate how other libraries solve this. For now, please add the --add-opens statement.

@graves501
Copy link

graves501 commented Jul 7, 2022

I guess I'm just doing something wrong, but I'm also currently trying the latest kryo version (5.3.0) in combination with Java 17.
I'm trying to clone an object that contains a list.

final var kryo = new Kryo();
kryo.register(ClassWithList.class);
ClassWithList copy = kryo.copy(classWithList);

I get this error messsage:

com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: java.util.ImmutableCollections$List12
Note: To register this class use: kryo.register(java.util.ImmutableCollections.List12.class);

Registering this class doesn't work, since List12 is not publicly accessible.

@theigl
Copy link
Collaborator

theigl commented Jul 10, 2022

@graves501: You can either set registrationRequired(false) or manually register the classes using one of the following:

kryo.register(List.of(1).getClass());
kryo.register(List.of(1,2,3).getClass());
kryo.register(Class.forName("java.util.ImmutableCollections$List12"));
kryo.register(Class.forName("java.util.ImmutableCollections$ListN"));

@graves501
Copy link

Thanks, that did the job!

@theigl
Copy link
Collaborator

theigl commented Dec 20, 2022

Quick update:

  1. I added the following serializers to Kryo. These serializers are not registered by default (because of backwards compatibility), but can be registered as needed:
  1. It should no longer be necessary to open up sun.nio.ch. Access to DirectBuffer is delayed until it is first used.

    See Delay access to DirectBuffers until first use #932

  2. I added a helper method to register serializers for ImmutableCollections:

    ImmutableCollectionsSerializers.registerSerializers(kryo);

    See #923 Add helper to register serializers for java.util.ImmutableCollections #933

It would be great if anyone could confirm the second change using the current snapshots.

@theigl theigl pinned this issue Dec 20, 2022
@leonchen83
Copy link

leonchen83 commented Jan 3, 2023

@theigl

Exception in thread "main" java.lang.reflect.InaccessibleObjectException: Unable to make field private transient java.lang.Object java.lang.Throwable.backtrace accessible: module java.base does not "opens java.lang" to unnamed module @2eafffde
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at com.esotericsoftware.kryo.serializers.CachedFields.addField(CachedFields.java:123)
	at com.esotericsoftware.kryo.serializers.CachedFields.rebuild(CachedFields.java:99)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:82)
	at com.esotericsoftware.kryo.SerializerFactory$FieldSerializerFactory.newSerializer(SerializerFactory.java:124)
	at com.esotericsoftware.kryo.SerializerFactory$FieldSerializerFactory.newSerializer(SerializerFactory.java:108)
	at com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:469)
	at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:454)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:89)
	at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:581)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:112)
	at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:613)
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:708)
	at cn.nextop.test.Main.writeClassAndObject(Main.java:59)
	at cn.nextop.test.Main.main(Main.java:37)

version: kryo 5.4.0

reproduce code

import java.util.concurrent.ExecutionException;

import org.objenesis.strategy.StdInstantiatorStrategy;

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.unsafe.UnsafeInput;
import com.esotericsoftware.kryo.unsafe.UnsafeOutput;
import com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy;

public class Main {
	public static void main(String[] args) {
		Main main = new Main();
		
		Kryo kryo = main.init();
		ExecutionException rpc = new ExecutionException(null, new Throwable());
		byte[] bytes = main.writeClassAndObject(kryo, rpc);
		ExecutionException value = (ExecutionException) main.readClassAndObject(kryo, bytes);
	}
	
	public Kryo init() {
		Kryo kryo = new Kryo();
		kryo.setReferences(true);
		kryo.setRegistrationRequired(false);
		kryo.setWarnUnregisteredClasses(false);
		
		DefaultInstantiatorStrategy init = (DefaultInstantiatorStrategy) kryo.getInstantiatorStrategy();
		init.setFallbackInstantiatorStrategy(new StdInstantiatorStrategy());
		return kryo;
	}
	
	public Object readClassAndObject(Kryo kryo, byte[] bytes ) {
		UnsafeInput input = new UnsafeInput();
		input.setBuffer(bytes); Object r = kryo.readClassAndObject(input); return r;
	}
	
	public byte[] writeClassAndObject (Kryo kryo, Object value) {
		UnsafeOutput output = new UnsafeOutput(1024, 32 * 1024 * 1024);
		kryo.writeClassAndObject(output, value); return output.toBytes();
	}
}

We need to serialize ExecutionException because we need to implement an RPC service by kryo. so could we add ExecutionExceptionSerializer to kryo 5.4.0?

@theigl
Copy link
Collaborator

theigl commented Jan 3, 2023

@leonchen83: I would not consider serializing exceptions a typical use-case for Kryo, so I'm against adding exception serializers to Kryo itself.

You have two options:

  1. Write your own serializer
  2. Use JavaSerializer for serializing exceptions

I'd go for the second option. Exceptions should occur very rarely, so speed of serialization and payload size usually doesn't matter that much.

Redisson for instance is using this approach.

@amodolo
Copy link

amodolo commented May 30, 2023

Hi @theigl,
also ClosureSerializer is not compatible with JDK 17. Is there any workarround to solve this

Version 5.5.0
JDK: temurin 17.0.7

Here is the code used to reproduce the exception

public static void main(String[] args) {
    Kryo kryo = new Kryo();
    kryo.register(ClosureSerializer.Closure.class, new ClosureSerializer());
}

here is the exception

00:00  WARN: Unable to obtain SerializedLambda#readResolve via reflection. Falling back on resolving lambdas via capturing class.
java.lang.reflect.InaccessibleObjectException: Unable to make private java.lang.Object java.lang.invoke.SerializedLambda.readResolve() throws java.io.ObjectStreamException accessible: module java.base does not "opens java.lang.invoke" to unnamed module @46238e3f
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
	at java.base/java.lang.reflect.Method.setAccessible(Method.java:193)
	at com.esotericsoftware.kryo.serializers.ClosureSerializer.<init>(ClosureSerializer.java:59)
	at dev.test.serialization.kryo.Main.main(Main.java:9)
00:00  WARN: Unable to obtain SerializedLambda#capturingClass via reflection. Falling back to resolving capturing class via Class.forName.
java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.lang.Class java.lang.invoke.SerializedLambda.capturingClass accessible: module java.base does not "opens java.lang.invoke" to unnamed module @46238e3f
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at com.esotericsoftware.kryo.serializers.ClosureSerializer.<init>(ClosureSerializer.java:69)
	at dev.test.serialization.kryo.Main.main(Main.java:9)

I've already try to set this argument to the JVM

--add-opens=java.base/java.lang.invoke=ALL-UNNAMED 

but it solves nothing

@theigl
Copy link
Collaborator

theigl commented May 30, 2023

@amodolo: I am using ClosureSerializer on OpenJDK 17 for quite a while now. It works fine. Are you sure you are passing the JVM argument correctly? I'm passing the following argument:

--add-opens java.base/java.lang.invoke=ALL-UNNAMED

Maybe the additional = causes problems in your case?

@amodolo
Copy link

amodolo commented May 30, 2023

You are right. there is an additional =. even without it, the problem is still there. Here is my configuration.
image

@theigl
Copy link
Collaborator

theigl commented May 30, 2023

I just ran ClosureSerializerTest in Kryo with the following settings:

image

It works fine.

The command run by IntelliJ looks like this:

/Users/XXX/Library/Java/JavaVirtualMachines/temurin-17.0.7/Contents/Home/bin/java --add-opens java.base/java.lang.invoke=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --enable-preview ......

How does it look in your case?

@amodolo
Copy link

amodolo commented May 30, 2023

Oh i see what's going wrong.
i passed the --add-opens to the program arguments instead of VM options. Your configuration solves the problem.
Thank you

@endlesstian
Copy link

Kryo is currently missing built-in, safe serializers for the following JDK classes:

  • java.net.URI
  • java.util.UUID
  • java.util.regex.Pattern
  • java.util.concurrent.atomic.AtomicBoolean
  • java.util.concurrent.atomic.AtomicInteger
  • java.util.concurrent.atomic.AtomicLong
  • java.util.concurrent.atomic.AtomicReference

These serializers are trivial to implement, but cannot be registered in Kryo 5 as default serializers because it would break backwards compatibility. I will add them to DefaultSerializers so users can register them, but they will only be registered by default in Kryo 6.

Hi, when will Kryo 6 release? Thanks.
BRs,
Tian

@theigl
Copy link
Collaborator

theigl commented Oct 18, 2023

@endlesstian: You can already register all these serializers yourself. They are available for manual registration since Kryo 5.4.0.

As for Kryo 6, I currently do not have a release date. I will have more spare time in December and I'll see if I can create a release candidate by the end of the year.

@vyalyh-oleg
Copy link

Can we expect, that the rkyo library will support modular system ?
--add-opens isn't a problem, but I would like to open the specific package not to all UNNAMED modules, but e.g. to a specific one (kryo).
I think it could solve some problems.

@theigl
Copy link
Collaborator

theigl commented Dec 23, 2023

@vyalyh-oleg: Kryo currently only adds an automatic module name. So far, there is no module definition. What is required for add-opens to be restricted to Kryo?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

8 participants