Question: How to make a library with custom serializers

[nielsbasjes]

Hi,

I have written this library that is used in various places in other projects: https://github.com/nielsbasjes/yauaa

Now to make it all efficient I have made sure it is all serializable with both the standard Java serialization and Kryo.
This is also needed to support systems like Apache Flink and Apache Beam.

With Kryo 5.0.0 a change has been introduced that essentially requires registering the classes with the Kryo instance OR indicate this is not needed (i.e. optional registration). ( See also question #770 )

To make the registration of all needed classes as easy as possible I started by creating a method that returns the list of classes that need to be registered.

I stopped this effort when I got this error from my test:

com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: java.util.ArrayList
Note: To register this class use: kryo.register(java.util.ArrayList.class);

So my question is what is the correct/best way of doing this for my library so that it will work 'nicely' for the applications that include my library?

[nielsbasjes]

Also: What should I do to make it compatible with both Kryo 4.x and 5.x? Or is that simply not possible?

[theigl]

@nielsbasjes:

With Kryo 5.0.0 a change has been introduced that essentially requires registering the classes with the Kryo instance OR indicate this is not needed (i.e. optional registration).

The only change in Kryo 5 is that the default value for registrationRequired is now true. In Kryo 4 it was false. If you want Kryo 5 to behave like 4, simply turn off registrationRequired.

Also: What should I do to make it compatible with both Kryo 4.x and 5.x? Or is that simply not possible?

What exactly do you mean? Do you simply want to write tests against Kryo 4 and Kryo 5 or do you want to include production code that references Kryo?

If you only want to write tests that make sure that your classes can be serialized with Kryo 4 and 5, you can include Kryo 4 as you did before and depend on the the versioned artifact for Kryo 5:

See https://github.com/EsotericSoftware/kryo#installation

[theigl]

If you want to find all classes that need to be registered for your test case, set the following flags and check the logs:

kryo.setRegistrationRequired(false);
kryo.setWarnUnregisteredClasses(true);

I hope that helps.

[nielsbasjes]

What I mean with the Kryo 4.x/5.x is that my library is used in applications written by other people.
I would like my library to work regardless if the upstream project has chosen to use Kryo 4.x or 5,x

[theigl]

What I mean with the Kryo 4.x/5.x is that my library is used in applications written by other people.
I would like my library to work regardless if the upstream project has chosen to use Kryo 4.x or 5,x

But on your side this simply means that you write two test-cases. One for Kryo 4 and one for Kryo 5. Correct? Or do you include some kind of KryoSerializer/KryoAdapter in your library?

[nielsbasjes]

Yes, for some of my classes I need custom code to ensure the transient elements are initialized correctly upon arrival.
So I have custom serialization code which I would like to be compatible with both Kryo versions.

[nielsbasjes]

So I run my test with the "setWarnUnregisteredClasses" you indicated and I found two things:

1. For nested classes the instructions are incorrect.
   One of the messages I get is

00:00  WARN: Class is not registered: nl.basjes.parse.useragent.UserAgent$MutableUserAgent
Note: To register this class use: kryo.register(nl.basjes.parse.useragent.UserAgent$MutableUserAgent.class);

which says
kryo.register(nl.basjes.parse.useragent.UserAgent$MutableUserAgent.class);
but I think should not have the '$' but a '.'
kryo.register(nl.basjes.parse.useragent.UserAgent.MutableUserAgent.class);

2. How do I handle private / package private / module 'non-exposed' classes like this one?

00:00  WARN: Class is not registered: java.util.Collections$EmptySet
Note: To register this class use: kryo.register(java.util.Collections$EmptySet.class);

From what I now understand this seems to be sliding into a situation where every library (even the Java runtime) gets a "registerClassesWithKryo(Kryo kryo)" method. Seems a bit much for a serialization system especially because it seems optional to register.

[theigl]

Thank you for the feedback Niels!

For nested classes the instructions are incorrect.

Good catch. I'll push an improvement shortly.

How do I handle private / package private / module 'non-exposed' classes like this one?

kryo.register(Collections.emptySet().getClass());

From what I now understand this seems to be sliding into a situation where every library (even the Java runtime) gets a "registerClassesWithKryo(Kryo kryo)" method. Seems a bit much for a serialization system especially because it seems optional to register.

As I said before: For your use-case (i.e. replacement of Java serialization for object graphs with a large number of classes), I'd recommend that you simply turn off registrationRequired and let Kryo do the registration for you.

[nielsbasjes]

Thanks for taking the time to explain. This really helps.

So the kryo.register(Collections.emptySet().getClass()); construct works for this example.

However a class in a separate library I use is a Java 9+ module and is not exported at all and not accessible in a similar way as what you can do with EmptySet. I simply cannot reach this class at all as it is only used in an internal datastructure.

So that makes an 'end-user' project essentially one of these two scenarios:

1. ALL used libraries and modules have a way to register all used classes. Either these classes are publicly accessible or a library specific registerAllClassesWithKryo method has been provided. In this scenario the application can register all these classes and do the serialization as intended. Problem is that only a handful of libraries have such a thing for Kryo.

2. You have to turn off registrationRequired

Now since the Java 11 JRE is modular and does not expose all classes and does not have a registerAllClassesWithKryo method ... I expect that scenario 1) will never occur.

My unfortunate conclusion right now is that the new default setting in 5.x regarding the registrationRequired is something that needs to be reverted in almost all cases.

[theigl]

My unfortunate conclusion right now is that the new default setting in 5.x regarding the registrationRequired is something that needs to be reverted in almost all cases.

See #398 for why the default value was changed.

[nielsbasjes]

Thanks for your explanations.

[nielsbasjes]

@theigl A quick headsup on how I solved my problem.

I have written a few libraries and I want to support downstream applications for the scenario's
"no Kryo" (Java Serialization), "Kryo 4" and "Kryo 5"

I have made this possible by essentially making Kryo optional, both in dependencies and code.
As an example a class that has this:
https://github.com/nielsbasjes/prefixmap/blob/master/prefixmap/src/main/java/nl/basjes/collections/prefixmap/StringPrefixMap.java#L61

To make sure the serialization actually works for the downstream projects as I want it; I added several maven sub modules with different dependencies to ensure I can test if everything works as expected in all cases https://github.com/nielsbasjes/prefixmap/tree/master/serialization

Note that this code is still a bit new and raw and I'm looking into cleaning it.