Is there an explicit release date for the 5.0 ？

[endink]

Jesus christ! 5.0 has been 2 years since RC1 to RC7, and when will the release version arrive?

[theigl]

@endink:

This is an open source libary created and maintained mostly by a single person. Sometimes priorities shift or the job schedule prevents people from working on their side projects.

Instead of demanding a release date, a better approach would be to get in contact with the project team and ask if you can help in any way. Look at the open bug reports, grab a ticket, create a PR.

Having said that, I can give you an update on the current status:

Kryo 5 is a major rewrite and cleans up a lot of technical debt that has accumulated in the previous versions. It has been relatively stable since RC3/4 published in early 2019, but there were some major issues with the rewritten generics optimization code. These issues didn't get addressed because Nate - who created the project - didn't have time to look into them.

I started to collect all these known issues, write tests for them, and create PRs to fix them starting in early May. RC6 and the newly released RC7 contain all these generics fixes and an option to opt-out of generics optimization altogether.

Last week, I deployed RC7 to my production system to verify that everything works as expected. After tens of millions of serialization roundtrips and more than 5000 different classes serialized, I'm pretty confident that all issues have been fixed and RC7 is stable enough for production.

There is one last issue though: Serialization is 10-15% slower with Kryo 5 than it it was with Kryo 4.0.2 in my case.
I'm serializing large object graphs, so this might not affect everybody, but I still want to do more profiling
and push some potential optimizations to production before a final release of Kryo 5.

[magro]

Thanks for your elaborate answer, @theigl!

And let me say that I completely share your thoughts/feelings, @endink! 🙂

From my perspective we're sorting out the last minor issues, and I'd hope that we can publish a release in the next weeks. The goal is to release a stable version for sure 🙂

Maybe we could publish RC7 as 5.0.0 already, but I'd prefer to get the last things sorted out (you could check the issues reported latest).
If you ask me, you could also go with RC7 as a 5.0.0 release already 😉

[magro]

@theigl Good to know about your observations regarding performance. Would you expect changes that impact (serialization or binary) compatibility? (of course that's hard to predict 😉)

[theigl]

Would you expect changes that impact (serialization or binary) compatibility?

@magro: I'm pretty sure it will not impact compatibility. My prime suspect is the new implementation of ObjectMap. I'll deploy a version to production today that uses the Kryo4 version of ObjectMap to see if it improves things.

[theigl]

@magro: I was able to resolve the performance issues in #751 and #752 and fixed the two most important known bugs in #745 and #750.

[isaki]

Great work @theigl! Let me know if you need any help!

[magro]

Many thanks @isaki and especially @theigl!

I just merged the mentioned PRs and released RC8.

[theigl]

Thanks @magro! I'll deploy RC8 to production later today and verify my performance improvements.

[nicknezis]

Was the RC8 performance testing successful? I'm eager to see a 5.0.0 release that includes #753 and the recent performance PRs.

[theigl]

Was the RC8 performance testing successful?

Yes and no.

I have spent many hours writing and running JMH benchmarks against most of Kryo's components.

The good news is: Kryo 5 is significantly faster in nearly all benchmarks.
The bad news is: Kryo 5 is still slower than Kryo 4 on my production system.

For the last couple of days I've been working on a JMH benchmark that reflects my production workload as closely as possible. Thousands of classes, potentially tens of thousands of references, object graphs of vastly different sizes.

With that benchmark in place, I managed to zero in on some potential issues. The problem is that I have to deploy each incremental change into production to verify it. I can do that - at most - once a day.

Earlier today, I deployed a version that is already very close to Kryo 4 performance.

My goal is to finish these investigations some time next week.

[theigl]

Good news everyone. I managed to track down and resolve my performance issues. Kryo 5 is now faster than Kryo 4 on my production system as well.

The problems only affect object graphs with references and a large number of different classes.

I created two PRs to address these issues:

• Use CuckooObjectMap from Kryo 4 for class registrations #755
• Customizable value for maximum capacity in MapReferenceResolver #756

Then there is one more housekeeping PR that updates licence headers and copyright dates:

• Upgrade copyright year and add missing licence headers #757

And one open issue that has to discussed but can probably be fixed by adding to the documentation:

• Kryo is reset twice when used in a pool #754

@magro: Please take a look.

[endink]

emmm.......

I didn't expect this issue to be so noisy.

The main thing that gives me this feeling is that I used the RC2 version in a early project, and I thought I would be get the release version before my project was finished, but when my project was done, I saw kryo was still in RC3, RC4, RC5, RC6... and release date go further and further. I have to change the code from 5.0 to 4.0.

So this release strategy gave me a very bad experience.

I'm sorry for the tone of my voice, but the fact is the 5.0 release plan is completely out of control.

BTW: If time is uncertain, might be able to use the beta version instead of the RC version, RC feels like it's not far from release. but nobody thought the RC version would be 2 years.

[magro]

@endink it's good to receive this feedback! I also didn't expect that it will take that long and so many RCs. There were things to rule out again and again, and I at least wanted to prevent to release something where quickly afterwards an incompatible change would be introduced or which contains bigger known issues.

The next time with such a big change of the code base I'd also start to release a beta - definitely something we can learn from this 🙂

At this point many big thanks @theigl, without you the final release would not be that close - awesome how you drive this forward! 👌

[theigl]

Here is a quick summary of my benchmark results:

JMH Settings: -f 2 -wi 5 -i 3 -t 2 -w 2s -r 2s FieldSerializerBenchmark.field

Kryo 3

Benchmark	(objectType)	(references)	Mode	Score	Units
FieldSerializerBenchmark.field	sample	true	thrpt	1201315,620	ops/s
FieldSerializerBenchmark.field	sample	false	thrpt	1756785,379	ops/s
FieldSerializerBenchmark.field	media	true	thrpt	766411,423	ops/s
FieldSerializerBenchmark.field	media	false	thrpt	1256159,311	ops/s

Kryo 4

Benchmark	(objectType)	(references)	Mode	Score	Units
FieldSerializerBenchmark.field	sample	true	thrpt	1186314,214	ops/s
FieldSerializerBenchmark.field	sample	false	thrpt	1751735,252	ops/s
FieldSerializerBenchmark.field	media	true	thrpt	774693,128	ops/s
FieldSerializerBenchmark.field	media	false	thrpt	1226555,621	ops/s

Kryo 5 RC7

Benchmark	(objectType)	(references)	Mode	Score	Units
FieldSerializerBenchmark.field	sample	true	thrpt	558445,033	ops/s
FieldSerializerBenchmark.field	sample	false	thrpt	1417836,700	ops/s
FieldSerializerBenchmark.field	media	true	thrpt	457497,248	ops/s
FieldSerializerBenchmark.field	media	false	thrpt	1162920,147	ops/s

Kryo 5 Master

Benchmark	(objectType)	(references)	Mode	Score	Units
FieldSerializerBenchmark.field	sample	true	thrpt	1251193,272	ops/s
FieldSerializerBenchmark.field	sample	false	thrpt	1708737,251	ops/s
FieldSerializerBenchmark.field	media	true	thrpt	858253,473	ops/s
FieldSerializerBenchmark.field	media	false	thrpt	1194925,892	ops/s

Kryo 5 Master (No Generics)

Benchmark	(objectType)	(references)	Mode	Score	Units
FieldSerializerBenchmark.field	sample	true	thrpt	1372031,253	ops/s
FieldSerializerBenchmark.field	sample	false	thrpt	1869505,801	ops/s
FieldSerializerBenchmark.field	media	true	thrpt	880767,959	ops/s
FieldSerializerBenchmark.field	media	false	thrpt	1305037,387	ops/s

TLDR: Kryo 5 was significantly slower than Kryo 4 until RC7. Now that all my optimizations have been applied, Kryo 5 and Kryo 4 have very similar out-of-the-box performance. Kryo 5 with setOptimizedGenerics(false) is faster than Kryo 4 in all benchmarks.

[nicknezis]

Just out of curiosity, how does Kryo 5 compare to Kryo 3 (i.e. 3.0.3)? 😄

[theigl]

@nicknezis: I just ran the benchmarks against Kryo 3.0.3 and added it to my comment. 😄

[magro]

@theigl Thanks for your PRs, I just pushed RC9 to maven central - from my point of view this could be the last RC ;-)

Do you want to test RC9 on your production system? Everybody else is also invited to test this version of course!

[theigl]

@magro: Thanks! I'll deploy it on Monday.

We should be good though, because I already have custom versions of ReferenceResolver and ClassResolver in production that contain my patches.

[theigl]

@magro: RC9 works great in production. I have no more changes planned and vote for releasing Kryo 5.0.0.✌️

[nicknezis]

Just following up on an official 5.0.0 release. I'd like to use that in Apache Heron if it will be produced soon.

[isaki]

Just commenting to make sure this doesn't fall through the cracks; how are we looking for a full 5.0.0 release build?

[theigl]

@isaki: I haven't heard from @magro since his last comment. I don't have write access to the repo and we can't move ahead without him.

[theigl]

We have to discuss if we want to add support for records in Kryo 5.0 (#766) or if we wait for the next minor release. Apart from this, the current master is ready for release.

[isaki]

I would argue for a 5.1 for a new feature at this point given 5.0.0 has been in release candidacy for so long, but that is just my opinion. Thanks for the update!

[payne911]

Considering records are still a feature in preview, and will remain in that state for at least a few months, I'm not sure it's worth waiting.

Especially since it's probable many people will wait for Java 17 before including records into production code.

[theigl]

@isaki @payne911: I agree.

The current master can be released as is. I'm just waiting for @magro to return and perform the release.

[nicknezis]

@magro Is there any update on this?

[magro]

I'm so sorry, was too busy at work without any resources left. I'll check the current status now and latest open MRs and then do a release (if my connection on the train allows this).

[magro]

Finally we have the 5.0.0 release (see also https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-5.0.0).

Many thanks at @theigl for all the great work! 👍

[endink]

great job!!

[payne911]

Shouldn't this issue get closed now?

[nicknezis]

Should the 5.X release also have a published kryo-shaded artifact?
https://groups.google.com/g/kryo-users/c/hVlwHmBE_-s