Update Wiki Stats

[pascaldekloe]

Can someone with a recent high-end CPU [Xeon or equivalent] run the benchmarks and update the wiki content?

[pascaldekloe]

Thank you @tommyettinger.

[tommyettinger]

Oh, please don't close the issue! There's some work needed still to use any newer Java version, and getting Java 9, 11, or especially 13 results would really be more informative. I don't have much idea of specific reasons why Thrift fails so hard all over; it seems to use javax.annotation and that's not present in Java 11 or 13, while building Thrift requires using a lower Java version (or not building Java support) because they still use Gradle 5.6.2 in the latest release and that doesn't work with Java 13... or with 11, for some other reason I couldn't figure out. Thrift, at least on the JVM, seems like outdated and broken software, and the most important info I need when comparing serializers starts with "does it even work?" I'm fully in favor of removing Thrift entirely from the benchmarks if doing so allows recent Java versions to be tested.

Then again, the Android Gradle Plugin doesn't even seem compatible with Gradle versions newer than 5.6.2 (so it doesn't allow using Java 13 either, last I checked). At least Thrift has company. It's also possible other serializers would block compilation with newer Java even after Thrift was removed or worked-around...

[FelixGV]

I can take a stab at updating the results, but I would probably keep things simple and do it on the currently supported Java version.

My hardware isn't exceptional, a 3 years old MBP, but I suppose it can serve. Let me know if you have any comments or concerns.

[pascaldekloe]

If MBP stands for MacBook Pro then you should get some reasonable numbers. Xeon would be better as that is what most production systems use. Yes please @ the current Java. 👍

[tommyettinger]

Nobody seems to have desktops available any more, heh. I didn't disable the processor clock variation feature ("Turbo Boost") that's on by default on most modern Intel mobile processors. That could affect the results if heat forced Turbo Boost to turn down the clock speed, or if something happened in the background.

[pascaldekloe]

@FelixGV, the project runs with recent Java versions now. Can you please update the wiki now?

[YShow]

Hello everyone, ill be running the benchmarks on a r7 3700x running linux Opensuse Tumbleweed, the scheduler will be on performance so the clock wont drop in any case, ill post results here so people can update the wiki, ill be using java 14.0.7, any requests please ask.

[YShow]

Seems dsl isnt working, im getting this error:
if i remember correctly dsl uses a proprietary json parser made in C# and i dont think it should be considered in this benchmark

com.dslplatform.json.ConfigurationException: Unable to serialize provided object. Failed to find serializer for: class data.media.MediaContent
        at com.dslplatform.json.DslJson.serialize(DslJson.java:2850)
        at serializers.dsljson.DSLJson$DSLJsonSerializer.serialize(DSLJson.java:42)
        at serializers.dsljson.DSLJson$DSLJsonSerializer.serialize(DSLJson.java:20)
        at serializers.BenchmarkBase.checkSingleItem(BenchmarkBase.java:554)
        at serializers.MediaItemBenchmark.checkCorrectness(MediaItemBenchmark.java:33)
        at serializers.BenchmarkBase.runMeasurements(BenchmarkBase.java:402)
        at serializers.BenchmarkBase.runTests(BenchmarkBase.java:281)
        at serializers.BenchmarkBase.runBenchmark(BenchmarkBase.java:76)
        at serializers.MediaItemBenchmark.runBenchmark(MediaItemBenchmark.java:15)
        at serializers.BenchmarkRunner.main(BenchmarkRunner.java:28)
com.dslplatform.json.ConfigurationException: Unable to serialize provided object. Failed to find serializer for: class data.media.MediaContent
        at com.dslplatform.json.DslJson.serialize(DslJson.java:2850)
        at serializers.dsljson.DSLJson$DSLJsonSerializer.serialize(DSLJson.java:42)
        at serializers.dsljson.DSLJson$DSLJsonSerializer.serialize(DSLJson.java:20)
        at serializers.MediaItemBenchmark$2.run(MediaItemBenchmark.java:75)
        at serializers.TestCaseRunner.run(TestCaseRunner.java:24)
        at serializers.BenchmarkBase.warmTest(BenchmarkBase.java:517)
        at serializers.BenchmarkBase.runMeasurements(BenchmarkBase.java:416)
        at serializers.BenchmarkBase.runTests(BenchmarkBase.java:281)
        at serializers.BenchmarkBase.runBenchmark(BenchmarkBase.java:76)
        at serializers.MediaItemBenchmark.runBenchmark(MediaItemBenchmark.java:15)
        at serializers.BenchmarkRunner.main(BenchmarkRunner.java:28)
ERROR: exit code 1

[YShow]

anyway every other one seems to have passed, i had to rename report.texfile to .txt so github would accept the upload so here it is. Ill try to fix the dsl problem or remove it completely.

stats.txt
report.txt

[tommyettinger]

Maybe some change recently broke the DSL-Platform serialization bench? I remember it being extremely fast, in the top 3 most of the time. I don't know if they moved to a toolchain that works with just a JVM. Thanks for running this, YShow !

[YShow]

Maybe some change recently broke the DSL-Platform serialization bench? I remember it being extremely fast, in the top 3 most of the time. I don't know if they moved to a toolchain that works with just a JVM. Thanks for running this, YShow !

Yes its one of the fastest serializers that exists but its not one of the most used, ill try to fix the error and run the bench again, ill also do some working on the code with minor fixes that may help some benchmarks

[YShow]

I have been upgrading the benchmarks to newer libs versions, should some of the old serializers be remove or should i keep them?
Seems a lot has changed in most libs, it will take some time before i can get a running version

[pascaldekloe]

Thanks @YShow. I probably broke DLS with commit 65c0bf0 recently. Will fix tomorrow evening if nobody beats me to it. The amount of incompatible changes is surprising indeed. If something old breaks, and the upgrade path costs too much time, then remove it. We shouldn't let this repo rot into an obsolete state, just because some implementations mess up big time.

[pascaldekloe]

Just had a quick look. It seems we need lib/dsl-clc-1.9.9-jar-with-dependencies.jar and lib/dsl-json-processor-1.9.5.jar too. This version on longer supports "array" though and it runs out of memory. 🤦 Feel free to drop it. Mono dependency + closed source+ broken state is just too much.

[YShow]

Ill make a list of all the serializers that have too much breakage and u can choose which one should we maintain, in my opinion we could remove some not so famous libraries that are outdated or not talked about anymore like jsonij of which changed a lot and doesn't have a JSONMarshaler anymore

[zapov]

@pascaldekloe you could have pinged me if you had issues instead of creating confusion ;(
Your commit: 65c0bf0 broke the implementation as you removed 2 jars, but put back only 1. You need to put java8 jar back too. Also not a fan of you changing the implementation. Was there a reason for that?

Also the latest PR I did drop the Mono dependency so it definitely not needed here.

[pascaldekloe]

No confusion. Like I said @zapov: libs are highly incompatible, even in-between minor versions.

[zapov]

:sigh:

DSL-JSON works just fine as long as you keep the java8 annotation processor jar.
Fix for your commit: #86

[zapov]

@YShow can you rerun it now from the master? DSL-JSON should work now

[YShow]

@YShow can you rerun it now from the master? DSL-JSON should work now

Sure, ill be running right now, as soon it finishes im sending the results

[YShow]

@YShow can you rerun it now from the master? DSL-JSON should work now

Sure, ill be running right now, as soon it finishes im sending the results

All benchmarks passed!!

report.txt
stats.txt

[pascaldekloe]

Thanks for correcting the DSL patch @zapov. The numbers from @YShow with the updates in place plus a newer JVM still won't match a 7 year old Mac Pro. No notable changes in ranking either. ✔

[tommyettinger]

Yeah, it's funny how Java 8 is still often faster than later Java versions. Funny in a sad way, but funny I guess...

Thanks as well to everyone here for being mindful of performance! It's overlooked too often. I'm still somewhat irked by the old assumption that Java is slow, when there's serious effort applied to benchmarking and profiling here on the JVM... Plus, now that I'm attempting to port some Java code to C#, I realize how little attention has been paid to micro-optimizations on .NET for over a decade. No bitwise rotation optimization except on some very recent .NET Core versions; Java has had that since Java 6 or earlier I think. Nothing that compiles down to a popcnt instruction, against except for very recent .NET Core; Java has had Long.bitCount() for a long time. This completely derailed an attempt to port a specialized bitset data structure, and I had to change the design entirely to one that's not nearly as efficient. I'm mostly pretty happy with coding for performance in Java. There's a few special processor instructions that might be nice to have for serialization or compression, but there's also JNI as an option for especially slow methods. Sorry to derail, it's just nice to see JVM performance being valued 😄

[YShow]

Honestly the change from java8 parallelgc to g1gc on recent version might be the reason for the not so great performance advantage, but still, recent version does have some very nice optimization in other areas like memory usage with string with StringCompact and StringDedup, quality of code and other things that i don't remember by head. Although this benchmark can be flawed because JVM might do some things that may not change on one version to another thats why i prefer benchmarking everything in JMH. This benchmark may not show performance improvements but sure did on memory reductions, another thing to consider is that serialization is already really fast so not much change would apply i think, correct me if im wrong.

[cowtowncoder]

I think I saw non-trivial performance degradation with JVM 9 two years ago and tracked it down to GC change as @YShow suggested. It seemed peculiar to have -20% throughput for CPU-bound, not GC heavy single-threaded test but I assume trade-off for more typical heavier multi-threading is worth it.