Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabled cmake Release build type (-O3) and LTO for C/C++ code #27

Merged
merged 1 commit into from
Jul 24, 2021

Conversation

hilarious-viking
Copy link
Contributor

Few weeks ago compared brotli4j with 5 years old jbrotli with my private benchmarks, and was surprised that brotli4j with compression level 10 in many cases is 2-3x slower than old jbrotli

Issue is that only JNI C++ code is compiled with -O3 optimizations, and all Brotli C code is compiled with default level, which is -O0 + Debug info

...
[ 88%] Building C object CMakeFiles/brotli.dir/brotli/dec/bit_reader.c.o
/usr/bin/cc -Dbrotli_EXPORTS -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux -I/home/andrew/workspace/Brotli4j/brotli/include -I/home/andrew/workspace/Brotli4j/brotli/common -fPIC -MD -MT CMakeFiles/brotli.dir/brotli/dec/bit_reader.c.o -MF CMakeFiles/brotli.dir/brotli/dec/bit_reader.c.o.d -o CMakeFiles/brotli.dir/brotli/dec/bit_reader.c.o -c /home/andrew/workspace/Brotli4j/brotli/dec/bit_reader.c
[ 91%] Building CXX object CMakeFiles/brotli.dir/natives/src/main/cpp/common_jni.cc.o
/usr/bin/c++ -Dbrotli_EXPORTS -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux -I/home/andrew/workspace/Brotli4j/brotli/include -I/home/andrew/workspace/Brotli4j/brotli/common -Wall -O3 -fPIC -std=gnu++11 -MD -MT CMakeFiles/brotli.dir/natives/src/main/cpp/common_jni.cc.o -MF CMakeFiles/brotli.dir/natives/src/main/cpp/common_jni.cc.o.d -o CMakeFiles/brotli.dir/natives/src/main/cpp/common_jni.cc.o -c /home/andrew/workspace/Brotli4j/natives/src/main/cpp/common_jni.cc
...

Log is obtained with
SET (CMAKE_VERBOSE_MAKEFILE ON)
As result, on linux, libbrotli.so is pretty large 3.7MB and resulting native jar is 1.8MB

This PR

  • enables by default build type: Release which enables -O3 -DNDEBUG (the same way as in brotli cmake file). This will make brotli4j with compression level 10 up to 2x faster than jbrotli, so 4-6x faster than now. libbrotli.so 850KB and native jar 386KB
  • added LTO, which surprisingly improves level 9 in many cases by 5-10x, again compared to jbrotli, for files <=64KB
  • other levels than 9 and 10 seems also been affected, but effects seems negligible, at best 5-10% improvement

Test system:
OS: Ubuntu 18.04.5 LTS
CPU 4 core: Intel(R) Core(TM) i5-7440HQ CPU @ 2.80GHz
GLIBC 2.27-3ubuntu1.4
C/C++ compiler GNU 11.1.0
cmake 3.21.0
Java: /usr/lib/jvm/java-8-oracle/bin/java (found version "1.8.0.201")

My benchmarks take about 12 hours, here sample output, ns/byte is column to consider for speed (ignore ms), brotli4=brotli4j

Pattern: ^.*\.(xml|blm|txt)$
Number of test files: 40, is text: 40, is utf8: 40
Size: low=32Kb, high=48Kb
Run       : 0, size: 1549341
Entropy   : 0.6288340791378071
codec   lvl	size	total	s/file	total		ns/byte
lz4	1	418509	2 ms	0 ms	67135 ns	1ns
prod	0	254423	333 ms	8 ms	8337830 ns	215ns
prodOld	0	276392	48 ms	1 ms	1215603 ns	31ns
deflate	1	315680	12 ms	0 ms	324673 ns	8ns
deflate	6	278555	34 ms	0 ms	860996 ns	22ns
deflate	9	276392	48 ms	1 ms	1216792 ns	31ns
brotli4	1	299730	5 ms	0 ms	149004 ns	3ns
brotli4	4	272242	23 ms	0 ms	576361 ns	14ns
brotli4	5	262525	34 ms	0 ms	868104 ns	22ns
brotli4	6	261684	36 ms	0 ms	910252 ns	23ns
brotli4	7	261033	41 ms	1 ms	1048639 ns	27ns
brotli4	8	260581	45 ms	1 ms	1149514 ns	29ns
brotli4	9	260337	52 ms	1 ms	1310648 ns	33ns
brotli4	10	227481	875 ms	21 ms	21887162 ns	565ns
jbrotli	1	299712	6 ms	0 ms	166962 ns	4ns
jbrotli	4	272269	21 ms	0 ms	545736 ns	14ns
jbrotli	5	262916	32 ms	0 ms	806092 ns	20ns
jbrotli	6	262037	34 ms	0 ms	866372 ns	22ns
jbrotli	7	261330	43 ms	1 ms	1093978 ns	28ns
jbrotli	8	260903	46 ms	1 ms	1175014 ns	30ns
jbrotli	9	260726	287 ms	7 ms	7176603 ns	185ns
jbrotli	10	227291	1826 ms	45 ms	45663620 ns	1178ns
zstd	1	291057	5 ms	0 ms	138455 ns	3ns
zstd	3	285924	6 ms	0 ms	169190 ns	4ns
zstd	6	273657	17 ms	0 ms	429056 ns	11ns
zstd	7	272412	19 ms	0 ms	490370 ns	12ns
zstd	8	271471	22 ms	0 ms	552990 ns	14ns
zstd	9	270841	25 ms	0 ms	630298 ns	16ns
zstd	10	270403	29 ms	0 ms	732569 ns	18ns
zstd	11	270047	48 ms	1 ms	1213507 ns	31ns
zstd	12	269986	65 ms	1 ms	1630983 ns	42ns
zstd	13	270654	127 ms	3 ms	3185662 ns	82ns
zstd	14	270891	162 ms	4 ms	4061578 ns	104ns
zstd	15	263421	249 ms	6 ms	6234321 ns	160ns
zstd	16	262797	399 ms	9 ms	9981701 ns	257ns
zstd	17	262733	468 ms	11 ms	11715571 ns	302ns
zstd	18	262714	504 ms	12 ms	12624275 ns	325ns
zstd	19	262050	849 ms	21 ms	21226754 ns	548ns
zstd	20	262004	1018 ms	25 ms	25474546 ns	657ns
zstd	21	261999	996 ms	24 ms	24916614 ns	643ns
zstd	22	261996	1012 ms	25 ms	25315279 ns	653ns
bzip2	0	254423	275 ms	6 ms	6898941 ns	178ns
bzip2	9	254423	291 ms	7 ms	7298205 ns	188ns

@hyperxpro
Copy link
Owner

Thanks for the PR. I will review it, test and get back to you. :)

@hyperxpro hyperxpro merged commit bb7e572 into hyperxpro:main Jul 24, 2021
@hyperxpro
Copy link
Owner

@hilarious-viking

Thanks a lot :)

hyperxpro added a commit that referenced this pull request Jul 24, 2021
normanmaurer pushed a commit to netty/netty that referenced this pull request Jul 26, 2021
Motivation:
Brotli4j had some changes for performance improvements. So we should upgrade to the latest version of Brotli4j.

See hyperxpro/Brotli4j#27

Modification:
Upgraded Broti4j from 1.4.2 to 1.5.0.

Result:
Up-to-date Broti4j library.
normanmaurer pushed a commit to netty/netty that referenced this pull request Jul 26, 2021
Motivation:
Brotli4j had some changes for performance improvements. So we should upgrade to the latest version of Brotli4j.

See hyperxpro/Brotli4j#27

Modification:
Upgraded Broti4j from 1.4.2 to 1.5.0.

Result:
Up-to-date Broti4j library.
laosijikaichele pushed a commit to laosijikaichele/netty that referenced this pull request Dec 16, 2021
Motivation:
Brotli4j had some changes for performance improvements. So we should upgrade to the latest version of Brotli4j.

See hyperxpro/Brotli4j#27

Modification:
Upgraded Broti4j from 1.4.2 to 1.5.0.

Result:
Up-to-date Broti4j library.
laosijikaichele pushed a commit to laosijikaichele/netty that referenced this pull request Dec 16, 2021
Motivation:
Brotli4j had some changes for performance improvements. So we should upgrade to the latest version of Brotli4j.

See hyperxpro/Brotli4j#27

Modification:
Upgraded Broti4j from 1.4.2 to 1.5.0.

Result:
Up-to-date Broti4j library.
raidyue pushed a commit to raidyue/netty that referenced this pull request Jul 8, 2022
Motivation:
Brotli4j had some changes for performance improvements. So we should upgrade to the latest version of Brotli4j.

See hyperxpro/Brotli4j#27

Modification:
Upgraded Broti4j from 1.4.2 to 1.5.0.

Result:
Up-to-date Broti4j library.
@hilarious-viking hilarious-viking deleted the compile-O3-LTO branch May 15, 2023 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants