Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(java): scoped meta share mode for type forward/backward compaibility #1660

Merged
merged 15 commits into from
May 30, 2024
Merged

feat(java): scoped meta share mode for type forward/backward compaibility #1660

merged 15 commits into from
May 30, 2024

Conversation

chaokunyang
Copy link
Collaborator

@chaokunyang chaokunyang commented May 29, 2024 •
edited
Loading

What does this PR do?

This PR implements scoped meta share mode for type forward/backward compaibility

Related issues

#202

Does this PR introduce any user-facing change?

  • Does this PR introduce any public API change?
  • Does this PR introduce any binary protocol compatibility change?

Benchmark

Perf increased from 1900102.586 to 2430410.064

Before:

Benchmark                                                       (bufferType)   (objectType)  (references)   Mode  Cnt        Score        Error  Units
fury_deserialize                              array  MEDIA_CONTENT         false  thrpt   10  2734151.212 ± 253921.628  ops/s
fury_deserialize_compatible                   array  MEDIA_CONTENT         false  thrpt   10  1900102.586 ±  62176.872  ops/s
furymetashared_deserialize_compatible         array  MEDIA_CONTENT         false  thrpt   10  3011439.327 ± 260518.752  ops/s

After:

Benchmark                                                       (bufferType)   (objectType)  (references)   Mode  Cnt        Score        Error  Units
fury_deserialize                              array  MEDIA_CONTENT         false  thrpt   10  2661186.814 ± 279377.198  ops/s
fury_deserialize_compatible                   array  MEDIA_CONTENT         false  thrpt   10  2430410.064 ± 164165.865  ops/s
furymetashared_deserialize_compatible         array  MEDIA_CONTENT         false  thrpt   10  3098083.064 ± 259391.053  ops/s

Size decreased from 732 to 577:

Before
2024-05-30 01:00:49 INFO  FuryState:157 [fury_deserialize_compatible-jmh-worker-1] - ======> Fury | MEDIA_CONTENT | false | array | 732 |

After
2024-05-30 12:57:00 INFO  FuryState:157 [fury_deserialize_compatible-jmh-worker-1] - ======> Fury | MEDIA_CONTENT | false | array | 577 |

@chaokunyang
Copy link
Collaborator Author

Hi @MrChang0, this PR should make the meta share mode replace kv compatible mode for better performance and smaller space cost

theweipeng
theweipeng approved these changes May 30, 2024
View reviewed changes
Copy link
Member

@theweipeng theweipeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MrChang0
Copy link
Contributor

It fine for me, I will try to experiment this feature.
btw, how about increase that init buffer size in org.apache.fury.meta.ClassDefEncoder#encodeClassDef. take less growth operate may take more performance.

@chaokunyang
Copy link
Collaborator Author

encodeClassDef

Would you like to submit a PR?

@chaokunyang
Copy link
Collaborator Author

Struct perf:

Before:
fury_deserialize                    array        STRUCT         false  thrpt   10  4103952.355 ± 461542.704  ops/s
fury_deserialize_compatible         array        STRUCT         false  thrpt   10  2104530.838 ± 283950.067  ops/s

After:
Benchmark                                             (bufferType)  (objectType)  (references)   Mode  Cnt        Score        Error  Units
fury_deserialize                    array        STRUCT         false  thrpt   10  3846945.672 ± 281139.320  ops/s
fury_deserialize_compatible         array        STRUCT         false  thrpt   10  3119394.319 ± 476422.038  ops/s

Size:

Before:
2024-05-30 02:06:42 INFO  FuryState:157 [fury_deserialize-jmh-worker-1] - ======> Fury | STRUCT | false | array | 710 |
2024-05-30 02:07:19 INFO  FuryState:157 [fury_deserialize_compatible-jmh-worker-1] - ======> Fury | STRUCT | false | array | 1134 |

After:
2024-05-30 02:01:16 INFO  FuryState:157 [fury_deserialize-jmh-worker-1] - ======> Fury | STRUCT | false | array | 710 |
2024-05-30 02:01:52 INFO  FuryState:157 [fury_deserialize_compatible-jmh-worker-1] - ======> Fury | STRUCT | false | array | 1227 |

Size bloat a little, we can optimize it later

@chaokunyang chaokunyang merged commit a0a6d7b into apache:main May 30, 2024
32 checks passed
@chaokunyang chaokunyang mentioned this pull request May 30, 2024
Merged
2 tasks
chaokunyang added a commit that referenced this pull request Jun 1, 2024
@chaokunyang
feat(java): support meta compression by Deflater (#1663)
da5f847 
## What does this PR do?

This PR support meta compression and add Deflater as default compressor.

In our test, it can compress meta by reduce size of **243** without
introducing any performance cost:

```
before:
Fury | STRUCT | false | array | 1227 |

after
STRUCT | false | array | 984 |

```

## Related issues
#1660 


## Does this PR introduce any user-facing change?

<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/incubator-fury/issues/new/choose)
describing the need to do so and update the document if necessary.
-->

- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?


## Benchmark

<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@theweipeng theweipeng theweipeng approved these changes

@PragmaTwice PragmaTwice Awaiting requested review from PragmaTwice

@LiangliangSui LiangliangSui Awaiting requested review from LiangliangSui

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@chaokunyang @MrChang0 @theweipeng