Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Java] Align object array to collection serialization protocol v2 #1229

Open
chaokunyang opened this issue Dec 10, 2023 · 0 comments
Open

[Java] Align object array to collection serialization protocol v2 #1229

chaokunyang opened this issue Dec 10, 2023 · 0 comments
Labels
enhancement New feature or request java

Comments

@chaokunyang
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

In #923 , we implemented the collection protocol in #927 , which can be 1X faster at most. This can improve space cost by 1X at most. Such optimization should be applied to array too.

Describe the solution you'd like

Refactor io.fury.serializer.ArraySerializers.ObjectArraySerializer to forward serialization to FuryArrayAsListSerializer.

  • Wrap array into ArrayAsList`, note this wrapped object can't be reused if nested serialization invoke to this place too.
  • Push array component generics to FuryArrayAsListSerializer
  • If array component doesn't have nested generics, reimplement the serialization in ObjectArraySerializer for better peroformance.

Additional context

#1228
@chaokunyang chaokunyang added enhancement New feature or request java labels Dec 10, 2023
@chaokunyang chaokunyang changed the title [Java] Use collection serialization v2 for array [Java] Align collection serialization protocol v2 for array Dec 10, 2023
@chaokunyang chaokunyang changed the title [Java] Align collection serialization protocol v2 for array [Java] Align object array to collection serialization protocol v2 Dec 10, 2023
@chaokunyang chaokunyang mentioned this issue Dec 21, 2023
Merged
2 tasks
chaokunyang added a commit that referenced this issue Feb 28, 2024
@chaokunyang @PragmaTwice
docs(java): standardizing fury java spec (#1240)
0eee00d 
<!--
Thank you for your contribution!

Please review https://github.com/alipay/fury/blob/main/CONTRIBUTING.rst
before opening a pull request.
-->

## What do these changes do?
This PR refine fury java serialization format spec. The cross-language
object graph serialization spec is similar and will be added in a later
PR, but it needs more discuss.

This PR added some new spec which hasn't been implemented in current
java implementation:
- chunk-by-chunk predictive map serialization: #925
- layed class meta
- new class meta encoding
- #1229 
- object serialization with schema evolution support by auto meta share.

Some parts has been omitted in this spec:
- object serialization with schema evolution support by write field in a
KV like pattern: this will be replaced by schema evolution mode
described in this spec in the future.

Currently fury doesn't provide binary compatibility, the spec may be
revised in the future.

<!-- Please give a short brief about these changes. -->

## Related issue number

<!-- Are there any issues opened that will be resolved by merging this
change? -->
Closes #1239 

#1238

## Check code requirements

- [ ] tests added / passed (if needed)
- [ ] Ensure all linting tests pass, see
[here](https://github.com/alipay/fury/blob/main/CONTRIBUTING.rst) for
how to run them

---------

Co-authored-by: Twice <twice@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request java
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@chaokunyang