wRPC binary serialization with data structure versioning support #480
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…via workflow-store::fs)
* range support added for transactions pagination * cargo fmt/clippy
* provisional RpcConnection propagation via RpcApi methods * lints
…mpatibility patterns;
…to rpc-api-version
|
Merged into #506 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements support for "wRPC serialization versioning".
This is accomplished using a set of primitives and macros added to
workflow-rs(workflow-serializercrate) that implementsSerializertrait and a helperSerializable<T>(T)struct (https://github.com/workflow-rs/workflow-rs/tree/serializer/serializer/src).wRPC calls require
BorshSerializeandBorshDeserialize. These traits have been removed form RK RPC data structures. Instead they now implementSerializertrait to perform binary serialization. TheSerializable<T>(T) where T: Serializerstruct implementsBorshSerializeandBorshDeserializeand is#[repr(transparent)], effectively converting data structures that implementSerializertrait toBorshSerializeandBorshDeserializeinvocations.Each RPC (and related) data structure that opts-in for versioning now manually serializes it's fields, where the first field is always a version number (currently
u32). By checking the version number on deserialization, structures can opt-in to deserialize old or new binary layouts.This is a "future compatible" implementation that allows newer clients/nodes to understand older requests, but will not allow older clients/nodes to understand newer requests. (each structure can implement forward compatibility internally if desired by pre-buffering and storing the size of the binary payload and ignoring (while skipping) additional data; this is not really a common approach as it poses other issues where field types can change, breaking serialization).
By augmenting Kaspa wRPC client and server interfaces with these traits, the constraints on these interfaces change requiring all RPC data structures to implement the
Serializertrait, which is now implemented for each such struct.The implementation of serialization is done beside each RPC data structure. A new
rpc/core/src/test.rsfile implements serialization unit tests designed to catch basic data misalignments.IMPORTANT: When this PR is merged, we will need to make a new node release as well as the corresponding WASM SDK and KNG releases as this will break wRPC Borsh encoding compatibility (this PR addresses that very problem, allowing us to perform protocol upgrades in the future without breaking such compatibility).