[1/N][zero serialization] Create data structure for passing encode chunks #690
Conversation
| return result, nil | ||
| } | ||
|
|
||
| func (cd *ChunksData) ToGobFormat() (*ChunksData, error) { |
There was a problem hiding this comment.
The ToGobForamt and ToGnarkFormat conversions are used to handle compatibility when we enable the Gnark at Encoder server: during the transition, some Encoder servers may return new format (Gnark) and some may still return the old (Gob), and the EncoderClient/Batcher will need to work correctly during transition.
There was a problem hiding this comment.
We could just first drain all the encoder pods to 0. Then quickly spin up new pods. But it could results in some increased latency
There was a problem hiding this comment.
Yep. This is really a choice between software complexity v.s. operation complexity with a short/temporary latency spike. I'm just to make operation simpler here with some added software complexity.
| return size | ||
| } | ||
|
|
||
| func (cd *ChunksData) FlattenToBundle() ([]byte, error) { |
There was a problem hiding this comment.
This is the part that will replace the serialization of chunks when dispersing them to nodes.
| if cd.Format != GnarkChunkEncodingFormat { | ||
| return nil, fmt.Errorf("unsupported chunk encoding format: %d", cd.Format) | ||
| } | ||
| gobChunks := make([][]byte, 0, len(cd.Chunks)) |
There was a problem hiding this comment.
So for a while, serialization will still be done in batcher?
There was a problem hiding this comment.
I guess the plan is to gob encode in the encoder.
There was a problem hiding this comment.
Yes, the serialization will happen only once, which will be at Encoder. That'll in a separate PR.
Why are these changes needed?
Create a
ChunksDatadata structure for the encoded chunks data. This will be used to pass chunks around Encoder/Batcher/Dispatcher and eventually used to create dispersal requests to operator nodes.Checks