[server] [client] [test] Global RT DIV: Chunking Support

[KaiSernLim]

Summary

Continuation from #1257. Schema changes in #1523.

This PR mainly focuses on adding chunking support for DIV messages when they are produced to Kafka topics, as the size of the DIV message can surpass the ~1MB Kafka message limit. The existing chunking mechanism is reused, including the CHUNK and CHUNKED_VALUE_MANIFEST values in the message's schemaId:

Every DIV message has GLOBAL_RT_DIV for the header byte in its KafkaKey. The corresponding KafkaMessageEnvelope has a Put payload utilizing the putValue field containing the GlobalRtDiv data, and which has the following schemaId:

1. If the DIV message is non-chunked, the KME schemaId is set to the current protocol version of GLOBAL_RT_DIV.
2. If the DIV message is chunk message, the KME schemaId is set to CHUNK.
3. If the DIV message is a chunk manifest message, the KME schemaId is set to CHUNKED_VALUE_MANIFEST. The schemaId of the ChunkedValueManifest will be the current protocol version of GLOBAL_RT_DIV.

Changes

1. Added a new MessageType called GlobalRtDiv, which reuses the Put message type format and objects. When the Venice server encounters a message with KafkaKey containing the GlobalRtDiv header byte, it will know to process this message differently from a regular Put.
   • The only indication of this new GlobalRtDiv message type is the header byte in KafkaKey. Otherwise, it's identical to a regular Put.
   • KafkaMessageEnvelope.avsc will not be updated to avoid the unnecessary risk of incompatible avro formats when upgrading the cluster.
   • The risk in not creating a dedicated GlobalRtDiv message type in KME is that the GlobalRtDiv objects will be processed as user records and stored in the storage engine, which seems to be much less scary than a cluster upgrade issue.
2. GlobalRtDiv messages should not be processed if they originate from remote VT and RT, because those are invalid scenarios. These two conditions are checked.

Minor Changes

1. Renamed resetUpstreamOffsetMap() to mergeUpstreamOffsets() in OffsetRecord.
2. Updated toString() in KafkaKey, which incorrectly assumed all messages would be ControlMessage, Put, or Delete. This misses Update messages and the new GlobalRtDiv message that is being added.
3. Added various helper methods (buildPutPayload() and buildManifestPayload()) in VeniceWriter for creating the Put payloads and when chunking is involved.

Testing

1. Unit Tests
   1. testGlobalRtDivChunking() in VeniceWriterUnitTest
   2. testShouldProcessRecordForGlobalRtDivMessage() in StoreIngestionTaskTest
   3. testProcessGlobalRtDivMessage() in StoreIngestionTaskTest
2. Integration Test
   1. testChunkedDiv() in TestGlobalRtDiv

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.

[gaojieliu]

I would like to call out another item we didn't discuss in the review meeting:

1. Cleanup the Global RT DIV messages from VT.

This new type of message can be large and with chunking support, we are leaking the data chunks as chunk id is unique.
I was thinking some strategy to let follower to send out a Kafka delete or Kafka message with empty value for the previous key after consuming a new Global RT DIV message, and if we don't do the cleanup, the size of Kafka topic might grow a lot depending on the sending frequency.
Maybe we don't need to implement such cleanup in the MVP, but I think eventually, we need some way to clean them up from the version topics.

[lluwm]

Looks good overall. Left a few minor comments. ChunkAssembler has been mentioned in the description part, can we remove it if it's not part of this change anymore?

[lluwm]

LGTM!