-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug Report: Inconsistent ENUM and SET handling in VStream/vstreamer #15750
Comments
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Overview of the Issue
In
VReplication
workflows and theVStream
API there is a copy phase (initial snapshot copy) and a running/replicating phase. You can read more about the lifecycle here.For the copy phase, a
rowstreamer
(a specific type ofvstreamer
) streams the rows resulting from a query used to get a consistent snapshot of the table at that logical point in time (a GTID set which matches the consistent snapshot of the table). For the running/replicating phase a standardvstreamer
streams filtered binary log events from the source tablet to the target tablet(s).Because the
rowstreamer
sends the results of a query, forENUM
andSET
columns it sends the string value that it gets back from MySQL. Because thevstreamer
sends filtered binary log events, forENUM
andSET
columns it sends the integer based value as that's what is in the binary log events.This has been (partially) addressed in the past on the consumer side. For
VReplication
workflows the consumers arevcopier
for therowstreamer
andvplayer
for thevstreamer
and thevplayer
started doing the integer to string mapping in #15349 (this was done as it caused problems for Vitess OnlineDDL when the schema change was shuffling the order of elements in the ENUM around). InVStream
consumers such as the Debezium Vitess connector and the PlanetScale Airbyte connector this mapping was done when processing the incomingVEvents
(debezium connector PR, airbyte connector PR).Rather than pushing this work on to each current and future consumer, we should unify the behavior in the
VReplication
vstreamer
and its subtypes such asrowstreamer
.Reproduction Steps
Setup the test env:
Modify the
vstream_client
example to stream everything from thecustomer
table in the unshardedcommerce
keyspace:In another Terminal, start a vtgate VStream of the
customer
table. It will start with the copy phase and then continue on in the running/replicating phase:Generate some more data now that the vstream is past the copy phase and in the running/replicating phase:
Clean up:
Sample results:
Binary Version
❯ vtgate --version vtgate version Version: 20.0.0-SNAPSHOT (Git revision da1301b1cec3ba7c12b148eda94d7fbaa063e6ef branch 'main') built on Thu Apr 18 10:44:19 EDT 2024 by matt@pslord.local using go1.22.2 darwin/arm64
Operating System and Environment details
Log Fragments
No response
The text was updated successfully, but these errors were encountered: