VReplication errors when fixed-length binary column is used as sharding key

[deepthi]

Overview of the Issue

For fixed length binary columns, mysql internally pads the values on the right will nulls. However the binlogs contain the original value. This leads to a problem in vreplication workflows if a binary column is used as a sharding key.

During the copy phase the row is read using a mysql select and the column has the padded value. However while replicating we get the unpadded value. This means that we get different keyspace_ids. We found bugs where the copy phase inserted the row in one shard but the update during the catchup phase routed this row to another shard where, of course, the update failed.

[deepthi]

A fix has already been made in #7969. However that PR references a catch-all issue #3984. Creating an issue for this specific bug for reporting purposes.

[deepthi]

Tasks

• Complete testing of VReplication: Pad binlog values for binary() columns to match the value returned by mysql selects #7969 on a production size dataset
• Document as a known issue on 10.0, 9.0. 8.0
• Backport to 10.0 and 8.0 after fix has been validated. Already backported to 9.0.

EDIT 5/12/2021: @aquarapid has reported that the resharding diffs they were seeing are now resolved.