Collations Module Integration

[king-11]

Description

Integration of Collation Module into Vitess Engine

• MySQL-compatible Collations #8991

Adds in the collation module into the Vitess Code which can then be used to replace instances of the weight_string function used in the vitess codebase. The major change that this PR makes is with the NullSafeComparator

vitess/go/vt/vtgate/evalengine/arithmetic.go

Line 187 in 315d907

func NullsafeCompare(v1, v2 sqltypes.Value) (int, error) {

which now takes in additional collation info parameter which is used by the collations.LookupByID to find out the collation and then use it to collate the varchar strings based on their collation set.

Update Function Calls

So given the NullSafeComparator needs access to collation info we intend to provide the information available to it by updating the function calls and trying to fetch charset info from the result or Fields available at the least distant parent function calls and trickle it down the succeeding calls.

Related Issue(s)

• Collation and Character set support #8606

Checklist

• Should this PR be backported?
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

[vmg]

So I ran some integration tests for this yesterday, and we got some bad results. Here's a log:

 collations_test.go:175: exec: CREATE TABLE unicode (rowid INTEGER PRIMARY KEY, str_coll_utf8mb4_0900_ai_ci CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci, str_coll_utf8mb4_unicode_ci CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, str_coll_utf8mb4_unicode_520_ci CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci);
    collations_test.go:175: exec: INSERT INTO unicode VALUES (0 , _utf8mb4 X'61626320c3a6c3b8c3a520e697a5e69cace8aa9e' COLLATE "utf8mb4_0900_ai_ci", _utf8mb4 X'61626320c3a6c3b8c3a520e697a5e69cace8aa9e' COLLATE "utf8mb4_unicode_ci", _utf8mb4 X'61626320c3a6c3b8c3a520e697a5e69cace8aa9e' COLLATE "utf8mb4_unicode_520_ci")
    collations_test.go:175: exec: SELECT str_coll_utf8mb4_0900_ai_ci, WEIGHT_STRING(str_coll_utf8mb4_0900_ai_ci), str_coll_utf8mb4_unicode_ci, WEIGHT_STRING(str_coll_utf8mb4_unicode_ci), str_coll_utf8mb4_unicode_520_ci, WEIGHT_STRING(str_coll_utf8mb4_unicode_520_ci) FROM unicode
    collations_test.go:269: field0(str_coll_utf8mb4_0900_ai_ci) charsetnr = 33
    collations_test.go:269: field1(WEIGHT_STRING(str_coll_utf8mb4_0900_ai_ci)) charsetnr = 63
    collations_test.go:269: field2(str_coll_utf8mb4_unicode_ci) charsetnr = 33
    collations_test.go:269: field3(WEIGHT_STRING(str_coll_utf8mb4_unicode_ci)) charsetnr = 63
    collations_test.go:269: field4(str_coll_utf8mb4_unicode_520_ci) charsetnr = 33
    collations_test.go:269: field5(WEIGHT_STRING(str_coll_utf8mb4_unicode_520_ci)) charsetnr = 63

As you can see, we're creating a simple table where each column has a different charset + collation and inserting the same string on it. When performing a SELECT on the different columns, we can see that the charsetnr field in the returned Fields (i.e. the field that this PR is using for detecting the collation for a column) is always set to the default collation for the connection, not to the actual collation from the original table.

This is bad! It looks like a bug in MySQL but this behavior is not going to change in the future, so we need to work around it.

@harshit-gangal had some ideas on using Vitess' table schemas to detect these collations instead of depending on the Field data. Harshit, could you please explain these to @king-11? How confident are you feeling about this approach?

[systay]

General comment - this PR all about collations and not so much about charsets. We should use the word collation and not charset for this. https://dev.mysql.com/doc/refman/8.0/en/charset-general.html

[systay]

Another general comment - since we need to do planner work to make sure that the engine primitives and evalengine get the right info, what about focusing this PR to make it work well with grouping (orderedAggregate)? We can work on the rest of the primitives once we are happy with grouping.

[vmg]

@systay: Yes, but do note that the field in Request.Fields that contains the numeric identifier for the collation is actually called charsetnr, even though it's a collation identifier. MySQL is silly like that.

[vmg]

what about focusing this PR to make it work well with grouping (orderedAggregate)? We can work on the rest of the primitives once we are happy with grouping.

This sounds like a great idea. Let's make grouping work perfectly and leave the other callsites for another time. This means that everywhere else we don't pass a collation ID to the comparison function. That should be an easy way to unblock @king-11.

[systay]

@systay: Yes, but do note that the field in Request.Fields that contains the numeric identifier for the collation is actually called charsetnr, even though it's a collation identifier. MySQL is silly like that.

My reading of https://dev.mysql.com/doc/internals/en/character-set.html#packet-Protocol::CharacterSet makes me think that it is the charset, and not the collation the charsetnr is representing. With charsets come a default collation, which is how we can get the collation from a charset. 🤷

You are right, it's all very confusing. Anyway - does it make sense to not perpetuate this confusion more than necessary in our code base? :)

[king-11]

@systay @frouioui awaiting your review made the changes and added in tests

[frouioui]

Nothing more than what @systay wrote to add. His comments need to be addressed, otherwise lgtm

[king-11]

@systay made the changes as per your review.

[systay]

🎉

[shlomi-noach]

I'm not sure how, but this seems to have broken main. The following line:

vitess/go/vt/vtgate/evalengine/arithmetic.go

Line 227 in ab2f0b4

collation := collations.FromID(collationID)

Does not compile:

# vitess.io/vitess/go/vt/vtgate/evalengine
go/vt/vtgate/evalengine/arithmetic.go:227:16: undefined: collations.FromID

right now unit tests and some vreplication tests are failing.

[systay]

#9131 contains the fix for main ☝️