Collation and Character set support

[GuptaManan100]

Feature Description

Vitess does not yet have support for collations and character-sets. So, to compare varchar strings Vitess needs to rely on WEIGHT_STRING function for now. As per MySQL documentation, WEIGHT_STRING is a debugging function, meant only for internal use. We can move away from this implementation after we have collation and character-set support in Vitess

Use Case(s)

• Having the ability to compare strings using collation and character set support we will be able to better implement ORDER BY, GROUP BY, JOIN.
• It will also allow us to leverage more advanced join techniques than what we currently implement.

[king-11]

Hey, @GuptaManan100 I would like to apply for this project under LFX. I am not much familiar with Vitess as of now can you point me somewhere I can look at current implementation so that I can design the changes that need to be made. Thanks 😄

[GuptaManan100]

Hello, @king-11 https://github.com/vitessio/vitess/tree/main/go/vt/vtgate/planbuilder is the package which has the code for building the plans of how queries are to be executed in Vitess. You can look at this https://github.com/vitessio/vitess/blob/main/go/vt/vtgate/planbuilder/testdata/aggr_cases.txt file which contains the test data for aggregation queries. You will find weight_string function added to a lot of those. You can run the tests locally from https://github.com/vitessio/vitess/blob/main/go/vt/vtgate/planbuilder/plan_test.go file.

[GuptaManan100]

@rgrupesh There is no other explicit requirement. The mentor for this project is going to be @vmg.

[rgrupesh]

@rgrupesh There is no other explicit requirement. The mentor for this project is going to be @vmg.

Thanks again @GuptaManan100 !

[vmg]

For anybody else who is interested on applying for this task: the actual integration into Vitess is not that important, I will assist you with it once you've implemented the collation code.

The most important thing for your application is understanding what are Character Sets and Collations in MySQL, how do they work, and how they're implemented (i.e. https://dev.mysql.com/doc/refman/8.0/en/charset-charsets.html), then looking at the x/text/collate package from the Go maintainers (https://pkg.go.dev/golang.org/x/text/collate) and figuring out: how many of the existing MySQL collations are implemented in that package, how many behave exactly the same as in MySQL and how many will need to be implemented from scratch.

A strong application cover letter will give rough answers to all those questions (obviously they don't have to be fully correct/definitive -- I'm sure we'll find many nuances and bugs once we start integrating the collations into Vitess).

Thanks everyone and good luck with your applications!

[king-11]

Can we still submit the cover letter? I thought the deadline was 22nd and questions that needed to be answered were the mentioned ones only 😅

[vmg]

@king-11: ah, fair enough, the deadline went by while I was on holidays. The point still stands about what's important for the project, but don't worry if you've already submitted. :)

[king-11]

@vmg I have read about what, how they work and how they are implemented. So I was moving to the go collate package how can I compare their implementations vs MySQL for each collation can you help me out on that.

[vmg]

So, there are many ways to accomplish this. The first thing I'd try is implementing a WEIGHT_STRING function in Go on top of the collate package. It should take a string and a collation, and return a binary string like MySQL's implementation does. With this function in place, it should be trivial to start writing unit tests that compare the output of our collation package with the results that MySQL returns. Let me know if you run into any roadblocks attempting this.