Track complete parsing support in Vitess

[GuptaManan100]

Description

Vitess has its own in-built SQL-parser which it uses to understand the query and represent as structs for further processing. As of now, a lot of MySQL structs are not parsed and result in syntax errors. This issue is to track the progress on adding parsing for such constructs

• Partitions
  • Compatibility: support ALTER TABLE... PARTITION BY #9682
  • sqlparser: support PARTITION BY LIST #9696
  • Partitiondefinition, sub partitions
• Prepare
• Collations
• Charsets
• Functions (MySQL 5.7)
• Functions (MySQL 8.0)
• String functions (https://dev.mysql.com/doc/refman/8.0/en/string-functions.html)
• Date and time literals (https://dev.mysql.com/doc/refman/8.0/en/date-and-time-literals.html)
• Support new 8.0 functions #4099
• Add support for extended TRIM function grammar #4347
• Multi INDEX HINTS not supported. #4038
• https://dev.mysql.com/doc/refman/8.0/en/spatial-analysis-functions.html
• QueryServing: support LATERAL keyword #9716
• vttablet: query regression for json extract operator -> #9951
• Feature Request: Updating can get the updated value #10760

Work Done:

• Added UNION testcases and auxilary code for running tests #8797
• Changed parsing for unions #8821
• Add parsing support for column list with subquery alias #8884
• Add parsing for common table expressions using WITH clause #8918
• Added tokenizer support for NCHAR_STRING #9006
• Added brackets in default value if column is of type TEXT, BLOB, GEOMETRY or JSON #9011
• Support extract function #9029
• Changed Collations and Expression grammar #9075
• Initial implementation of partitions (Create statement) #8691

[ritwizsinha]

Hello @GuptaManan100 I have been trying out and understanding various projects in CNCF ecosystem and I am particularly interested in those related to databases and networking so I would like to get involved with this project and work on this issue. I would like to contribute to this project and increase my knowledge. I would be familiarising myself with terminology and get to know better about Vitess in the coming days. Any help appreciated.

[GuptaManan100]

@ritwizsinha good to hear! This project is part of the LFX program. More information at https://github.com/cncf/mentoring/tree/main/lfx-mentorship. We would love to see you apply in it. All the best!

[ritwizsinha]

Thanks @GuptaManan100 I would like to apply to it, what all do I need to provide to improve my applications, do I need to have previous knowledge of Vitess, or previous contributions in this repository to get selected? Do I have to write out an implementation proposal for this feature(I am asking because I heard that every org did it differently)? What all would you like to see in the application ?

[GuptaManan100]

@ritwizsinha No, we do not expect any previous knowledge of Vitess from the applicants. There is no explicit requirement for a implementation proposal either, but it would be great if you would be able to do a gap analysis of what all Vitess parser is missing from MySQL. I have written some of the constructs that I know that Vitess does not parse correctly

[derekperkins]

There are still a significant number of 8.0 functions without support: #4099

[ritwizsinha]

Okay thanks @GuptaManan10 and @derekperkins I will look into it

[aribalam]

Hi @GuptaManan100 , a student from IIT Kharagpur here. I would like apply for this project under the LFX mentorship program. I have read the previous comments and the requirements you mentioned. Looking forward to it. :)

[ritwizsinha]

@GuptaManan100 I have read bit of the documentation of Vitess and set it up locally using Docker, now to identify which constructs are missing, I am thinking of trying to execute the methods you mentioned above, for example for partitioning. then for collation etc. Is their any document which specifies what all the Vitess parser recognizes and parses or we have to check it all manually?

[GuptaManan100]

@ritwizsinha Have a look at https://github.com/vitessio/vitess/blob/main/go/vt/sqlparser/sql.y. This file is our yacc parser configuration. So this is the authoritative source on what we parse and what we don't. Within the same package we have a parse_test.go file which has the parsing tests.

[ritwizsinha]

@GuptaManan100 I wrote a quick script to match all the functions in here https://dev.mysql.com/doc/refman/8.0/en/built-in-function-reference.html which aren't present in the https://github.com/vitessio/vitess/blob/main/go/vt/sqlparser/sql.y. using grep and it turns out that 340/482 of those functions were not present in the yacc parser config. I may be wrong because I am not an expert in bash scripts but I searched some and they didn't exist

>>
>=
<>
<<
<=
<=>
->
->>
ABS
ACOS
ADDDATE
ADDTIME
AES_DECRYPT
AES_ENCRYPT
&&
ANY_VALUE
ASCII
ASIN
ATAN
ATAN2
ATAN
BENCHMARK
BIN_TO_UUID
BIT_AND
BIT_COUNT
BIT_LENGTH
BIT_OR
BIT_XOR
CAN_ACCESS_COLUMN
CAN_ACCESS_DATABASE
CAN_ACCESS_TABLE
CAN_ACCESS_USER
CAN_ACCESS_VIEW
CEIL
CEILING
CHAR_LENGTH
CHARACTER_LENGTH
COERCIBILITY
CONCAT_WS
CONNECTION_ID
CONVERT_TZ
COT
COUNT
COUNTDISTINCT
CRC32
CURDATE
CURRENT_ROLE
CURTIME
DATE_ADD
DATE_FORMAT
DATE_SUB
DATEDIFF
DAY
DAYNAME
DAYOFMONTH
DAYOFWEEK
DAYOFYEAR
DEGREES
ELT
EXPORT_SET
ExtractValue
FIND_IN_SET
FLOOR
FORMAT_BYTES
FORMAT_PICO_TIME
FROM_BASE64
FROM_DAYS
FROM_UNIXTIME
GeomCollection
GeometryCollection
GET_DD_COLUMN_PRIVILEGES
GET_DD_CREATE_OPTIONS
GET_DD_INDEX_SUB_PART_LENGTH
GET_FORMAT
GET_LOCK
GREATEST
GTID_SUBSET
GTID_SUBTRACT
HOUR
ICU_VERSION
IFNULL
INET_ATON
INET_NTOA
INET6_ATON
INET6_NTOA
INSTR
INTERNAL_AUTO_INCREMENT
INTERNAL_AVG_ROW_LENGTH
INTERNAL_CHECK_TIME
INTERNAL_CHECKSUM
INTERNAL_DATA_FREE
INTERNAL_DATA_LENGTH
INTERNAL_DD_CHAR_LENGTH
INTERNAL_GET_COMMENT_OR_ERROR
INTERNAL_GET_ENABLED_ROLE_JSON
INTERNAL_GET_HOSTNAME
INTERNAL_GET_USERNAME
INTERNAL_GET_VIEW_WARNING_OR_ERROR
INTERNAL_INDEX_COLUMN_CARDINALITY
INTERNAL_INDEX_LENGTH
INTERNAL_IS_ENABLED_ROLE
INTERNAL_IS_MANDATORY_ROLE
INTERNAL_KEYS_DISABLED
INTERNAL_MAX_DATA_LENGTH
INTERNAL_TABLE_ROWS
INTERNAL_UPDATE_TIME
IS_FREE_LOCK
IS_IPV4
IS_IPV4_COMPAT
IS_IPV4_MAPPED
IS_IPV6
IS_USED_LOCK
IS_UUID
ISNULL
JSON_ARRAY
JSON_ARRAY_APPEND
JSON_ARRAY_INSERT
JSON_ARRAYAGG
JSON_CONTAINS
JSON_CONTAINS_PATH
JSON_DEPTH
JSON_INSERT
JSON_KEYS
JSON_LENGTH
JSON_MERGE
JSON_MERGE_PATCH
JSON_MERGE_PRESERVE
JSON_OBJECT
JSON_OBJECTAGG
JSON_OVERLAPS
JSON_PRETTY
JSON_QUOTE
JSON_REMOVE
JSON_REPLACE
JSON_SCHEMA_VALID
JSON_SCHEMA_VALIDATION_REPORT
JSON_SEARCH
JSON_SET
JSON_STORAGE_FREE
JSON_STORAGE_SIZE
JSON_TYPE
JSON_VALID
JSON_VALUE
LAST_DAY
LCASE
LEAST
LineString
LOAD_FILE
LOCATE
LOG10
LOG2
LPAD
LTRIM
MAKE_SET
MAKEDATE
MAKETIME
MASTER_POS_WAIT
MBRContains
MBRCoveredBy
MBRCovers
MBRDisjoint
MBREquals
MBRIntersects
MBROverlaps
MBRTouches
MBRWithin
MD5
MICROSECOND
MID
MINUTE
MONTH
MONTHNAME
MultiLineString
MultiPoint
MultiPolygon
NAME_CONST
NULLIF
OCT
OCTET_LENGTH
||
PERIOD_ADD
PERIOD_DIFF
Point
Polygon
POSITION
POW
POWER
PS_CURRENT_THREAD_ID
PS_THREAD_ID
QUARTER
RADIANS
RANDOM_BYTES
REGEXP_INSTR
REGEXP_LIKE
REGEXP_REPLACE
REGEXP_SUBSTR
RELEASE_ALL_LOCKS
RELEASE_LOCK
REVERSE
RLIKE
ROLES_GRAPHML
ROUND
ROW_COUNT
RPAD
RTRIM
SEC_TO_TIME
SESSION_USER
SHA1
SHA2
SLEEP
SOUNDEX
SOUNDS
SOURCE_POS_WAIT
SQRT
ST_Area
ST_AsBinary
ST_AsWKB
ST_AsGeoJSON
ST_AsText
ST_AsWKT
ST_Buffer
ST_Buffer_Strategy
ST_Centroid
ST_Collect
ST_Contains
ST_ConvexHull
ST_Crosses
ST_Difference
ST_Dimension
ST_Disjoint
ST_Distance
ST_Distance_Sphere
ST_EndPoint
ST_Envelope
ST_Equals
ST_ExteriorRing
ST_FrechetDistance
ST_GeoHash
ST_GeomCollFromText
ST_GeometryCollectionFromText
ST_GeomCollFromTxt
ST_GeomCollFromWKB
ST_GeometryCollectionFromWKB
ST_GeometryN
ST_GeometryType
ST_GeomFromGeoJSON
ST_GeomFromText
ST_GeometryFromText
ST_GeomFromWKB
ST_GeometryFromWKB
ST_HausdorffDistance
ST_InteriorRingN
ST_Intersection
ST_Intersects
ST_IsClosed
ST_IsEmpty
ST_IsSimple
ST_IsValid
ST_LatFromGeoHash
ST_Latitude
ST_Length
ST_LineFromText
ST_LineStringFromText
ST_LineFromWKB
ST_LineStringFromWKB
ST_LineInterpolatePoint
ST_LineInterpolatePoints
ST_LongFromGeoHash
ST_Longitude
ST_MakeEnvelope
ST_MLineFromText
ST_MultiLineStringFromText
ST_MLineFromWKB
ST_MultiLineStringFromWKB
ST_MPointFromText
ST_MultiPointFromText
ST_MPointFromWKB
ST_MultiPointFromWKB
ST_MPolyFromText
ST_MultiPolygonFromText
ST_MPolyFromWKB
ST_MultiPolygonFromWKB
ST_NumGeometries
ST_NumInteriorRing
ST_NumInteriorRings
ST_NumPoints
ST_Overlaps
ST_PointAtDistance
ST_PointFromGeoHash
ST_PointFromText
ST_PointFromWKB
ST_PointN
ST_PolyFromText
ST_PolygonFromText
ST_PolyFromWKB
ST_PolygonFromWKB
ST_Simplify
ST_SRID
ST_StartPoint
ST_SwapXY
ST_SymDifference
ST_Touches
ST_Transform
ST_Union
ST_Validate
ST_Within
ST_X
ST_Y
STATEMENT_DIGEST
STATEMENT_DIGEST_TEXT
STDDEV
STDDEV_POP
STDDEV_SAMP
STR_TO_DATE
STRCMP
SUBDATE
SUBSTRING_INDEX
SUBTIME
SYSDATE
SYSTEM_USER
TAN
TIME_FORMAT
TIME_TO_SEC
TIMEDIFF
TO_BASE64
TO_DAYS
TO_SECONDS
TRIM
UCASE
UNCOMPRESS
UNCOMPRESSED_LENGTH
UNHEX
UNIX_TIMESTAMP
UpdateXML
UPPER
UUID_SHORT
UUID_TO_BIN
VALIDATE_PASSWORD_STRENGTH
VAR_POP
VAR_SAMP
VARIANCE
VERSION
WAIT_FOR_EXECUTED_GTID_SET
WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS
WAIT_FOR_EXECUTED_GTID_SET
WEEK
WEEKDAY
WEEKOFYEAR
WEIGHT_STRING
YEARWEEK

EDIT: There might be even more because I saw a section in the yacc parser config which said
MySQL reserved words that are unused by this grammar will map to this token.
As these also aren't used by the grammar

[GuptaManan100]

@ritwizsinha Yes that is a great first step in gap analysis!

[ritwizsinha]

@GuptaManan100 are you available on slack we can discuss there

[GuptaManan100]

Yes I am available on VItess slack

[aribalam]

@GuptaManan100 Looking at the sql.y file, there are already support for parsing the COLLATE and CHARACTER SET constructs. The same is also present in the parse_test.go. Could you please confirm?

[GuptaManan100]

@aribalam please take a look at this thread in Vitess slack https://vitess.slack.com/archives/C0PQY0PTK/p1628839955078000

[Thirumalai-Shaktivel]

I would recommend keeping track of all the Built-In Functions and Operators in MySQL 5.7 which has already been implemented.

Marked one is already parsed by the sqlparser, Rest has to be implemented

• &
  Bitwise AND
• >
  Greater than operator
• >>
  Right shift
• >=
  Greater than or equal operator
• <
  Less than operator
• <>, !=
  Not equal operator
• <<
  Left shift
• <=
  Less than or equal operator
• <=>
  NULL-safe equal to operator
• %, MOD
  Modulo operator
• *
  Multiplication operator
• +
  Addition operator
• -
  Minus operator
• -
  Change the sign of the argument
• ->
  Return value from JSON column after evaluating path; equivalent to JSON_EXTRACT().
• ->>
  Return value from JSON column after evaluating path and unquoting the result; equivalent to JSON_UNQUOTE(JSON_EXTRACT()).
• /
  Division operator
• :=
  Assign a value
• =
  Assign a value (as part of a SET statement, or as part of the SET clause in an UPDATE statement)
• =
  Equal operator
• ^
  Bitwise XOR
• ABS()
  Return the absolute value
• ACOS()
  Return the arc cosine
• ADDDATE()
  Add time values (intervals) to a date value
• ADDTIME()
  Add time
• AES_DECRYPT()
  Decrypt using AES
• AES_ENCRYPT()
  Encrypt using AES
• AND, &&
  Logical AND
• ANY_VALUE()
  Suppress ONLY_FULL_GROUP_BY value rejection
• Area()
  Return Polygon or MultiPolygon area
• AsBinary(), AsWKB()
  Convert from internal geometry format to WKB
• ASCII()
  Return numeric value of left-most character
• ASIN()
  Return the arc sine
• AsText(), AsWKT()
  Convert from internal geometry format to WKT
• ATAN()
  Return the arc tangent
• ATAN2(), ATAN()
  Return the arc tangent of the two arguments
• AVG()
  Return the average value of the argument
• BENCHMARK()
  Repeatedly execute an expression
• BETWEEN ... AND ...
  Whether a value is within a range of values
• BIN()
  Return a string containing binary representation of a number
• BINARY
  Cast a string to a binary string
• BIT_AND()
  Return bitwise AND
• BIT_COUNT()
  Return the number of bits that are set
• BIT_LENGTH()
  Return length of argument in bits
• BIT_OR()
  Return bitwise OR
• BIT_XOR()
  Return bitwise XOR
• Buffer()
  Return geometry of points within given distance from geometry
• CASE
  Case operator
• CAST()
  Cast a value as a certain type
• CEIL()
  Return the smallest integer value not less than the argument
• CEILING()
  Return the smallest integer value not less than the argument
• Centroid()
  Return centroid as a point
• CHAR()
  Return the character for each integer passed
• CHAR_LENGTH()
  Return number of characters in argument
• CHARACTER_LENGTH()
  Synonym for CHAR_LENGTH()
• CHARSET()
  Return the character set of the argument
• COALESCE()
  Return the first non-NULL argument
• COERCIBILITY()
  Return the collation coercibility value of the string argument
• COLLATION()
  Return the collation of the string argument
• COMPRESS()
  Return result as a binary string
• CONCAT()
  Return concatenated string
• CONCAT_WS()
  Return concatenate with separator
• CONNECTION_ID()
  Return the connection ID (thread ID) for the connection
• Contains()
  Whether MBR of one geometry contains MBR of another
• CONV()
  Convert numbers between different number bases
• CONVERT()
  Cast a value as a certain type
• CONVERT_TZ()
  Convert from one time zone to another
• ConvexHull()
  Return convex hull of geometry
• COS()
  Return the cosine
• COT()
  Return the cotangent
• COUNT()
  Return a count of the number of rows returned
• COUNT(DISTINCT)
  Return the count of a number of different values
• CRC32()
  Compute a cyclic redundancy check value
• Crosses()
  Whether one geometry crosses another
• CURDATE()
  Return the current date
• CURRENT_DATE(), CURRENT_DATE
  Synonyms for CURDATE()
• CURRENT_TIME(), CURRENT_TIME
  Synonyms for CURTIME()
• CURRENT_TIMESTAMP(), CURRENT_TIMESTAMP
  Synonyms for NOW()
• CURRENT_USER(), CURRENT_USER
  The authenticated user name and host name
• CURTIME()
  Return the current time
• DATABASE()
  Return the default (current) database name
• DATE()
  Extract the date part of a date or datetime expression
• DATE_ADD()
  Add time values (intervals) to a date value
• DATE_FORMAT()
  Format date as specified
• DATE_SUB()
  Subtract a time value (interval) from a date
• DATEDIFF()
  Subtract two dates
• DAY()
  Synonym for DAYOFMONTH()
• DAYNAME()
  Return the name of the weekday
• DAYOFMONTH()
  Return the day of the month (0-31)
• DAYOFWEEK()
  Return the weekday index of the argument
• DAYOFYEAR()
  Return the day of the year (1-366)
• DECODE()
  Decode a string encrypted using ENCODE()
• DEFAULT()
  Return the default value for a table column
• DEGREES()
  Convert radians to degrees
• DES_DECRYPT()
  Decrypt a string
• DES_ENCRYPT()
  Encrypt a string
• Dimension()
  Dimension of geometry
• Disjoint()
  Whether MBRs of two geometries are disjoint
• Distance()
  The distance of one geometry from another
• DIV
  Integer division
• ELT()
  Return string at index number
• ENCODE()
  Encode a string
• ENCRYPT()
  Encrypt a string
• EndPoint()
  End Point of LineString
• Envelope()
  Return MBR of geometry
• Equals()
  Whether MBRs of two geometries are equal
• EXP()
  Raise to the power of
• EXPORT_SET()
  Return a string such that for every bit set in the value bits, you get an on string and for every unset bit, you get an off string
• ExteriorRing()
  Return exterior ring of Polygon
• EXTRACT()
  Extract part of a date
• ExtractValue()
  Extract a value from an XML string using XPath notation
• FIELD()
  Index (position) of first argument in subsequent arguments
• FIND_IN_SET()
  Index (position) of first argument within second argument
• FLOOR()
  Return the largest integer value not greater than the argument
• FORMAT()
  Return a number formatted to specified number of decimal places
• FOUND_ROWS()
  For a SELECT with a LIMIT clause, the number of rows that would be returned were there no LIMIT clause
• FROM_BASE64()
  Decode base64 encoded string and return result
• FROM_DAYS()
  Convert a day number to a date
• FROM_UNIXTIME()
  Format Unix timestamp as a date
• GeomCollFromText(), GeometryCollectionFromText()
  Return geometry collection from WKT
• GeomCollFromWKB(), GeometryCollectionFromWKB()
  Return geometry collection from WKB
• GeometryCollection()
  Construct geometry collection from geometries
• GeometryN()
  Return N-th geometry from geometry collection
• GeometryType()
  Return name of geometry type
• GeomFromText(), GeometryFromText()
  Return geometry from WKT
• GeomFromWKB(), GeometryFromWKB()
  Return geometry from WKB
• GET_FORMAT()
  Return a date format string
• GET_LOCK()
  Get a named lock
• GLength()
  Return length of LineString
• GREATEST()
  Return the largest argument
• GROUP_CONCAT()
  Return a concatenated string
• GTID_SUBSET()
  Return true if all GTIDs in subset are also in set; otherwise false.
• GTID_SUBTRACT()
  Return all GTIDs in set that are not in subset.
• HEX()
  Hexadecimal representation of decimal or string value
• HOUR()
  Extract the hour
• IF()
  If/else construct
• IFNULL()
  Null if/else construct
• IN()
  Whether a value is within a set of values
• INET_ATON()
  Return the numeric value of an IP address
• INET_NTOA()
  Return the IP address from a numeric value
• INET6_ATON()
  Return the numeric value of an IPv6 address
• INET6_NTOA()
  Return the IPv6 address from a numeric value
• INSERT()
  Insert substring at specified position up to specified number of characters
• INSTR()
  Return the index of the first occurrence of substring
• InteriorRingN()
  Return N-th interior ring of Polygon
• Intersects()
  Whether MBRs of two geometries intersect
• INTERVAL()
  Return the index of the argument that is less than the first argument
• IS
  Test a value against a boolean
• IS_FREE_LOCK()
  Whether the named lock is free
• IS_IPV4()
  Whether argument is an IPv4 address
• IS_IPV4_COMPAT()
  Whether argument is an IPv4-compatible address
• IS_IPV4_MAPPED()
  Whether argument is an IPv4-mapped address
• IS_IPV6()
  Whether argument is an IPv6 address
• IS NOT
  Test a value against a boolean
• IS NOT NULL
  NOT NULL value test
• IS NULL
  NULL value test
• IS_USED_LOCK()
  Whether the named lock is in use; return connection identifier if true
• IsClosed()
  Whether a geometry is closed and simple
• IsEmpty()
  Whether a geometry is empty
• ISNULL()
  Test whether the argument is NULL
• IsSimple()
  Whether a geometry is simple
• JSON_APPEND()
  Append data to JSON document
• JSON_ARRAY()
  Create JSON array
• JSON_ARRAY_APPEND()
  Append data to JSON document
• JSON_ARRAY_INSERT()
  Insert into JSON array
• JSON_ARRAYAGG()
  Return result set as a single JSON array`
  
• JSON_CONTAINS()
  Whether JSON document contains specific object at path
• JSON_CONTAINS_PATH()
  Whether JSON document contains any data at path
• JSON_DEPTH()
  Maximum depth of JSON document
• JSON_EXTRACT()
  Return data from JSON document
• JSON_INSERT()
  Insert data into JSON document
• JSON_KEYS()
  Array of keys from JSON document
• JSON_LENGTH()
  Number of elements in JSON document
• JSON_MERGE()
  Merge JSON documents, preserving duplicate keys. Deprecated synonym for JSON_MERGE_PRESERVE()
• JSON_MERGE_PATCH()
  Merge JSON documents, replacing values of duplicate keys`
  
• JSON_MERGE_PRESERVE()
  Merge JSON documents, preserving duplicate keys`
  
• JSON_OBJECT()
  Create JSON object
• JSON_OBJECTAGG()
  Return result set as a single JSON object`
  
• JSON_PRETTY()
  Print a JSON document in human-readable format`
  
• JSON_QUOTE()
  Quote JSON document
• JSON_REMOVE()
  Remove data from JSON document
• JSON_REPLACE()
  Replace values in JSON document
• JSON_SEARCH()
  Path to value within JSON document
• JSON_SET()
  Insert data into JSON document
• JSON_STORAGE_SIZE()
  Space used for storage of binary representation of a JSON document`
  
• JSON_TYPE()
  Type of JSON value
• JSON_UNQUOTE()
  Unquote JSON value
• JSON_VALID()
  Whether JSON value is valid
• LAST_DAY
  Return the last day of the month for the argument
• LAST_INSERT_ID()
  Value of the AUTOINCREMENT column for the last INSERT
• LCASE()
  Synonym for LOWER()
• LEAST()
  Return the smallest argument
• LEFT()
  Return the leftmost number of characters as specified
• LENGTH()
  Return the length of a string in bytes
• LIKE
  Simple pattern matching
• LineFromText(), LineStringFromText()
  Construct LineString from WKT
• LineFromWKB(), LineStringFromWKB()
  Construct LineString from WKB
• LineString()
  Construct LineString from Point values
• LN()
  Return the natural logarithm of the argument
• LOAD_FILE()
  Load the named file
• LOCALTIME(), LOCALTIME
  Synonym for NOW()
• LOCALTIMESTAMP, LOCALTIMESTAMP()
  Synonym for NOW()
• LOCATE()
  Return the position of the first occurrence of substring
• LOG()
  Return the natural logarithm of the first argument
• LOG10()
  Return the base-10 logarithm of the argument
• LOG2()
  Return the base-2 logarithm of the argument
• LOWER()
  Return the argument in lowercase
• LPAD()
  Return the string argument, left-padded with the specified string
• LTRIM()
  Remove leading spaces
• MAKE_SET()
  Return a set of comma-separated strings that have the corresponding bit in bits set
• MAKEDATE()
  Create a date from the year and day of year
• MAKETIME()
  Create time from hour, minute, second
• MASTER_POS_WAIT()
  Block until the replica has read and applied all updates up to the specified position
• MATCH
  Perform full-text search
• MAX()
  Return the maximum value
• MBRContains()
  Whether MBR of one geometry contains MBR of another
• MBRCoveredBy()
  Whether one MBR is covered by another
• MBRCovers()
  Whether one MBR covers another
• MBRDisjoint()
  Whether MBRs of two geometries are disjoint
• MBREqual()
  Whether MBRs of two geometries are equal
• MBREquals()
  Whether MBRs of two geometries are equal
• MBRIntersects()
  Whether MBRs of two geometries intersect
• MBROverlaps()
  Whether MBRs of two geometries overlap
• MBRTouches()
  Whether MBRs of two geometries touch
• MBRWithin()
  Whether MBR of one geometry is within MBR of another
• MD5()
  Calculate MD5 checksum
• MICROSECOND()
  Return the microseconds from argument
• MID()
  Return a substring starting from the specified position
• MIN()
  Return the minimum value
• MINUTE()
  Return the minute from the argument
• MLineFromText(), MultiLineStringFromText()
  Construct MultiLineString from WKT
• MLineFromWKB(), MultiLineStringFromWKB()
  Construct MultiLineString from WKB
• MOD()
  Return the remainder
• MONTH()
  Return the month from the date passed
• MONTHNAME()
  Return the name of the month
• MPointFromText(), MultiPointFromText()
  Construct MultiPoint from WKT
• MPointFromWKB(), MultiPointFromWKB()
  Construct MultiPoint from WKB
• MPolyFromText(), MultiPolygonFromText()
  Construct MultiPolygon from WKT
• MPolyFromWKB(), MultiPolygonFromWKB()
  Construct MultiPolygon from WKB
• MultiLineString()
  Contruct MultiLineString from LineString values
• MultiPoint()
  Construct MultiPoint from Point values
• MultiPolygon()
  Construct MultiPolygon from Polygon values
• NAME_CONST()
  Cause the column to have the given name
• NOT, !
  Negates value
• NOT BETWEEN ... AND ...
  Whether a value is not within a range of values
• NOT IN()
  Whether a value is not within a set of values
• NOT LIKE
  Negation of simple pattern matching
• NOT REGEXP
  Negation of REGEXP
• NOW()
  Return the current date and time
• NULLIF()
  Return NULL if expr1 = expr2
• NumGeometries()
  Return number of geometries in geometry collection
• NumInteriorRings()
  Return number of interior rings in Polygon
• NumPoints()
  Return number of points in LineString
• OCT()
  Return a string containing octal representation of a number
• OCTET_LENGTH()
  Synonym for LENGTH()
• OR, ||
  Logical OR
• ORD()
  Return character code for leftmost character of the argument
• Overlaps()
  Whether MBRs of two geometries overlap
• PASSWORD()
  Calculate and return a password string
• PERIOD_ADD()
  Add a period to a year-month
• PERIOD_DIFF()
  Return the number of months between periods
• PI()
  Return the value of pi
• Point()
  Construct Point from coordinates
• PointFromText()
  Construct Point from WKT
• PointFromWKB()
  Construct Point from WKB
• PointN()
  Return N-th point from LineString
• PolyFromText(), PolygonFromText()
  Construct Polygon from WKT
• PolyFromWKB(), PolygonFromWKB()
  Construct Polygon from WKB
• Polygon()
  Construct Polygon from LineString arguments
• POSITION()
  Synonym for LOCATE()
• POW()
  Return the argument raised to the specified power
• POWER()
  Return the argument raised to the specified power
• PROCEDURE ANALYSE()
  Analyze the results of a query
• QUARTER()
  Return the quarter from a date argument
• QUOTE()
  Escape the argument for use in an SQL statement
• RADIANS()
  Return argument converted to radians
• RAND()
  Return a random floating-point value
• RANDOM_BYTES()
  Return a random byte vector
• REGEXP
  Whether string matches regular expression
• RELEASE_ALL_LOCKS()
  Release all current named locks
• RELEASE_LOCK()
  Release the named lock
• REPEAT()
  Repeat a string the specified number of times
• REPLACE()
  Replace occurrences of a specified string
• REVERSE()
  Reverse the characters in a string
• RIGHT()
  Return the specified rightmost number of characters
• RLIKE
  Whether string matches regular expression
• ROUND()
  Round the argument
• ROW_COUNT()
  The number of rows updated
• RPAD()
  Append string the specified number of times
• RTRIM()
  Remove trailing spaces
• SCHEMA()
  Synonym for DATABASE()
• SEC_TO_TIME()
  Converts seconds to 'hh:mm:ss' format
• SECOND()
  Return the second (0-59)
• SESSION_USER()
  Synonym for USER()
• SHA1(), SHA()
  Calculate an SHA-1 160-bit checksum
• SHA2()
  Calculate an SHA-2 checksum
• SIGN()
  Return the sign of the argument
• SIN()
  Return the sine of the argument
• SLEEP()
  Sleep for a number of seconds
• SOUNDEX()
  Return a soundex string
• SOUNDS LIKE
  Compare sounds
• SPACE()
  Return a string of the specified number of spaces
• SQRT()
  Return the square root of the argument
• SRID()
  Return spatial reference system ID for geometry
• ST_Area()
  Return Polygon or MultiPolygon area
• ST_AsBinary(), ST_AsWKB()
  Convert from internal geometry format to WKB
• ST_AsGeoJSON()
  Generate GeoJSON object from geometry
• ST_AsText(), ST_AsWKT()
  Convert from internal geometry format to WKT
• ST_Buffer()
  Return geometry of points within given distance from geometry
• ST_Buffer_Strategy()
  Produce strategy option for ST_Buffer()
• ST_Centroid()
  Return centroid as a point
• ST_Contains()
  Whether one geometry contains another
• ST_ConvexHull()
  Return convex hull of geometry
• ST_Crosses()
  Whether one geometry crosses another
• ST_Difference()
  Return point set difference of two geometries
• ST_Dimension()
  Dimension of geometry
• ST_Disjoint()
  Whether one geometry is disjoint from another
• ST_Distance()
  The distance of one geometry from another
• ST_Distance_Sphere()
  Minimum distance on earth between two geometries
• ST_EndPoint()
  End Point of LineString
• ST_Envelope()
  Return MBR of geometry
• ST_Equals()
  Whether one geometry is equal to another
• ST_ExteriorRing()
  Return exterior ring of Polygon
• ST_GeoHash()
  Produce a geohash value
• ST_GeomCollFromText(), ST_GeometryCollectionFromText(), ST_GeomCollFromTxt()
  Return geometry collection from WKT
• ST_GeomCollFromWKB(), ST_GeometryCollectionFromWKB()
  Return geometry collection from WKB
• ST_GeometryN()
  Return N-th geometry from geometry collection
• ST_GeometryType()
  Return name of geometry type
• ST_GeomFromGeoJSON()
  Generate geometry from GeoJSON object
• ST_GeomFromText(), ST_GeometryFromText()
  Return geometry from WKT
• ST_GeomFromWKB(), ST_GeometryFromWKB()
  Return geometry from WKB
• ST_InteriorRingN()
  Return N-th interior ring of Polygon
• ST_Intersection()
  Return point set intersection of two geometries
• ST_Intersects()
  Whether one geometry intersects another
• ST_IsClosed()
  Whether a geometry is closed and simple
• ST_IsEmpty()
  Whether a geometry is empty
• ST_IsSimple()
  Whether a geometry is simple
• ST_IsValid()
  Whether a geometry is valid
• ST_LatFromGeoHash()
  Return latitude from geohash value
• ST_Length()
  Return length of LineString
• ST_LineFromText(), ST_LineStringFromText()
  Construct LineString from WKT
• ST_LineFromWKB(), ST_LineStringFromWKB()
  Construct LineString from WKB
• ST_LongFromGeoHash()
  Return longitude from geohash value
• ST_MakeEnvelope()
  Rectangle around two points
• ST_MLineFromText(), ST_MultiLineStringFromText()
  Construct MultiLineString from WKT
• ST_MLineFromWKB(), ST_MultiLineStringFromWKB()
  Construct MultiLineString from WKB
• ST_MPointFromText(), ST_MultiPointFromText()
  Construct MultiPoint from WKT
• ST_MPointFromWKB(), ST_MultiPointFromWKB()
  Construct MultiPoint from WKB
• ST_MPolyFromText(), ST_MultiPolygonFromText()
  Construct MultiPolygon from WKT
• ST_MPolyFromWKB(), ST_MultiPolygonFromWKB()
  Construct MultiPolygon from WKB
• ST_NumGeometries()
  Return number of geometries in geometry collection
• ST_NumInteriorRing(), ST_NumInteriorRings()
  Return number of interior rings in Polygon
• ST_NumPoints()
  Return number of points in LineString
• ST_Overlaps()
  Whether one geometry overlaps another
• ST_PointFromGeoHash()
  Convert geohash value to POINT value
• ST_PointFromText()
  Construct Point from WKT
• ST_PointFromWKB()
  Construct Point from WKB
• ST_PointN()
  Return N-th point from LineString
• ST_PolyFromText(), ST_PolygonFromText()
  Construct Polygon from WKT
• ST_PolyFromWKB(), ST_PolygonFromWKB()
  Construct Polygon from WKB
• ST_Simplify()
  Return simplified geometry
• ST_SRID()
  Return spatial reference system ID for geometry
• ST_StartPoint()
  Start Point of LineString
• ST_SymDifference()
  Return point set symmetric difference of two geometries
• ST_Touches()
  Whether one geometry touches another
• ST_Union()
  Return point set union of two geometries
• ST_Validate()
  Return validated geometry
• ST_Within()
  Whether one geometry is within another
• ST_X()
  Return X coordinate of Point
• ST_Y()
  Return Y coordinate of Point
• StartPoint()
  Start Point of LineString
• STD()
  Return the population standard deviation
• STDDEV()
  Return the population standard deviation
• STDDEV_POP()
  Return the population standard deviation
• STDDEV_SAMP()
  Return the sample standard deviation
• STR_TO_DATE()
  Convert a string to a date
• STRCMP()
  Compare two strings
• SUBDATE()
  Synonym for DATE_SUB() when invoked with three arguments
• SUBSTR()
  Return the substring as specified
• SUBSTRING()
  Return the substring as specified
• SUBSTRING_INDEX()
  Return a substring from a string before the specified number of occurrences of the delimiter
• SUBTIME()
  Subtract times
• SUM()
  Return the sum
• SYSDATE()
  Return the time at which the function executes
• SYSTEM_USER()
  Synonym for USER()
• TAN()
  Return the tangent of the argument
• TIME()
  Extract the time portion of the expression passed
• TIME_FORMAT()
  Format as time
• TIME_TO_SEC()
  Return the argument converted to seconds
• TIMEDIFF()
  Subtract time
• TIMESTAMP()
  With a single argument, this function returns the date or datetime expression; with two arguments, the sum of the arguments
• TIMESTAMPADD()
  Add an interval to a datetime expression
• TIMESTAMPDIFF()
  Subtract an interval from a datetime expression
• TO_BASE64()
  Return the argument converted to a base-64 string
• TO_DAYS()
  Return the date argument converted to days
• TO_SECONDS()
  Return the date or datetime argument converted to seconds since Year 0
• Touches()
  Whether one geometry touches another
• TRIM()
  Remove leading and trailing spaces
• TRUNCATE()
  Truncate to specified number of decimal places
• UCASE()
  Synonym for UPPER()
• UNCOMPRESS()
  Uncompress a string compressed
• UNCOMPRESSED_LENGTH()
  Return the length of a string before compression
• UNHEX()
  Return a string containing hex representation of a number
• UNIX_TIMESTAMP()
  Return a Unix timestamp
• UpdateXML()
  Return replaced XML fragment
• UPPER()
  Convert to uppercase
• USER()
  The user name and host name provided by the client
• UTC_DATE()
  Return the current UTC date
• UTC_TIME()
  Return the current UTC time
• UTC_TIMESTAMP()
  Return the current UTC date and time
• UUID()
  Return a Universal Unique Identifier (UUID)
• UUID_SHORT()
  Return an integer-valued universal identifier
• VALIDATE_PASSWORD_STRENGTH()
  Determine strength of password
• VALUES()
  Define the values to be used during an INSERT
• VAR_POP()
  Return the population standard variance
• VAR_SAMP()
  Return the sample variance
• VARIANCE()
  Return the population standard variance
• VERSION()
  Return a string that indicates the MySQL server version
• WAIT_FOR_EXECUTED_GTID_SET()
  Wait until the given GTIDs have executed on the replica.
• WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS()
  Use WAIT_FOR_EXECUTED_GTID_SET().
• WEEK()
  Return the week number
• WEEKDAY()
  Return the weekday index
• WEEKOFYEAR()
  Return the calendar week of the date (1-53)
• WEIGHT_STRING()
  Return the weight string for a string
• Within()
  Whether MBR of one geometry is within MBR of another
• X()
  Return X coordinate of Point
• XOR
  Logical XOR
• Y()
  Return Y coordinate of Point
• YEAR()
  Return the year
• YEARWEEK()
  Return the year and week
• |
  Bitwise OR
• ~
  Bitwise inversion

[GuptaManan100]

That is good work documenting the functions that we do not handle explicitly in MySQL @Thirumalai-Shaktivel.

[tharun208]

@GuptaManan100 I like to work on this as of the LFX program. I am interested in databases and I would be familiarising myself with Vitess and getting to know better about Vitess. Any help is appreciated.

[K-Kumar-01]

@GuptaManan100 I would like to work apply for this project under LFX. I am not familiar with Vitess as of now so currently I am going through the codebase and familiarizing myself with the concepts.
I do have some work flow which I have thought of to follow and I would like to discuss it with you, if possible.

[GuptaManan100]

All discussions about the project will happen on Slack. Here is the link to the general channel - https://vitess.slack.com/archives/C0PQY0PTK

[GuptaManan100]

Added #9682 to the description for tracking progress

[Weijun-H]

Hi @GuptaManan100 , I am really interested in this project under LFX. Could you give me some suggestions how to start it? Should I figure out which functions is still missing?

[GuptaManan100]

@Weijun-H, the issue has the list of things that the parser is missing. Anything that is unticked, can be worked on. You can look at any of the linked PRs to see where the tests reside and how to make parser changes. You can also refer to https://vitess.io/docs/15.0/contributing/contributing-to-ast-parser/ and https://vitess.io/docs/15.0/contributing/sample-first-issue/ for guidance.

[monalisha31]

Hi @GuptaManan100, I would like apply for this project under the LFX mentorship program. This project seems interesting. Looking forward to contributing to it.

[skant7]

Hi @GuptaManan100 ,I'm interested in working on this as part of the LFX Mentorship program. Can I start working on the String functions just to get a better understanding of the project or should I focus on the spatial functions to start with ?

[GuptaManan100]

@skant7 Sure, go right ahead, pick up whichever one you feel most comfortable with.

[SAEb-ai]

Hi @GuptaManan100 !! I am interested to contribute to this project under the LFX mentorship program and hence I applied to it now. Looking forward to contribute. Thanks!!

[GuptaManan100]

Everyone interested in the LFX project, please join the Vitess slack, #lfx-winter-2022 channel

[ktwillcode]

hi @GuptaManan100
Kartikeya this side, I applied for the mentorship and submitted a Cover letter and Resume. Is there anything more to complete the process?

[GuptaManan100]

@ktwillcode There is nothing else required to complete the process. You can try your hand at implementing one of the functions that Vitess doesn’t already have parsing for. It will give you an idea on how the project work will be and it will give us confidence in your ability to do the project.

[ktwillcode]

@ktwillcode There is nothing else required to complete the process. You can try your hand at implementing one of the functions that Vitess doesn’t already have parsing for. It will give you an idea on how the project work will be and it will give us confidence in your ability to do the project.

@GuptaManan100 Okay. Looking forward to work on it

[Ayman161803]

The following type of functions are yet to be implemented

• Spatial Aggregate functions
• Spatial Convenience Functions
• Spatial Operator Functions
• Functions That Test Spatial Relations Between Geometry Objects