index problem of substring function and charset support

[zyguan]

It should be if pos > int64(len(str)) || pos < int64(0) { here, otherwise substring(s, 1) always return an empty string. Beside, I see that the code doesn't handle the charset of the string for now, which leads to following result:

mysql> select substring("你好世界", 1, 3);
+---------------------------------+
| substring("你好世界", 1, 3)     |
+---------------------------------+
| 你                              |
+---------------------------------+
1 row in set (0.00 sec)

I'm willing to help, however, I'm not sure how to get the charset info from the context of such a builtin function call.

---

1. What version of Go are you using (go version)?
   go1.6
2. What operating system and processor architecture are you using (go env)?
   GOHOSTARCH="amd64"
   GOHOSTOS="linux"
3. What did you do?
   exec select substring("hello", 1, 5);
4. What did you expect to see?

+--------------------------+
| substring("hello", 1, 5) |
+--------------------------+
| hello                    |
+--------------------------+

1. What did you see instead?

+--------------------------+
| substring("hello", 1, 5) |
+--------------------------+
|                          |
+--------------------------+

[shenli]

@zyguan Thanks for your report!
String literal will be wrapped into a ast.ValueExpr. And you can get charset/collation info from ValueExpr.
https://github.com/pingcap/tidb/blob/master/parser/parser.y#L2008

[zyguan]

Thanks for your reply, @shenli .

I see, buitin functions only accept an array of types.Datum as the input.
https://github.com/pingcap/tidb/blob/master/evaluator/evaluator.go#L647

However, the charset/collation info is stored in the exprNode.Type field. Thus, there is (maybe) no way for us to get the info within these builtin functions currently (I thought we can get it from the Context). I think it might be better to pass datums as well as their types to a FuncCall.

[shenli]

@zyguan Yes, we have a plan to add collation info into datum.

[morgo]

This issue has been resolved in TiDB. Confirming with a recent version:

tidb> select substring("你好世界", 1, 3);
+---------------------------------+
| substring("你好世界", 1, 3)     |
+---------------------------------+
| 你好世                          |
+---------------------------------+
1 row in set (0.00 sec)

tidb> select substring("hello", 1, 5);
+--------------------------+
| substring("hello", 1, 5) |
+--------------------------+
| hello                    |
+--------------------------+
1 row in set (0.00 sec)

tidb> select tidb_version()\G
*************************** 1. row ***************************
tidb_version(): Release Version: v2.1.0-rc.3-219-g1e0876fe8-dirty
Git Commit Hash: 1e0876fe810a832721aac52275dd2b7792fd2892
Git Branch: flush
UTC Build Time: 2018-11-24 01:12:47
GoVersion: go version go1.11 linux/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
Check Table Before Drop: false
1 row in set (0.00 sec)

I am going to close this issue for now, but please feel free to re-open if you have any followup questions. Thanks!