Tracking issue for moving parser back to TiDB

[xhebox]

Intro

This issue tracks the work of rfc #28015, which will move parser back to TiDB for convenience. For more details, check the detailed design https://github.com/pingcap/tidb/blob/master/docs/design/2021-09-13-parser-as-submodule-of-tidb.md.

The work will not be assigned to members of the community, but the whole migration progress is public.

The migration can be roughly divided into two phases. The second phase will focus on migration for the community of parser.

Phase 1

• parser migration: create the sub module #28454
  • deadline 10-9
  • *: Delete parser/.github directory #28685
• Migrate tests for the new sub module.
  • deadline 10-16
  • parser migration: add unit tests #28699
• Announce the migration plan officially to stop sending new PRs/issues.
• Transfer old issues.
  • after tests is moved
• Clean up
  • after tests is moved
  • The parser cannot be compiled by executing the make parser command in the project root directory. #29322
• Synchronize new changes in pingcap/parser to the sub-module in pingcap/tidb. This task will be executed regularly until phase 2 started.
  • after tests is moved
• proposal about the boloated parser.go file
• update https://github.com/pingcap/tidb/blob/master/parser/docs/update-parser-for-tidb.md
• update tidb-dev-guide
  • update links in parser.md to TiDB repo tidb-dev-guide#172
  • update more parser links to TiDB repo tidb-dev-guide#174

Phase 2 (Not started)

• Announce the deprecation officially to the public.
• Create wrapper packages for pingcap/parser, which re-exports things from pingcap/tidb/parser.
• Remove the old CI.
• Check with the community that if we can archive pingcap/parser. Ideally, I hope this can be done after 3 or 4 months.

[tisonkun]

Announce the migration plan officially to stop sending new PRs/issues.

Is there a link for the announcement? I don't see an announcement on internals.tidb.io.

[tisonkun]

In order to drop the generated parser file, we may learn from how crdb did it and Golang supports go generate while I don't know whether it works well with go get.

cc @siddontang @zhangjinpeng1987 you make be interested in this topic.

[kennytm]

go get won't run go generate, by design. golang/go#15536.

If you check https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md you'll see that the go generated result is checked into git. Using go generate is no different from our make parser here.

[xhebox]

Announce the migration plan officially to stop sending new PRs/issues.

Is there a link for the announcement? I don't see an announcement on internals.tidb.io.

Nope. I sent the mail, and I will create a new reply in the forum. Or do you prefer a new thread?

[xhebox]

In order to drop the generated parser file, we may learn from how crdb did it and Golang supports go generate while I don't know whether it works well with go get.

cc @siddontang @zhangjinpeng1987 you make be interested in this topic.

I've discussed that, we may be able to release the .go file only in released tarballs. You can check the last question in RFC https://github.com/pingcap/tidb/blob/master/docs/design/2021-09-13-parser-as-submodule-of-tidb.md

[tisonkun]

Nope. I sent the mail, and I will create a new reply in the forum. Or do you prefer a new thread?

@xhebox a new thread is better, see also https://internals.tidb.io/t/topic/104 and other posts with "announcement" tag for example.

[cyliu0]

Some docs need to be updated

$ rg "pingcap/parser" -t markdown
docs/update-parser-for-tidb.md:    GO111MODULE=on go mod edit -replace github.com/pingcap/parser=github.com/your-repo/parser@your-branch
docs/update-parser-for-tidb.md:GO111MODULE=on go get -u github.com/pingcap/parser@master
README.md:[![Go Report Card](https://goreportcard.com/badge/github.com/pingcap/parser)](https://goreportcard.com/report/github.com/pingcap/parser)
README.md:[![CircleCI Status](https://circleci.com/gh/pingcap/parser.svg?style=shield)](https://circleci.com/gh/pingcap/parser)
README.md:[![GoDoc](https://godoc.org/github.com/pingcap/parser?status.svg)](https://godoc.org/github.com/pingcap/parser)
README.md:[![codecov](https://codecov.io/gh/pingcap/parser/branch/master/graph/badge.svg)](https://codecov.io/gh/pingcap/parser)
README.md:- Highly compatible with MySQL: it supports almost all features of MySQL. For the complete details, see [parser.y](https://github.com/pingcap/parser/blob/master/parser.y) and [hintparser.y](https://github.com/pingcap/parser/blob/master/hintparser.y).
README.md:- Extensible: adding a new syntax requires only a few lines of Yacc and Golang code changes. As an example, see [PR-680](https://github.com/pingcap/parser/pull/680/files).
README.md:Please read the [quickstart](https://github.com/pingcap/parser/blob/master/docs/quickstart.md).
README.md:- [GitHub Issue](https://github.com/pingcap/parser/issues)
README.md:Here is how to [update parser for TiDB](https://github.com/pingcap/parser/blob/master/docs/update-parser-for-tidb.md).
docs/quickstart.md:First of all, you need to use `go get` to fetch the dependencies through git hash. The git hashes are available in [release page](https://github.com/pingcap/parser/releases). Take `v4.0.2` as an example:
docs/quickstart.md:go get -v github.com/pingcap/parser@3a18f1e
docs/quickstart.md:1. Use the [`parser.New()`](https://pkg.go.dev/github.com/pingcap/parser?tab=doc#New) function to instantiate a parser, and
docs/quickstart.md:2. Invoke the method [`Parse(sql, charset, collation)`](https://pkg.go.dev/github.com/pingcap/parser?tab=doc#Parser.Parse) on the parser.
docs/quickstart.md:	"github.com/pingcap/parser"
docs/quickstart.md:	"github.com/pingcap/parser/ast"
docs/quickstart.md:	_ "github.com/pingcap/parser/test_driver"
docs/quickstart.md:>   You can use [`github.com/pingcap/parser/test_driver`](https://pkg.go.dev/github.com/pingcap/parser/test_driver) as the `parser_driver` for test. Again, if you need advanced features, please use the `parser_driver` in TiDB (run `go get -v github.com/pingcap/tidb/types/parser_driver@328b6d0` and import it).
docs/quickstart.md:> - The 2nd and 3rd arguments of [`parser.Parse()`](https://pkg.go.dev/github.com/pingcap/parser?tab=doc#Parser.Parse) are charset and collation respectively. If you pass an empty string into it, a default value is chosen.
docs/quickstart.md:Parser implements the interface [`ast.Node`](https://pkg.go.dev/github.com/pingcap/parser/ast?tab=doc#Node) for each kind of AST node, such as SelectStmt, TableName, ColumnName. [`ast.Node`](https://pkg.go.dev/github.com/pingcap/parser/ast?tab=doc#Node) provides a method `Accept(v Visitor) (node Node, ok bool)` to allow any struct that has implemented [`ast.Visitor`](https://pkg.go.dev/github.com/pingcap/parser/ast?tab=doc#Visitor) to traverse itself.
docs/quickstart.md:[`ast.Visitor`](https://pkg.go.dev/github.com/pingcap/parser/ast?tab=doc#Visitor) is defined as follows:

[xhebox]

Some docs need to be updated

$ rg "pingcap/parser" -t markdown
docs/update-parser-for-tidb.md:    GO111MODULE=on go mod edit -replace github.com/pingcap/parser=github.com/your-repo/parser@your-branch
docs/update-parser-for-tidb.md:GO111MODULE=on go get -u github.com/pingcap/parser@master
README.md:[![Go Report Card](https://goreportcard.com/badge/github.com/pingcap/parser)](https://goreportcard.com/report/github.com/pingcap/parser)
README.md:[![CircleCI Status](https://circleci.com/gh/pingcap/parser.svg?style=shield)](https://circleci.com/gh/pingcap/parser)
README.md:[![GoDoc](https://godoc.org/github.com/pingcap/parser?status.svg)](https://godoc.org/github.com/pingcap/parser)
README.md:[![codecov](https://codecov.io/gh/pingcap/parser/branch/master/graph/badge.svg)](https://codecov.io/gh/pingcap/parser)
README.md:- Highly compatible with MySQL: it supports almost all features of MySQL. For the complete details, see [parser.y](https://github.com/pingcap/parser/blob/master/parser.y) and [hintparser.y](https://github.com/pingcap/parser/blob/master/hintparser.y).
README.md:- Extensible: adding a new syntax requires only a few lines of Yacc and Golang code changes. As an example, see [PR-680](https://github.com/pingcap/parser/pull/680/files).
README.md:Please read the [quickstart](https://github.com/pingcap/parser/blob/master/docs/quickstart.md).
README.md:- [GitHub Issue](https://github.com/pingcap/parser/issues)
README.md:Here is how to [update parser for TiDB](https://github.com/pingcap/parser/blob/master/docs/update-parser-for-tidb.md).
docs/quickstart.md:First of all, you need to use `go get` to fetch the dependencies through git hash. The git hashes are available in [release page](https://github.com/pingcap/parser/releases). Take `v4.0.2` as an example:
docs/quickstart.md:go get -v github.com/pingcap/parser@3a18f1e
docs/quickstart.md:1. Use the [`parser.New()`](https://pkg.go.dev/github.com/pingcap/parser?tab=doc#New) function to instantiate a parser, and
docs/quickstart.md:2. Invoke the method [`Parse(sql, charset, collation)`](https://pkg.go.dev/github.com/pingcap/parser?tab=doc#Parser.Parse) on the parser.
docs/quickstart.md:	"github.com/pingcap/parser"
docs/quickstart.md:	"github.com/pingcap/parser/ast"
docs/quickstart.md:	_ "github.com/pingcap/parser/test_driver"
docs/quickstart.md:>   You can use [`github.com/pingcap/parser/test_driver`](https://pkg.go.dev/github.com/pingcap/parser/test_driver) as the `parser_driver` for test. Again, if you need advanced features, please use the `parser_driver` in TiDB (run `go get -v github.com/pingcap/tidb/types/parser_driver@328b6d0` and import it).
docs/quickstart.md:> - The 2nd and 3rd arguments of [`parser.Parse()`](https://pkg.go.dev/github.com/pingcap/parser?tab=doc#Parser.Parse) are charset and collation respectively. If you pass an empty string into it, a default value is chosen.
docs/quickstart.md:Parser implements the interface [`ast.Node`](https://pkg.go.dev/github.com/pingcap/parser/ast?tab=doc#Node) for each kind of AST node, such as SelectStmt, TableName, ColumnName. [`ast.Node`](https://pkg.go.dev/github.com/pingcap/parser/ast?tab=doc#Node) provides a method `Accept(v Visitor) (node Node, ok bool)` to allow any struct that has implemented [`ast.Visitor`](https://pkg.go.dev/github.com/pingcap/parser/ast?tab=doc#Visitor) to traverse itself.
docs/quickstart.md:[`ast.Visitor`](https://pkg.go.dev/github.com/pingcap/parser/ast?tab=doc#Visitor) is defined as follows:

Thanks, I did not managed to get enough time working on this in the past weeks. But is in my todo list.