*: make parser package dependency as small as possible #7989

tiancaiamao · 2018-10-22T09:31:26Z

What problem does this PR solve?

Make parser package's dependency as small as possible.

This is nearly the final step before we move parser to a separate repository

All package in that picture will be moved to this new repo https://github.com/pingcap/parser

What is changed and how it works?

types is split into two, a smaller one put into parser/types which does not have much dependency and seldom changes.
define NewDecimal/NewHexLiteral/NewBitLiteral function and types/parser_driver provide the implementation

Check List

Tests

No code

@lamxTyler @zz-jason

Statement use RecordSet and RecordSet use chunk After moving them out, the ast package do not depends on chunk any more

winoros · 2018-10-22T16:40:32Z

parser/misc.go

@@ -584,7 +583,7 @@ func (s *Scanner) isTokenIdentifier(lit string, offset int) int {
 		}
 	}

-	checkBtFuncToken, tokenStr := false, hack.String(data)


Will this influence performance much?

Perhaps.
When parser is moved into a separate repository, optimize its performance would be another task. I suggest that we can get the first step done.

alivxxx

Why they are so many code addition? If we just split, the code deletion and addition should almost the same.

parser/types/etc.go

alivxxx

LGTM

jackysp · 2018-10-23T12:02:01Z

Should we merge pingcap/parser#1 with this one?

tiancaiamao · 2018-10-23T12:41:09Z

@jackysp pingcap/parser#1 is actually this code, with the import path revised

After this PR is merged, I will

update the pingcap/parser repo, make sure it's latest
remove the tidb/parser directory from tidb, import pingcap/parser and update vendor
remove vendor, migrate to go module

jackysp

LGTM

jackysp · 2018-10-24T03:02:13Z

/run-all-tests

bugfix fixed pingcap#7518 expression: MySQL compatible current_user function (pingcap#7801) plan: propagate constant over outer join (pingcap#7794) - extract `outerCol = const` from join conditions and filter conditions, substitute `outerCol` in join conditions with `const`; - extract `outerCol = innerCol` from join conditions, derive new join conditions based on this column equal condition and `outerCol` related expressions in join conditions and filter conditions; util/timeutil: fix data race caused by forgetting set stats lease to 0 (pingcap#7901) stats: handle ddl event for partition table (pingcap#7903) plan: implement Operand and Pattern of cascades planner. (pingcap#7910) planner: not convert to TableDual if empty range is derived from deferred constants (pingcap#7808) plan: move projEliminate behind aggEliminate (pingcap#7909) admin: fix admin check table bug of byte compare (pingcap#7887) * admin: remove reflect deepEqual stats: fix panic caused by empty histogram (pingcap#7912) plan: fix panic caused by empty schema of LogicalTableDual (pingcap#7906) * fix drop view if exist error (pingcap#7833) executor: refine `explain analyze` (pingcap#7888) executor: add an variable to compatible with MySQL insert for OGG (pingcap#7863) expression: maintain `DeferredExpr` in aggressive constant folding. (pingcap#7915) stats: fix histogram boundaries overflow error (pingcap#7883) ddl:support the definition of `null` change to `not null` using `alter table` (pingcap#7771) * ddl:support the definition of null change to not null using alter table ddl: add check when create table with foreign key. (pingcap#7885) * ddl: add check when create table with foreign key planner: eliminate if null on non null column (pingcap#7924) executor: fix a bug in point get (pingcap#7934) planner, executor: refine ColumnPrune for LogicalUnionAll (pingcap#7930) executor: fix panic when limit is too large (pingcap#7936) ddl: add TiDB version to metrics (pingcap#7902) stats: limit the length of sample values (pingcap#7931) vendor: update tipb (pingcap#7893) planner: support the Group and GroupExpr for the cascades planner (pingcap#7917) store/tikv: log more information when other err occurs (pingcap#7948) types: fix date time parse (pingcap#7933) ddl: just print error message when ddl job is normal to calcel, to eliminate noisy log (pingcap#7875) stats: update delta info for partition table (pingcap#7947) explaintest: add explain test for partition pruning (pingcap#7505) util: move disjoint set to util package (pingcap#7950) util: add PreAlloc4Row and Insert for Chunk and List (pingcap#7916) executor: add the slow log for commit (pingcap#7951) expression: add builtin json_keys (pingcap#7776) privilege: add USAGE in `show grants` for mysql compatibility (pingcap#7955) ddl: fix invailid ddl job panic (pingcap#7940) *: move ast.NewValueExpr to standalone parser_driver package (pingcap#7952) Make the ast package get rid of the dependency of types.Datum server: allow cors http request (pingcap#7939) *: move `Statement` and `RecordSet` from ast to sqlexec package (pingcap#7970) pr suggestion update executor/aggfuncs: split unit tests to corresponding file (pingcap#7993) store/tikv: fix typo (pingcap#7990) executor, planner: clone proj schema for different children in buildProj4Union (pingcap#7999) executor: let information_schema be the first database in ShowDatabases (pingcap#7938) stats: use local feedback for partition table (pingcap#7963) executor: add unit test for aggfuncs (pingcap#7966) server: add log for binary execute statement (pingcap#7987) admin: refine admin check decoder (pingcap#7862) executor: improve wide table insert & update performance (pingcap#7935) ddl: fix reassigned partition id in `truncate table` does not take effect (pingcap#7919) fix reassigned partition id in truncate table does not take effect add changelog for 2.1.0 rc4 (pingcap#8020) *: make parser package dependency as small as possible (pingcap#7989) parser: support `:=` in the `set` syntax (pingcap#8018) According to MySQL document, `set` use the = assignment operator, but the := assignment operator is also permitted stats: garbage collect stats for partition table (pingcap#7962) docs: add the proposal for the column pool (pingcap#7988) expression: refine built-in func truncate to support uint arg (pingcap#8000) stats: support show stats for partition table (pingcap#8023) stats: update error rate for partition table (pingcap#8022) stats: fix estimation for out of range point queries (pingcap#8015) *: move parser to a separate repository (pingcap#8036) executor: fix wrong result when index join on union scan. (pingcap#8031) Do not modify Plan of dataReaderBuilder directly, because it would impact next batch of outer rows, as well as other concurrent inner workers. Instead, build a local child builder to store the child plan. planner: fix a panic of a cached prepared statement with IndexScan (pingcap#8017) *: fix the issue of executing DDL after executing SQL failure in txn (pingcap#8044) * ddl, executor: fix the issue of executing DDL after executing SQL failure in txn add unit test remove debug info add like evaluator case sensitive test ddl, domain: make schema correct after canceling jobs (pingcap#7997) unit test fix code format proposal: maintaining histograms in plan. (pingcap#7605) support _tidb_rowid for table scan range (pingcap#8047) var rename fix

tiancaiamao added 3 commits October 20, 2018 11:40

*: move Statement and RecordSet from ast to sqlexec package

e3b49f6

Statement use RecordSet and RecordSet use chunk After moving them out, the ast package do not depends on chunk any more

Merge branch 'master' into statement-recordset

89b7e37

tiny update

04661ba

winoros reviewed Oct 22, 2018

View reviewed changes

zz-jason added type/enhancement The issue or PR belongs to an enhancement. component/parser labels Oct 23, 2018

alivxxx reviewed Oct 23, 2018

View reviewed changes

parser/types/etc.go Show resolved Hide resolved

address comment

aeb53f4

alivxxx reviewed Oct 23, 2018

View reviewed changes

alivxxx added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 23, 2018

jackysp approved these changes Oct 24, 2018

View reviewed changes

Merge branch 'master' into parser-dependency

6df3f9c

jackysp added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 24, 2018

jackysp mentioned this pull request Oct 24, 2018

parser: support := in the set syntax #8018

Merged

zz-jason merged commit bd08b0b into pingcap:master Oct 24, 2018

zz-jason added the status/all tests passed label Oct 24, 2018

tiancaiamao added a commit to tiancaiamao/tidb that referenced this pull request Nov 21, 2018

*: make parser package dependency as small as possible (pingcap#7989)

b06a5d4

tiancaiamao deleted the parser-dependency branch August 9, 2019 08:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: make parser package dependency as small as possible #7989

*: make parser package dependency as small as possible #7989

tiancaiamao commented Oct 22, 2018

winoros Oct 22, 2018 •

edited

Loading

tiancaiamao Oct 23, 2018

alivxxx left a comment

alivxxx left a comment

jackysp commented Oct 23, 2018

tiancaiamao commented Oct 23, 2018

jackysp left a comment

jackysp commented Oct 24, 2018

*: make parser package dependency as small as possible #7989

*: make parser package dependency as small as possible #7989

Conversation

tiancaiamao commented Oct 22, 2018

What problem does this PR solve?

What is changed and how it works?

Check List

winoros Oct 22, 2018 • edited Loading

Choose a reason for hiding this comment

tiancaiamao Oct 23, 2018

Choose a reason for hiding this comment

alivxxx left a comment

Choose a reason for hiding this comment

alivxxx left a comment

Choose a reason for hiding this comment

jackysp commented Oct 23, 2018

tiancaiamao commented Oct 23, 2018

jackysp left a comment

Choose a reason for hiding this comment

jackysp commented Oct 24, 2018

winoros Oct 22, 2018 •

edited

Loading