-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
plan: calc access path
when doing deriveStats
.
#6346
Conversation
simple benchmark result
Now:
If the sql have |
sysbench |
/run-all-tests |
plan/logical_plans.go
Outdated
type availableIndices struct { | ||
indices []*model.IndexInfo | ||
includeTableScan bool | ||
type indexPath struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add comments for this type.
plan/logical_plans.go
Outdated
// availableIndices is used for storing result of availableIndices function. | ||
availableIndices *availableIndices | ||
// possibleIndexPaths stores all the possible index path for physical plan, including table scan. | ||
// Please make true table scan is always the first element. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make sure
plan/logical_plans.go
Outdated
eqCondCount int | ||
indexFilters []expression.Expression | ||
tableFilters []expression.Expression | ||
filterUnmatched bool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not used?
plan/logical_plans.go
Outdated
index *model.IndexInfo | ||
ranges []*ranger.NewRange | ||
countAfterAccess float64 | ||
countAfterIndex float64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add comments about the two count.
plan/stats.go
Outdated
path.countAfterAccess = float64(ds.statisticTable.Count) | ||
path.countAfterIndex = float64(ds.statisticTable.Count) | ||
var err error | ||
if path.isRowID { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we extract a function for table path and another function for index path?
types/datum.go
Outdated
@@ -122,6 +122,11 @@ func (d *Datum) IsNull() bool { | |||
return d.k == KindNull | |||
} | |||
|
|||
// IsMaxValue checks if datum is max value. | |||
func (d *Datum) IsMaxValue() bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not used?
/run-all-tests |
index path
when doing deriveStats
.access path
when doing deriveStats
.
plan/logical_plans.go
Outdated
// possibleIndexPaths stores all the possible index path for physical plan, including table scan. | ||
// Please make true table scan is always the first element. | ||
possibleIndexPaths []*indexPath | ||
// possibleAccessPaths stores all the possible index path for physical plan, including table scan. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all the possible access paths
plan/logical_plans.go
Outdated
forced bool | ||
} | ||
|
||
func (ds *DataSource) prepareTablePath(path *accessPath) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since it's called by deriveStats
, How about renaming it to deriveTablePathStats
?
plan/logical_plans.go
Outdated
return nil | ||
} | ||
|
||
func (ds *DataSource) prepareIndexPath(path *accessPath) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
plan/physical_plan_builder.go
Outdated
@@ -308,25 +310,10 @@ func (ds *DataSource) convertToIndexScan(prop *requiredProp, idx *model.IndexInf | |||
is.Hist = &statsTbl.Indices[idx.ID].Histogram | |||
} | |||
rowCount := float64(statsTbl.Count) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can be removed
plan/stats.go
Outdated
// PushDownNot here can convert query 'not (a != 1)' to 'a = 1'. | ||
for i, expr := range ds.pushedDownConds { | ||
ds.pushedDownConds[i] = expression.PushDownNot(nil, expr, false) | ||
} | ||
ds.statsAfterSelect = ds.getStatsByFilter(ds.pushedDownConds) | ||
return ds.statsAfterSelect | ||
for _, path := range ds.possibleAccessPaths { | ||
var err error |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can remove this line and use err :=
in line 135 and 141
plan/physical_plan_builder.go
Outdated
t = idxTask | ||
if tblTask.cost() < t.cost() { | ||
t = tblTask | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about adding a continue in this code block ?
plan/logical_plans.go
Outdated
} else { | ||
path.ranges = ranger.FullIntNewRange(false) | ||
} | ||
path.countAfterAccess = float64(ds.statisticTable.Count) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Duplicated with line 333?
} else { | ||
path.indexFilters, path.tableFilters = splitIndexFilterConditions(ds.pushedDownConds, path.index.Columns, ds.tableInfo) | ||
} | ||
path.countAfterIndex = path.countAfterAccess |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Duplicated with line 364?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed one.
plan/gen_physical_plans.go
Outdated
includeTableScan := x.availableIndices.includeTableScan | ||
if includeTableScan && len(innerJoinKeys) == 1 { | ||
indexPaths := x.possibleAccessPaths | ||
if len(x.possibleAccessPaths) > 0 && x.possibleAccessPaths[0].isRowID { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not use len(indexPaths) > 0 ...
plan/logical_plans.go
Outdated
sc := ds.ctx.GetSessionVars().StmtCtx | ||
path.countAfterAccess = float64(ds.statisticTable.Count) | ||
var pkCol *expression.Column | ||
if pkCol != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
when will pkCol != nil
?
plan/logical_plans.go
Outdated
} | ||
path.countAfterAccess = float64(ds.statisticTable.Count) | ||
if len(ds.pushedDownConds) > 0 { | ||
if pkCol != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
plan/logical_plans.go
Outdated
if len(ds.pushedDownConds) > 0 { | ||
if pkCol != nil { | ||
path.accessConds, path.tableFilters = ranger.DetachCondsForTableRange(ds.ctx, ds.pushedDownConds, pkCol) | ||
path.ranges, err = ranger.BuildTableRange(path.accessConds, sc, pkCol.RetType) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We may put sc as the first parameter.
plan/logical_plans.go
Outdated
type availableIndices struct { | ||
indices []*model.IndexInfo | ||
includeTableScan bool | ||
func (ds *DataSource) prepareTablePath(path *accessPath) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this be more concise?
func (ds *DataSource) prepareTablePath(path *accessPath) error {
var err error
var pkCol *expression.Column
sc := ds.ctx.GetSessionVars().StmtCtx
path.countAfterAccess = float64(ds.statisticTable.Count)
path.ranges = ranger.FullIntNewRange(false)
path.tableFilters = ds.pushedDownConds
if pkCol == nil {
return nil
}
path.ranges = ranger.FullIntNewRange(mysql.HasUnsignedFlag(pkCol.RetType.Flag))
if len(ds.pushedDownConds) == 0 {
return nil
}
path.accessConds, path.tableFilters = ranger.DetachCondsForTableRange(ds.ctx, ds.pushedDownConds, pkCol)
path.ranges, err = ranger.BuildTableRange(path.accessConds, sc, pkCol.RetType)
if err != nil {
return errors.Trace(err)
}
path.countAfterAccess, err = ds.statisticTable.GetRowCountByIntColumnRanges(sc, pkCol.ID, path.ranges)
return errors.Trace(err)
}
/run-all-tests |
plan/gen_physical_plans.go
Outdated
includeTableScan := x.availableIndices.includeTableScan | ||
if includeTableScan && len(innerJoinKeys) == 1 { | ||
accessPaths := x.possibleAccessPaths | ||
if len(accessPaths) > 0 && x.possibleAccessPaths[0].isRowID { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we need to check the length of innerJoinKeys
?
4313fec
to
5d3620f
Compare
/run-all-tests |
// PushDownNot here can convert query 'not (a != 1)' to 'a = 1'. | ||
for i, expr := range ds.pushedDownConds { | ||
ds.pushedDownConds[i] = expression.PushDownNot(nil, expr, false) | ||
} | ||
ds.statsAfterSelect = ds.getStatsByFilter(ds.pushedDownConds) | ||
return ds.statsAfterSelect | ||
for _, path := range ds.possibleAccessPaths { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about:
if !ds. possibleAccessPaths[0].isRowID {
panic("table path should always be the first access path"
}
err := ds.deriveTablePathStats(path)
...
for _, path := range ds.possibleAccessPaths[1:] {
err := ds.deriveIndexPathStats(path)
...
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, the first element can be index path.
plan/property_cols_prune.go
Outdated
cols, _ := expression.IndexInfo2Cols(ds.schema.Columns, idx) | ||
if len(cols) > 0 { | ||
result = append(result, cols) | ||
for _, path := range ds.possibleAccessPaths { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about:
if !ds.possibleAccessPaths[0].isRowID {
panic("table path should always be the first access path")
}
col := ds.getPKIsHandleCol()
...
for _, path := range ds.possibleAccessPaths[1:] {
cols, _ := expression.IndexInfo2Cols(ds.schema.Columns, path.index)
...
}
plan/gen_physical_plans.go
Outdated
innerPlan := x.forceToTableScan(pkCol) | ||
return p.constructIndexJoin(prop, innerJoinKeys, outerJoinKeys, outerIdx, innerPlan, nil, nil) | ||
accessPaths := x.possibleAccessPaths | ||
if len(accessPaths) > 0 && x.possibleAccessPaths[0].isRowID { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can just:
if len(accessPaths) > 0 {
...
}
if !ds.relevantIndices[i] && len(prop.cols) == 0 { | ||
continue | ||
|
||
for _, path := range ds.possibleAccessPaths { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about:
if !ds.possibleAccessPaths[0].isRowID {
panic("xxx")
}
tblTask, err := ds.convertToTableScan(prop, ds.possibleAccessPaths[0])
...
for _, path := range ds.possibleAccessPaths[1:] {
if !(len(path.accessConds) > 0 || len(prop.cols) > 0 || path.forced) {
continue
}
...
}
is.filterCondition = ds.pushedDownConds | ||
} | ||
} | ||
is.AccessCondition, is.Ranges, is.filterCondition, eqCount = path.accessConds, path.ranges, path.indexFilters, path.eqCondCount |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this line is too long, can we split it to multi-lines ?
plan/logical_plans.go
Outdated
if pkCol != nil { | ||
path.ranges = ranger.FullIntRange(mysql.HasUnsignedFlag(pkCol.RetType.Flag)) | ||
} else { | ||
if pkCol == nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'better add some test cases to cover pkCol == nil
and pkCol !=nil
.
plan/logical_plans.go
Outdated
return errors.Trace(err) | ||
} | ||
path.countAfterAccess, err = ds.statisticTable.GetRowCountByIntColumnRanges(sc, pkCol.ID, path.ranges) | ||
if err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no need to check,
return errors.Trace(err) is ok.
plan/gen_physical_plans.go
Outdated
innerPlan := x.forceToTableScan(pkCol) | ||
return p.constructIndexJoin(prop, innerJoinKeys, outerJoinKeys, outerIdx, innerPlan, nil, nil) | ||
accessPaths := x.possibleAccessPaths | ||
if len(accessPaths) > 0 && x.possibleAccessPaths[0].isRowID { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/ x.possibleAccessPaths[0]/ accessPaths[0]
plan/logical_plans.go
Outdated
indexFilters []expression.Expression | ||
tableFilters []expression.Expression | ||
// isRowID indicates this path stores the information for table scan. | ||
isRowID bool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not just name it as isTablePath?
plan/planbuilder.go
Outdated
func getAvailableIndices(indexHints []*ast.IndexHint, tableInfo *model.TableInfo) (*availableIndices, error) { | ||
publicIndices := make([]*model.IndexInfo, 0, len(tableInfo.Indices)) | ||
for _, index := range tableInfo.Indices { | ||
func matchPathByIndexName(paths []*accessPath, idxName model.CIStr) *accessPath { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think matchXxx may return a bool value,
s/match/get
plan/planbuilder.go
Outdated
// If we have got "FORCE" or "USE" index hint but got no available index, | ||
// we have to use table scan. | ||
return &availableIndices{available, len(available) == 0}, nil | ||
if len(available) == 0 { | ||
available = append(available, &accessPath{isRowID: true}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can just use publicPath[0] here without building a new accessPath ?
plan/planbuilder.go
Outdated
continue | ||
} | ||
// Currently we don't distinguish between "FORCE" and "USE" because | ||
// our cost estimation is not reliable. | ||
hasUseOrForce = true | ||
available = append(available, idx) | ||
path.forced = true | ||
available = append(available, path) | ||
} | ||
} | ||
|
||
if !hasScanHint { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if !hasScanHint || !hasUseOrForce {
available = publicPaths
}
available = removeIgnorePaths(available, ignored)
if len(available) == 0 {
....
}
return avaibale, nil
plan/logical_plans.go
Outdated
tableFilters []expression.Expression | ||
// isRowID indicates this path stores the information for table scan. | ||
isRowID bool | ||
// forced means this index is generated by `use/force index()`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/ this index/ this path
plan/logical_plans.go
Outdated
eqCondCount int | ||
indexFilters []expression.Expression | ||
tableFilters []expression.Expression | ||
// isRowID indicates this path stores the information for table scan. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/ this path stores the information for table scan/ whether this path is a table path
plan/logical_plans.go
Outdated
// availableIndices is used for storing result of availableIndices function. | ||
availableIndices *availableIndices | ||
// possibleAccessPaths stores all the possible access path for physical plan, including table scan. | ||
// Please make sure table path is always the first element. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a comment that:
Note: The first element is `not` always the table path.
plan/physical_plan_builder.go
Outdated
idxTask, err := ds.convertToIndexScan(prop, idx) | ||
continue | ||
} | ||
if len(path.accessConds) > 0 || len(prop.cols) > 0 || path.forced { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Please add a comment on this check.
- path.forced is checked only here,
we do not really need this attribute,
since if an indexPath is involved, path.forced must be true.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But, path.accessConds
and prop.cols
may be nil.
Such as select * from t use index(idx)
; We need to choose idx, but there' no access cond and prop.
plan/stats.go
Outdated
return p.stats | ||
var err error | ||
p.stats, err = p.children[0].deriveStats() | ||
if err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return p.stats, errors.Trace(err)
@winoros Please address the above comments. |
plan/logical_plans.go
Outdated
// availableIndices is used for storing result of availableIndices function. | ||
availableIndices *availableIndices | ||
// possibleAccessPaths stores all the possible access path for physical plan, including table scan. | ||
// Please make sure table path is always the first element if we have table path. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we check isTablePath
in each path, then why do we need to make sure the table path is the first?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For index join, if there's pk and match the condition, we will directly choose it.
Use a loop to find the tablePath first.
So i removed this constraint.
pkCol = expression.ColInfo2Col(ds.schema.Columns, pkColInfo) | ||
} | ||
} | ||
if pkCol == nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'better add some test cases to cover pkCol == nil and pkCol !=nil.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the status of pkCol is covered executor's unit test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But why all test cases can be passed when line 336~340 was missed before,
we may need some test for checking this case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let me check it first.
plan/gen_physical_plans.go
Outdated
innerPlan := x.forceToTableScan(pkCol) | ||
return p.constructIndexJoin(prop, innerJoinKeys, outerJoinKeys, outerIdx, innerPlan, nil, nil) | ||
accessPaths := x.possibleAccessPaths | ||
if len(accessPaths) > 0 && accessPaths[0].isTablePath { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
when will len(accessPaths) == 0?
plan/gen_physical_plans.go
Outdated
for i, path := range accessPaths { | ||
if path.isTablePath { | ||
tblPath = path | ||
accessPaths = append(accessPaths[:i], accessPaths[i+1:]...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This operation modifies possibleAccessPaths
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We just iterate possibleAccessPaths
and ignore table path at line 278.
plan/physical_plan_builder.go
Outdated
// We will use index to generate physical plan if: | ||
// this path's access cond is not nil or | ||
// we have prop to match or | ||
// this index is force to choose. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is forced
col := ds.getPKIsHandleCol() | ||
if col != nil { | ||
result = append(result, []*expression.Column{col}) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add a continue
here so we do not need the following else
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
statsInfo
.