Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: track memory usage of index lookup join #6169

Merged
merged 8 commits into from
Mar 30, 2018
Merged

executor: track memory usage of index lookup join #6169

merged 8 commits into from
Mar 30, 2018

Conversation

zz-jason
Copy link
Member

track memory usage of index lookup join, after this PR:

TiDB(localhost) > set @@tidb_mem_quota_query=10;
Query OK, 0 rows affected (0.00 sec)

TiDB(localhost) > desc select /*+ TIDB_INLJ(t1, t2) */ * from t t1 join t t2 on t1.id = t2.id;
+----------------+--------------+------------------------------+------+--------------------------------------------------------------------+----------+
| id             | parents      | children                     | task | operator info                                                      | count    |
+----------------+--------------+------------------------------+------+--------------------------------------------------------------------+----------+
| TableScan_11   |              |                              | cop  | table:t1, range:[-inf,+inf], keep order:false                      | 10000.00 |
| TableReader_12 | IndexJoin_10 |                              | root | data:TableScan_11                                                  | 10000.00 |
| IndexScan_7    |              |                              | cop  | table:t2, index:id, range:[<nil>,+inf], keep order:false           | 10000.00 |
| TableScan_8    |              |                              | cop  | table:t, keep order:false                                          | 10000.00 |
| IndexLookUp_9  | IndexJoin_10 |                              | root | index:IndexScan_7, table:TableScan_8                               | 10000.00 |
| IndexJoin_10   |              | TableReader_12,IndexLookUp_9 | root | outer:TableReader_12, inner join, outer key:t1.id, inner key:t2.id | 12500.00 |
+----------------+--------------+------------------------------+------+--------------------------------------------------------------------+----------+
6 rows in set (0.00 sec)

TiDB(localhost) > select /*+ TIDB_INLJ(t1, t2) */ * from t t1 join t t2 on t1.id = t2.id;
ERROR 1105 (HY000): Out Of Memory Quota!
"select /*+ TIDB_INLJ(t1, t2) */ * from t t1 join t t2 on t1.id = t2.id"{
  "quota": 10 Bytes
  "consumed": 376 Bytes
  "IndexJoin_10"{
    "quota": 32 GB
    "consumed": 376 Bytes
    "lookup join task 0xc420586f50"{
      "consumed": 376 Bytes
    }
    "lookup join task 0xc4206da930"{
      "consumed": 0 Bytes
    }
  }
}

XuHuaiyu
XuHuaiyu reviewed Mar 28, 2018
View reviewed changes
executor/index_lookup_join.go Outdated
@@ -216,6 +227,12 @@ func (e *IndexLookUpJoin) getFinishedTask(ctx context.Context) (*lookUpJoinTask,
if task != nil && task.cursor < task.outerResult.NumRows() {
return task, nil
}
if task != nil {
// TODO: jianzhang.zj detach this memory tracker after PR:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this comment

@zz-jason
Copy link
Member Author

@XuHuaiyu @coocood PTAL

@zz-jason
Copy link
Member Author

/run-all-tests

coocood
coocood reviewed Mar 28, 2018
View reviewed changes
executor/index_lookup_join.go Outdated
for _, b := range e.innerPtrBytes {
ptr := *(*chunk.RowPtr)(unsafe.Pointer(&b[0]))
matchedInner := task.innerResult.GetRow(ptr)
task.matchedInners = append(task.matchedInners, matchedInner)
}
newMemUsage := int64(cap(task.matchedInners)) * int64(unsafe.Sizeof(chunk.Row{}))
task.memTracker.Consume(newMemUsage - oldMemUsage)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a frequent operation and the matchedInners takes a very small proportion of the memory of the task.
How about just ignore it?

coocood
coocood reviewed Mar 28, 2018
View reviewed changes
executor/index_lookup_join.go Outdated
}
task.memTracker.Consume(int64(cap(task.matchedInners)) * int64(unsafe.Sizeof(chunk.Row{})))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

matchedInners is empty at the moment.

@coocood
Copy link
Member

coocood commented Mar 29, 2018

LGTM

@coocood coocood added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 29, 2018
@zz-jason
Copy link
Member Author

/run-all-tests

1 similar comment
@coocood
Copy link
Member

coocood commented Mar 29, 2018

/run-all-tests

XuHuaiyu
XuHuaiyu approved these changes Mar 30, 2018
View reviewed changes
Copy link
Contributor

@XuHuaiyu XuHuaiyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@XuHuaiyu XuHuaiyu added status/LGT2 Indicates that a PR has LGTM 2. all-tests-passed and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 30, 2018
@zz-jason zz-jason merged commit 564e889 into pingcap:master Mar 30, 2018
@zz-jason zz-jason deleted the dev/mem/indexlookupjoin branch March 30, 2018 02:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@coocood coocood coocood left review comments

@XuHuaiyu XuHuaiyu XuHuaiyu approved these changes

Assignees
No one assigned
Labels
status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zz-jason @coocood @XuHuaiyu