TableReader consumes too much memory

[SunRunAway]

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

Run TPCH Q18, TableReader_86 consumes 886MB memory.
In TiDB v4.0, the default memory quota is 1GB, so we need reduce the memory consumption.

"explain analyze select  c_name,  c_custkey,  o_orderkey,  o_orderdate,  o_totalprice,  sum(l_quantity) from  customer,  orders,  lineitem where  o_orderkey in (   select    l_orderkey   from    lineitem   group by    l_orderkey having     sum(l_quantity) > 314  )  and c_custkey = o_custkey  and o_orderkey = l_orderkey group by  c_name,  c_custkey,  o_orderkey,  o_orderdate,  o_totalprice order by  o_totalprice desc,  o_orderdate limit 100"{
  "quota": 953.67431640625 MB
  "consumed": 970.9154815673828 MB
  "TopN_27"{
    "consumed": 0 Bytes
    "rowChunks"{
      "consumed": 0 Bytes
    }
  }
  "TableReader_88"{
    "consumed": 0 Bytes
  }
  "TableReader_86"{
    "consumed": 886.1686134338379 MB
  }
  "HashRightJoin_84"{
    "consumed": 43.61027526855469 MB
    "hashJoin.buildSideResult"{
      "consumed": 43.61027526855469 MB
    }
  }
  "TableReader_97"{
    "consumed": 4.5437774658203125 MB
  }
  "HashAgg_96"{
    "consumed": 36.55274963378906 MB
  }
  "Selection_89"{
    "consumed": 1.7578125 KB
  }
  "HashLeftJoin_72"{
    "consumed": 0 Bytes
    "hashJoin.buildSideResult"{
      "consumed": 0 Bytes
    }
  }
  "IndexHashJoin_46"{
    "consumed": 0 Bytes
    "lookup join task 0xc002d58200"{
      "consumed": 0 Bytes
      "chunk.List"{
        "consumed": 0 Bytes
      }
    }
  }
  "HashAgg_33"{
    "consumed": 34.90625 KB
  }
  "Projection_24"{
    "consumed": 4.36328125 KB
  }
}

2. What did you expect to see?

3. What did you see instead?

4. What version of TiDB are you using? (tidb-server -V or run select tidb_version(); on TiDB)

MySQL [test]> select tidb_version();
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                                                      |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v4.0.0-beta-446-g5268094af
Git Commit Hash: 5268094afe05c7efef0d91d2deeec428cc85abe6
Git Branch: master
UTC Build Time: 2020-03-17 02:22:07
GoVersion: go1.13
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

SIG slack channel

#sig-exec

Score

• 300

Mentor

• @XuHuaiyu

[SunRunAway]

Another suspect,
https://github.com/pingcap/tidb/blob/ddcc89d/store/tikv/client_batch.go#L580

This channel will cache one response in memory without memory tracking, finally double the actual memory usage of copIterator

[Yisaer]

It seems the TableReader and its selectResult would share the same MemoryTracker.

tidb/executor/table_reader.go

Lines 195 to 207 in fdfeac7

	func (e *TableReaderExecutor) buildResp(ctx context.Context, ranges []*ranger.Range) (distsql.SelectResult, error) {
	var builder distsql.RequestBuilder
	kvReq, err := builder.SetTableRanges(getPhysicalTableID(e.table), ranges, e.feedback).
	SetDAGRequest(e.dagPB).
	SetStartTS(e.startTS).
	SetDesc(e.desc).
	SetKeepOrder(e.keepOrder).
	SetStreaming(e.streaming).
	SetFromSessionVars(e.ctx.GetSessionVars()).
	SetMemTracker(e.memTracker).
	SetStoreType(e.storeType).
	SetAllowBatchCop(e.batchCop).
	Build()

And in fetchResp, selectResult would read all the data and record memory consume in its memory tracker.

tidb/distsql/select_result.go

Line 91 in fdfeac7

func (r *selectResult) fetchResp(ctx context.Context) error {

That's why TableReader_86 consumes 886MB which I think seems make sense, not sure whether if it is a bug.

[fzhedu]

TableReader does not to sum the memory size of all chunks, because it is a non-blocking operator, where each chuck is fetched from TiKV and is pulled by its parent operator. In fact, for each non-blocking operator, its max memory size = concurrency * size of each chunk. It is different for blocking operators, such as join, aggregation and sort.

In TableReader, there are two points where memory usage is collected.

1. copIterator add memory size in sendToRespCh and minus memory size in recvFromRespCh. The memory occupation depends on how many chunks (concurrency) reside in RespCh.

2. While in distSQL.SelectResult , fetchResp add memory usage for each chunk, and minus chunk's memory size in
   fetchResp and close.

The large memory occupation should result from 1, that means there are many resided chunks when using high concurrency. So it is necessary to shrink the concurrency of the tablescan or enlarge the concurrency of its parent operator to consume more chunks of scan.

[XuHuaiyu]

#18706
#18704

[Yisaer]

fixed in #21168

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Not a bug

1. Remove the 'type/bug' label
2. Add notes to indicate why it is not a bug

Duplicate bug

1. Add the 'type/duplicate' label
2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions