stats: dynamically update the average column size #6170

fipped · 2018-03-28T07:10:43Z

This PR is about the work of updating the tot_col_size of histograms when executing insert, delete and update statement. Therefore, a correct average column size can be computed when we use show stats_histograms statement.
The tot_col_size we dynamically maintain will be a little different from that computed by analyze statement since we get the size from the value directly instead of computing from the coded bytes, that is we leave out the length byte.
PTAL @lamxTyler @winoros

alivxxx · 2018-03-28T07:27:29Z

executor/write.go

-
-	ctx.GetSessionVars().TxnCtx.UpdateDeltaForTable(t.Meta().ID, 0, 1)
+	colSize := make(map[int64]int64)
+	for id, col := range t.WritableCols() {


This should also consider the column state?

alivxxx · 2018-03-28T07:28:13Z

executor/write.go

-	ctx.GetSessionVars().TxnCtx.UpdateDeltaForTable(t.Meta().ID, 0, 1)
+	colSize := make(map[int64]int64)
+	for id, col := range t.WritableCols() {
+		colSize[col.ID] = int64(len(newData[id].GetBytes()) - len(oldData[id].GetBytes()))


No need to record the col size for fix length type.

alivxxx · 2018-03-28T07:30:45Z

statistics/handle_test.go

+		hist.TotColSize = temp
+
+		c.Assert(hist.CMSketch, NotNil)
+		c.Assert(newStatsTbl.Columns[id].CMSketch, NotNil)


It is already checked at CMSketch.Equal.

alivxxx · 2018-03-28T07:38:46Z

statistics/update.go

+	}
+	updated := h.ctx.GetSessionVars().StmtCtx.AffectedRows() > 0
+	_, err = h.ctx.(sqlexec.SQLExecutor).Execute(ctx, "commit")
+	return updated, errors.Trace(err)


If the error is not nil, then updated will be always true.

alivxxx · 2018-03-28T07:44:59Z

statistics/update.go

+	version := h.ctx.Txn().StartTS()
+
+	for key, val := range delta.ColSize {
+		sql := fmt.Sprintf("update mysql.stats_histograms set version = %d, tot_col_size = tot_col_size + %d where hist_id = %d and table_id = %d", version, val, key, id)


There may be two histograms with the same hist_id and table_id.

coocood · 2018-03-28T09:07:07Z

executor/write.go

-	ctx.GetSessionVars().TxnCtx.UpdateDeltaForTable(t.Meta().ID, 0, 1)
+	colSize := make(map[int64]int64)
+	for id, col := range t.WritableCols() {
+		colSize[col.ID] = int64(len(newData[id].GetBytes()) - len(oldData[id].GetBytes()))


GetBytes is only valid when the Kind of the Datum is KindString or KindBytes.

Maybe we need to add a method for Datum to return the size?

alivxxx · 2018-03-28T11:24:53Z

executor/write.go

+	colSize := make(map[int64]int64)
+	for id, col := range t.WritableCols() {
+		if col.State == model.StatePublic && newData[id].Size() != 0 {
+			colSize[col.ID] = int64(newData[id].Size() - oldData[id].Size())


This is not right? If the new data happens to be null, but the old data is not null.

alivxxx · 2018-03-28T11:25:37Z

executor/write.go

+	colSize := make(map[int64]int64)
+	for id, col := range t.WritableCols() {
+		if col.State == model.StatePublic {
+			colSize[col.ID] = -int64(data[id].Size())


If the Size is 0, it is not needed to update it?

alivxxx · 2018-03-28T11:27:10Z

table/tables/tables.go

+	colSize := make(map[int64]int64)
+	for id, col := range t.WritableCols() {
+		if col.State == model.StatePublic {
+			colSize[col.ID] = int64(r[id].Size())


coocood · 2018-03-28T12:22:29Z

types/datum.go

@@ -122,6 +122,11 @@ func (d *Datum) IsNull() bool {
 	return d.k == KindNull
 }

+// Size gets the length of byte of the datum.
+func (d *Datum) Size() int {
+	return len(d.b)


We need to switch on d.k to determine the size of the datum.

Or if we only care about varlen size, then we don't need this method, use len(d.GetBytes()) will do.

shenli · 2018-04-02T15:19:06Z

@fipped Please resolve the conflicts.

shenli · 2018-04-02T15:19:15Z

@lamxTyler PTAL

alivxxx · 2018-04-03T08:08:49Z

statistics/update.go

+		if err = h.dumpTableStatColSizeToKV(id, item); err != nil {
+			return errors.Trace(err)
+		}
+		m := h.globalMap[id]


We do not need to update the map if the updated is true, just delete it.

alivxxx

LGTM

alivxxx · 2018-04-04T11:14:22Z

/run-all-tests

into dynamic_update_col_size

fipped · 2018-04-11T05:02:58Z

PTAL @coocood

coocood · 2018-04-12T12:14:55Z

sessionctx/variable/session.go

@@ -101,13 +101,21 @@ type TransactionContext struct {
 }

 // UpdateDeltaForTable updates the delta info for some table.
-func (tc *TransactionContext) UpdateDeltaForTable(tableID int64, delta int64, count int64) {
+func (tc *TransactionContext) UpdateDeltaForTable(tableID int64, delta int64, count int64, colSize *map[int64]int64) {


We don't need to use pointer for a map.

coocood · 2018-04-12T12:32:23Z

executor/write.go

-
-	ctx.GetSessionVars().TxnCtx.UpdateDeltaForTable(t.Meta().ID, 0, 1)
+	colSize := make(map[int64]int64)
+	for id, col := range t.WritableCols() {


We can just range over t.Cols(), it will return public columns only.

coocood · 2018-04-12T12:32:58Z

executor/write.go

@@ -392,7 +400,16 @@ func (e *DeleteExec) removeRow(ctx sessionctx.Context, t table.Table, h int64, d
 	}
 	ctx.StmtAddDirtyTableOP(DirtyTableDeleteRow, t.Meta().ID, h, nil)
 	ctx.GetSessionVars().StmtCtx.AddAffectedRows(1)
-	ctx.GetSessionVars().TxnCtx.UpdateDeltaForTable(t.Meta().ID, -1, 1)
+	colSize := make(map[int64]int64)
+	for id, col := range t.WritableCols() {


into dynamic_update_col_size

coocood · 2018-04-13T05:52:23Z

LGTM

stats: dynamically update the average column size

98c4e9f

alivxxx reviewed Mar 28, 2018

View reviewed changes

coocood reviewed Mar 28, 2018

View reviewed changes

address comments

94b0c86

alivxxx reviewed Mar 28, 2018

View reviewed changes

no update when colSize is 0

30174aa

coocood reviewed Mar 28, 2018

View reviewed changes

shenli added the component/statistics label Apr 2, 2018

fipped added 2 commits April 3, 2018 15:03

merge conflict

b370aa4

remove .Size()

31e3ecf

alivxxx reviewed Apr 3, 2018

View reviewed changes

fipped added 2 commits April 4, 2018 17:21

Merge branch 'master' into dynamic_update_col_size

75e7e75

address comment

6955210

alivxxx reviewed Apr 4, 2018

View reviewed changes

Merge branch 'master' into dynamic_update_col_size

ab70d0a

alivxxx added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 4, 2018

fipped added 4 commits April 10, 2018 19:42

merge conflict

dc1b505

Merge branch 'dynamic_update_col_size' of https://github.com/fipped/tidb

cc82999

into dynamic_update_col_size

fix CI

516d6f2

Merge branch 'master' into dynamic_update_col_size

a5c7326

coocood reviewed Apr 12, 2018

View reviewed changes

fipped added 2 commits April 13, 2018 11:20

address comment

414ac97

Merge branch 'dynamic_update_col_size' of https://github.com/fipped/tidb

cb659ed

into dynamic_update_col_size

fipped added 3 commits April 13, 2018 11:22

Merge branch 'master' into dynamic_update_col_size

7b4eae3

Merge branch 'dynamic_update_col_size' of https://github.com/fipped/tidb

ab7e73b

into dynamic_update_col_size

address comment

4aca564

coocood approved these changes Apr 13, 2018

View reviewed changes

coocood and others added 2 commits April 13, 2018 13:53

Merge branch 'master' into dynamic_update_col_size

b95ca17

Merge branch 'master' into dynamic_update_col_size

b726f66

winoros added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 13, 2018

zz-jason merged commit 9fbf8a6 into pingcap:master Apr 13, 2018

sre-bot added the contribution This PR is from a community contributor. label Dec 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stats: dynamically update the average column size #6170

stats: dynamically update the average column size #6170

fipped commented Mar 28, 2018 •

edited

Loading

alivxxx Mar 28, 2018

alivxxx Mar 28, 2018

alivxxx Mar 28, 2018

alivxxx Mar 28, 2018

alivxxx Mar 28, 2018

coocood Mar 28, 2018

alivxxx Mar 28, 2018

alivxxx Mar 28, 2018

alivxxx Mar 28, 2018

coocood Mar 28, 2018

coocood Mar 28, 2018

shenli commented Apr 2, 2018

shenli commented Apr 2, 2018

alivxxx Apr 3, 2018

alivxxx left a comment

alivxxx commented Apr 4, 2018

fipped commented Apr 11, 2018

coocood Apr 12, 2018

coocood Apr 12, 2018

coocood Apr 12, 2018

coocood commented Apr 13, 2018

stats: dynamically update the average column size #6170

stats: dynamically update the average column size #6170

Conversation

fipped commented Mar 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shenli commented Apr 2, 2018

shenli commented Apr 2, 2018

Choose a reason for hiding this comment

alivxxx left a comment

Choose a reason for hiding this comment

alivxxx commented Apr 4, 2018

fipped commented Apr 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coocood commented Apr 13, 2018

fipped commented Mar 28, 2018 •

edited

Loading