*: cut off duration.fsp in chunk #7043

lysu · 2018-07-12T14:40:49Z

What have you changed? (mandatory)

Remove fsp of types.Duration from Chunk to reduce memory in chunk
Fix fix time func ret decimal to fix decimals flag in timediff response is incorrect #6968 via 1e12b0e069cca8a8d12a8fcd43b24bd7fec8f167

This PR is "easy way" to #7013 which is more right and more clear in theory, but with big change and be nightmare to both reviewee and reviewer, but I still think it must be done in future.

Datum's collation, decimal, length is short-cut for some part coding but wrong for design.
Duration's fsp need be remove with Datum 3 field which need big change Decimal
Seperate type infer flow with data flow is best.

But this time, we solve chunk in this small PR first.

What is the type of the changes? (mandatory)

Improvement (non-breaking change which is an improvement to an existing feature)
Bug fix (non-breaking change which fixes an issue)

How has this PR been tested? (mandatory)

unit tests
integration tests
and I will Port duration mysql_test for test so WIP but ready for review

This change is

lysu · 2018-07-12T14:42:29Z

/run-all-tests

zz-jason · 2018-07-12T15:09:48Z

util/chunk/row.go

@@ -88,7 +89,8 @@ func (r Row) GetTime(colIdx int) types.Time {
 // GetDuration returns the Duration value with the colIdx.
 func (r Row) GetDuration(colIdx int) types.Duration {


Does this change affect the benchmark of GetDuration?

I will benchmark later, but it seems will allocate more a types.Duration than before, but this is just as GetTime does...

zz-jason · 2018-07-12T15:14:17Z

expression/builtin_time.go

+	if err != nil {
+		return nil, err
+	}
+	bf.tp.Decimal = mathutil.Max(arg0Dec, arg1Dec)


this is a bugfix?

this decimal field is no use before this PR, so it got wrong value but not affect the user, but now if the duration comes from chunk.row will need it to compensate missed fsp info.

zz-jason · 2018-07-12T15:17:21Z

expression/integration_test.go

@@ -1307,13 +1307,13 @@ func (s *testIntegrationSuite) TestTimeBuiltin(c *C) {

 	// test time
 	result = tk.MustQuery("select time('2003-12-31 01:02:03')")
-	result.Check(testkit.Rows("01:02:03"))
+	result.Check(testkit.Rows("01:02:03.000000"))


this result is different from mysql:

MySQL(localhost:3306) > select time('2003-12-31 01:02:03'); +-----------------------------+ | time('2003-12-31 01:02:03') | +-----------------------------+ | 01:02:03 | +-----------------------------+ 1 row in set (0.00 sec)

zz-jason · 2018-07-12T15:17:34Z

expression/integration_test.go

 	result = tk.MustQuery("select time('2003-12-31 01:02:03.000123')")
 	result.Check(testkit.Rows("01:02:03.000123"))
 	result = tk.MustQuery("select time('01:02:03.000123')")
 	result.Check(testkit.Rows("01:02:03.000123"))
 	result = tk.MustQuery("select time('01:02:03')")
-	result.Check(testkit.Rows("01:02:03"))
+	result.Check(testkit.Rows("01:02:03.000000"))


lysu · 2018-07-13T13:49:37Z

There are still some truncate test cases give the different result to MySQL(even in master branch), https://github.com/pingcap/tidb-test/pull/574/commits/21411bae6f795c51b152383dcd2dd06d4e3c94ea

zz-jason · 2018-07-13T14:07:48Z

LGTM

But we still need some benchmarks to know the exact performance regression.

XuHuaiyu · 2018-07-16T07:47:15Z

types/etc.go

 		return true
 	}
 	return false
 }

+// IsStr returns a boolean indicating
+// whether the field type is a string type.
+func IsStr(ft *FieldType) bool {


s/ IsStr/ IsString

XuHuaiyu · 2018-07-16T07:48:38Z

util/chunk/row.go

@@ -20,6 +20,7 @@ import (
 	"github.com/pingcap/tidb/types"
 	"github.com/pingcap/tidb/types/json"
 	"github.com/pingcap/tidb/util/hack"
+	"time"


put this line to line17

XuHuaiyu · 2018-07-16T07:52:04Z

expression/builtin_time.go

@@ -5370,3 +5401,19 @@ func (b *builtinLastDaySig) evalTime(row types.Row) (types.Time, bool, error) {
 	}
 	return ret, false, nil
 }
+
+func timePrecision(ctx sessionctx.Context, expression Expression) (int, error) {


add a comment for this function.

name is not very suitable, it was renamed and add comments in new commit.

XuHuaiyu · 2018-07-16T07:53:08Z

expression/builtin_time.go

+		if isNil || err != nil {
+			return 0, errors.Trace(err)
+		}
+		if n := strings.LastIndexByte(str, '.'); n >= 0 {


This check can be replaced by function GetFsp

XuHuaiyu · 2018-07-16T07:54:59Z

expression/integration_test.go

@@ -2663,7 +2663,7 @@ func (s *testIntegrationSuite) TestCompareBuiltin(c *C) {
 	tk.MustExec(`insert into t2 values(1, 1.1, "2017-08-01 12:01:01", "12:01:01", "abcdef", 0b10101)`)

 	result = tk.MustQuery("select coalesce(NULL, a), coalesce(NULL, b, a), coalesce(c, NULL, a, b), coalesce(d, NULL), coalesce(d, c), coalesce(NULL, NULL, e, 1), coalesce(f), coalesce(1, a, b, c, d, e, f) from t2")
-	result.Check(testkit.Rows(fmt.Sprintf("1 1.1 2017-08-01 12:01:01 12:01:01 %s 12:01:01 abcdef 21 1", time.Now().In(tk.Se.GetSessionVars().GetTimeZone()).Format("2006-01-02"))))
+	result.Check(testkit.Rows(fmt.Sprintf("1 1.1 2017-08-01 12:01:01 12:01:01.000000 %s 12:01:01 abcdef 21 1", time.Now().In(tk.Se.GetSessionVars().GetTimeZone()).Format("2006-01-02"))))


Is this compatible with mysql?

oh~ that's bug...fixed in new commits

XuHuaiyu · 2018-07-16T07:55:56Z

types/fsp.go

@@ -22,6 +22,8 @@ import (
 )

 const (
+	// WaitFillFsp is fsp need fill by caller if they want to use fsp.
+	WaitFillFsp = -2


When will the caller want to us fsp?

the old way needs GetDuration caller to check whether result.fsp == WaitFillFsp than refill Fsp or do nothing...that's buggy.

Now force pass fillFsp paramter in GetDuration(also add some comments) and no longer need this flag. PTAL~thx

lysu · 2018-07-16T14:18:18Z

a1fb7386ce5c3fe36bc9233f96c93eea1c34ec02 fix https://github.com/pingcap/tidb-test/pull/574/commits/cbeabfbf874b5ed3438c02b70c3066ea5fc64264 and https://github.com/pingcap/tidb-test/pull/574/commits/d53c88580fc5f450a8cf8076aff9593c5cda36bd

XuHuaiyu · 2018-07-17T03:00:11Z

@lysu DNM label can be removed?

XuHuaiyu · 2018-07-17T03:01:12Z

expression/builtin_time.go

@@ -5370,3 +5401,16 @@ func (b *builtinLastDaySig) evalTime(row types.Row) (types.Time, bool, error) {
 	}
 	return ret, false, nil
 }
+
+// expressionFsp calculates the fsp from given expression.
+func expressionFsp(ctx sessionctx.Context, expression Expression) (int, error) {


s/ expressionFsp/ getExpressionFsp

XuHuaiyu · 2018-07-17T03:02:08Z

expression/typeinfer_test.go

@@ -1249,7 +1249,7 @@ func (s *testInferTypeSuite) createTestCase4TimeFuncs() []typeInferTestCase {
 		{"addtime(c_timestamp, c_time_d)", mysql.TypeDatetime, charset.CharsetBin, mysql.BinaryFlag, 26, 4},
 		{"addtime(c_timestamp_d, c_time_d)", mysql.TypeDatetime, charset.CharsetBin, mysql.BinaryFlag, 26, 0},
 		{"addtime(c_time, c_time)", mysql.TypeDuration, charset.CharsetBin, mysql.BinaryFlag, 15, 3},
-		{"addtime(c_time_d, c_time)", mysql.TypeDuration, charset.CharsetBin, mysql.BinaryFlag, 15, 0},
+		{"addtime(c_time_d, c_time)", mysql.TypeDuration, charset.CharsetBin, mysql.BinaryFlag, 15, 3},


Do the changes in this file be more reasonable or more compatible?

Yes, in previous we didn't infer decimal and always be zero, but in MySQL takes that.

mysql> select version(); +---------------------------+ | version() | +---------------------------+ | 5.7.22-0ubuntu18.04.1-log | +---------------------------+ 1 row in set (0.00 sec) mysql> select addtime(c_time_d, c_time) from t; Field 1: `addtime(c_time_d, c_time)` Catalog: `def` Database: `` Table: `` Org_table: `` Type: TIME Collation: binary (63) Length: 14 Max_length: 0 Decimals: 3 Flags: BINARY 0 rows in set (0.00 sec)

XuHuaiyu · 2018-07-17T03:04:44Z

types/row.go

@@ -42,7 +42,9 @@ type Row interface {
 	GetTime(colIdx int) Time

 	// GetDuration returns the Duration value with the colIdx.
-	GetDuration(colIdx int) Duration
+	// fillFsp is needed for refill fsp info if duration came from chunk.Row which is no longer store fsp info.
+	// if caller make sure that data from Datum or only use Duration.Duration properties can pass 0 as fillFsp


s/ is/ If
add . at the end of this comment

XuHuaiyu · 2018-07-17T03:04:46Z

types/row.go

@@ -42,7 +42,9 @@ type Row interface {
 	GetTime(colIdx int) Time

 	// GetDuration returns the Duration value with the colIdx.
-	GetDuration(colIdx int) Duration
+	// fillFsp is needed for refill fsp info if duration came from chunk.Row which is no longer store fsp info.
+	// if caller make sure that data from Datum or only use Duration.Duration properties can pass 0 as fillFsp


s/ is/ If
add . at the end of this comment

XuHuaiyu · 2018-07-17T03:05:15Z

types/time.go

@@ -812,7 +813,7 @@ func (d Duration) String() string {
 	}

 	fmt.Fprintf(&buf, "%02d:%02d:%02d", hours, minutes, seconds)
-	if d.Fsp > 0 {
+	if d.Fsp > 0 && fraction >= 0 {


add a comment for this check

check fraction seems useless....I removed it

XuHuaiyu · 2018-07-17T03:12:40Z

types/datum.go

+		if err != nil {
+			return ret, errors.Trace(err)
+		}
+		if timeNum > MaxDuration && timeNum < 10000000000 {


what if timeNum > 10000000000

timeNum > MaxDuration and < 10000000000 must be a wrong value..
but timeNum > 10000000000 will try treat as 'YYYYMMDDHHmmss'

specially timeNum < 0 and timeNum < -10000000000 should NOT treat as 'YYYYMMDDHHmmss'

https://github.com/mysql/mysql-server/blob/e7586bb9c11953a81c816ca10708f612403edc24/sql-common/my_time.c#L846

I add some comments for this if~

XuHuaiyu · 2018-07-17T03:14:08Z

expression/builtin_time.go

+
+	arg0Dec, err := expressionFsp(ctx, args[0])
+	if err != nil {
+		return nil, err


errors.Trace(err)

and line 411

lysu · 2018-07-17T05:04:50Z

/run-all-tests tidb-test=pr/574

coocood · 2018-07-17T05:53:51Z

LGTM

lysu · 2018-07-17T06:04:10Z

@zz-jason

func BenchmarkGetDurationFromChunk(b *testing.B) {
	chk := chunk.NewChunkWithCapacity([]*types.FieldType{{Tp: mysql.TypeDuration}}, 1)
	chk.AppendDuration(0, types.Duration{Duration: 11233*time.Second, Fsp:1})
	row := chk.GetRow(0)
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		x: = row.GetDuration(0, 1)
                x = x
	}
}

for master:

BenchmarkGetDurationFromChunk-8   	2000000000	         1.06 ns/op	       0 B/op	       0 allocs/op
PASS

for this pr:

BenchmarkGetDurationFromChunk-8   	2000000000	         1.03 ns/op	       0 B/op	       0 allocs/op
PASS

XuHuaiyu

rest LGTM

XuHuaiyu · 2018-07-17T06:12:31Z

types/datum.go

+		if err != nil {
+			return ret, errors.Trace(err)
+		}
+		// For For huge numbers(>'0001-00-00 00-00-00') try full DATETIME in ParseDuration.


remove one For

fix #6968

XuHuaiyu

LGTM

lysu added the status/WIP label Jul 12, 2018

lysu changed the title *: cut off Duration.Fsp in chunk *: cut off duration.fsp in chunk Jul 12, 2018

zz-jason reviewed Jul 12, 2018

View reviewed changes

zz-jason added type/enhancement The issue or PR belongs to an enhancement. status/LGT1 Indicates that a PR has LGTM 1. sig/execution SIG execution and removed status/WIP labels Jul 13, 2018

lysu added the status/DNM label Jul 14, 2018

XuHuaiyu reviewed Jul 16, 2018

View reviewed changes

XuHuaiyu reviewed Jul 17, 2018

View reviewed changes

lysu added status/all tests passed and removed status/DNM labels Jul 17, 2018

XuHuaiyu reviewed Jul 17, 2018

View reviewed changes

lysu added 9 commits July 17, 2018 14:17

expression: fix time func ret decimal.

cae4b8f

fix #6968

*: make arrow no longer store fsp for duration

1ad63de

*: fix unit test.

0bd55ad

*: fix panic?

3ede654

*: resolve rebase question.

475a1e0

fix ci.

b37a89c

types: handle negative duration and overflow check.

f9c5b2f

types: fix coalesce duration type infer.

f42276d

address comment

43f271e

lysu added 4 commits July 17, 2018 14:17

address comment, remove WaitFillFsp

fe04133

fix rebase mistake

12250ae

address comments

78de578

address comments

0532bbb

XuHuaiyu approved these changes Jul 17, 2018

View reviewed changes

coocood added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 17, 2018

coocood merged commit 9cf670a into pingcap:master Jul 17, 2018

lysu deleted the dev/duration_cut_off_fsp_with_easy_way branch September 27, 2018 04:16

XuHuaiyu mentioned this pull request Dec 19, 2018

util/chunk: fix incorrect result when set duration to MutRow #8725

Merged

breezewish mentioned this pull request Mar 17, 2019

Reduce codec::mysql::Duration memory size tikv/tikv#4025

Closed

zz-jason mentioned this pull request Jun 10, 2019

Reduce the size of Duration to 8 bytes tikv/tikv#4858

Merged

		@@ -88,7 +89,8 @@ func (r Row) GetTime(colIdx int) types.Time {
		// GetDuration returns the Duration value with the colIdx.
		func (r Row) GetDuration(colIdx int) types.Duration {

*: cut off duration.fsp in chunk #7043

*: cut off duration.fsp in chunk #7043

Conversation

lysu commented Jul 12, 2018 • edited Loading

What have you changed? (mandatory)

What is the type of the changes? (mandatory)

How has this PR been tested? (mandatory)

lysu commented Jul 12, 2018

Choose a reason for hiding this comment

lysu Jul 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lysu commented Jul 13, 2018

zz-jason commented Jul 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lysu Jul 16, 2018 • edited Loading

Choose a reason for hiding this comment

lysu commented Jul 16, 2018

XuHuaiyu commented Jul 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lysu Jul 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lysu commented Jul 17, 2018

coocood commented Jul 17, 2018

lysu commented Jul 17, 2018

XuHuaiyu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XuHuaiyu left a comment

Choose a reason for hiding this comment

lysu commented Jul 12, 2018 •

edited

Loading

lysu Jul 13, 2018 •

edited

Loading

lysu Jul 16, 2018 •

edited

Loading

lysu Jul 17, 2018 •

edited

Loading