*: tidb tracing prototype #7016

zhexuany · 2018-07-09T06:17:50Z

Thank you for working on TiDB! Please read TiDB's CONTRIBUTING document BEFORE filing this PR.

What have you changed? (mandatory)

Please explain IN DETAIL what the changes are in this PR and why they are needed:

Syntax support was already covered in another PR. In this PR, I actually added the functionality by taking advantage of opentracing. A detailed design doc will be release in a couple of days.

make it work
import basictracer to our own tracer implementation to avoid import gogogen
adding tracer test

What are the type of the changes (mandatory)?

New feature.

How has this PR been tested (mandatory)?

Please describe the tests that you ran to verify your changes. Have you finished unit tests, integration tests, or manual tests?

a tracer test is needed and I will add shortly.

Does this PR affect documentation (docs/docs-cn) update? (optional)

If there is document change, please file a PR in (docs and docs-cn) and add the PR number here.

Yes. It will affect the way how ops engineer inverstigate slow query. I will file anothr PR to update.

Refer to a related PR or issue link (optional)

Benchmark result if necessary (optional)

I have not benchmakred my changes but I will add one later

Add a few positive/negative examples (optional)

Suppose you already have tpch data present in your database, runing the following command will give a tace.

 trace select l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extendedprice * (1 - l_discount)) as sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discount) as avg_disc, count(*) as count_order from lineitem where l_shipdate <= date_sub('1998-12-01', interval 108 day) group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus;

The output looks like the following:

+---------------------------+--------------+--------------------------------+------+---------------------------------------------------------------------------------------------------------------+
| timestamp                 | duration     | operation                      | pos  | log                                                                                                           |
+---------------------------+--------------+--------------------------------+------+---------------------------------------------------------------------------------------------------------------+
| 2018-07-09T14:09:48+08:00 | 10.811µs     | distsql_select                 |    0 | sending rpc:to [/Table 35, 35)                                                                                |
| 2018-07-09T14:09:48+08:00 | 12.324µs     | distsql_select                 |    0 | sending rpc:to [/Index 0, 0)                                                                                  |
| 2018-07-09T14:09:48+08:00 | 27.969µs     | distsql_select                 |    0 | event:finished sending rpc calls                                                                              |
| 2018-07-09T14:09:48+08:00 | 18.373µs     | cop_iterator_worker            |    1 | event:runing                                                                                                  |
| 2018-07-09T14:09:48+08:00 | 27.953µs     | cop_iterator_worker            |    1 | event:handling remain tasks                                                                                   |
| 2018-07-09T14:09:48+08:00 | 505.7641ms   | cop_iterator_worker            |    1 | event:[TIME_COP_PROCESS] resp_time:505.723081ms txn_start_ts:401373026897887232 region_id:3 store_addr:store1 |
| 2018-07-09T14:09:48+08:00 | 505.80708ms  | cop_iterator_task_sender       |    2 | event:finish running                                                                                          |
| 2018-07-09T14:09:48+08:00 | 3.795µs      | table_resule_handler           |    3 | event:finished                                                                                                |
| 2018-07-09T14:09:48+08:00 | 3.729µs      | table_resule_handler           |    4 | event:finished                                                                                                |
| 2018-07-09T14:09:48+08:00 | 505.138898ms | hash_agg_exec_partial_worker_0 |    8 | event:update partial result                                                                                   |
| 2018-07-09T14:09:48+08:00 | 505.14003ms  | hash_agg_exec_partial_worker_0 |    8 | event:iterating chunk's data                                                                                  |
| 2018-07-09T14:09:48+08:00 | 505.166254ms | hash_agg_exec_partial_worker_0 |    8 | event:shuffling interm data                                                                                   |
| 2018-07-09T14:09:48+08:00 | 25.625µs     | hash_agg_exec_final_worker_1   |    9 | event:consuming interm data                                                                                   |
| 2018-07-09T14:09:48+08:00 | 31.462µs     | hash_agg_exec_final_worker_0   |   10 | event:consuming interm data                                                                                   |
| 2018-07-09T14:09:48+08:00 | 97.623µs     | hash_agg_exec_final_worker_2   |   13 | event:consuming interm data                                                                                   |
| 2018-07-09T14:09:48+08:00 | 505.441517ms | projection_exec                |   14 | event:parallelExecute is done                                                                                 |
| 2018-07-09T14:09:48+08:00 | 506.110832ms | trace_exec                     |   15 | event:tracing completed                                                                                       |
| 2018-07-09T14:09:48+08:00 | 20.897µs     | hash_agg_exec_final_worker_3   |   16 | event:consuming interm data                                                                                   |
+---------------------------+--------------+--------------------------------+------+---------------------------------------------------------------------------------------------------------------+
18 rows in set (0.51 sec)

Duration is dependent with your data size. Scale of 0.05 was used to generate data and that is why it is kind of fast here.

ngaut · 2018-07-11T06:50:58Z

Good job.

zhexuany · 2018-07-16T06:27:49Z

Here is an overview of this PR and I sincerely wish this can help you understand how I implement this PR and speed the review process up.

Firstly I add a new executor TraceExecutor and the following is the snippet of it.

// TraceExec represents a root executor of trace query.
type TraceExec struct {
	baseExecutor
	// CollectedSpans collects all span during execution. Span is appended via
	// callback method which passes into tracer implementation.
	CollectedSpans []basictracer.RawSpan
	// exhausted being true means there is no more result.
	exhausted bool
	// plan is the real query plan and it is used for building real query's executor.
	plan plan.Plan
	// rootTrace represents root span which is father of all other span.
	rootTrace opentracing.Span
}

In addition, I add a buildTrace function which calls real build function later in a switch statement.

I create Open method which overrides basicExecutor's Open. Inside Open, I create a rootTrace via NewSnowball function in tracing package. I will talk about this later. But just remember, this will create a non-nil value span. That is being said. calling span.LogKV is safe since each span in our codebase is non-nil value.

You may wonder how span got injected into child executor. Well, there are two entry point. First one is in Open. Inside Open, it wraps rootTrace into ctx which is an instance of Context. At child executor's Open method, you can take advantage of ChildSpanFromContext from tracing package. Such method will create a child span of the span in ctx. If no span available in ctx, a nonop span will be returned. Another entry point is Next method. The process is similar.

You may want to know what is noop span and how could span being non-nil. Let me explain to you. noop is an abbreviation of no-operation. As its name suggests, it does nothing even you call any available method of it. It creates minimal overhead to the current system when tracing is not enable. non-nil is easy just return a noop span when there is no available span in ctx

winoros · 2018-07-16T06:30:58Z

If one sql runs 2 minutes, will the trace command run 2 minutes too?

zhexuany · 2018-07-16T06:34:42Z

Yes. tracing traces how query got executed.

tiancaiamao

PLEASE SPLIT THIS PR for reviewer

tiancaiamao · 2018-07-16T09:49:12Z

Gopkg.toml

@@ -89,3 +89,7 @@ required = ["github.com/golang/protobuf/jsonpb"]
  name = "github.com/sirupsen/logrus"


Make vendor change in another PR please.

tiancaiamao · 2018-07-16T09:49:33Z

distsql/distsql.go

@@ -16,16 +16,20 @@ package distsql
 import (
 	"time"



Remove empty line

tiancaiamao · 2018-07-16T09:51:50Z

distsql/distsql.go

 	if !sctx.GetSessionVars().EnableStreaming {
 		kvReq.Streaming = false
 	}
+
+	for _, key := range kvReq.KeyRanges {


Is there a session variable to control the behavior?
I don't like the performance degrade if the feature is not used.

length of KeyRangs is constant. The complexity is just O(1); hence performance degradation not need to consider.

length of KeyRangs is constant.

You sure?

By saying it is a constant, I mean it will not grow linearly. In general, even it is 1000 or much larger, it is still constant.

tiancaiamao · 2018-07-16T09:52:39Z

executor/aggregate.go

@@ -16,7 +16,10 @@ package executor
 import (
 	"sync"



tiancaiamao · 2018-07-16T09:53:40Z

executor/aggregate.go

@@ -64,6 +68,8 @@ type HashAggPartialWorker struct {
 	// chk stores the input data from child,
 	// and is reused by childExec and partial worker.
 	chk *chunk.Chunk
+
+	trace opentracing.Span


Could this kind of things be hidden in context.Context ?

it will be used across different methods. Wrapping into ctx does not resolve this. So, I add this trace as part of the struct.

IMHO, the trace related information should come with the channel message for a worker.

What do you mean by saying "come with the channel message for a worker"?

It will be really nice if you couple explains it more explicitly.

tiancaiamao · 2018-07-16T09:59:30Z

executor/builder.go

@@ -617,17 +619,6 @@ func (b *executorBuilder) buildDDL(v *plan.DDL) Executor {
 	return e
 }

-func (b *executorBuilder) buildExplain(v *plan.Explain) Executor {


Move code makes a reviewer waste more time to view a PR

Yup. Thanks for the kind and good advice. Already did this, please refer to #7065 and #7063.

XuHuaiyu · 2018-07-23T06:55:50Z

plan/trace.go

+	targetPlan, err := Optimize(b.ctx, trace.Stmt, b.is)
+	if err != nil {
+		// TODO: imporve this error handle once everything works
+		b.err = errors.New("fail to optimize")


This error message is really hard to read for users.

XuHuaiyu · 2018-07-23T07:33:10Z

distsql/distsql.go

+	for _, key := range kvReq.KeyRanges {
+		sTid, sIid, _, err := tablecodec.DecodeKeyHead(key.StartKey)
+		if err != nil {
+			return nil, err


s/ err/ errors.Trace(err)

I think add a String method of Key maybe better?

XuHuaiyu · 2018-07-23T07:33:17Z

distsql/distsql.go

+		}
+		eTid, eIid, _, err := tablecodec.DecodeKeyHead(key.EndKey)
+		if err != nil {
+			return nil, err


XuHuaiyu · 2018-07-23T07:35:46Z

distsql/distsql.go

@@ -54,6 +73,7 @@ func Select(ctx context.Context, sctx sessionctx.Context, kvReq *kv.Request, fie
 		}, nil
 	}

+	child.LogKV("event", "finished sending rpc calls")


We do not need this?

We need it because it can help us know when rpc is finished so we can understand how many millisecond is spent at sending rpc call.

XuHuaiyu · 2018-07-23T07:41:31Z

distsql/distsql.go

+		if err != nil {
+			return nil, err
+		}
+		child.LogKV("sending rpc", fmt.Sprintf("to [/Table/%d/%d, %d/%d)", sTid, sIid, eTid, eIid))


why make the string format like this?

It tells us which table id and index id that this query may touch.

XuHuaiyu · 2018-07-23T08:07:44Z

executor/trace.go

+	chk.Reset()
+	for !e.exhausted {
+		var timeVal string
+		var baseIdx int64


this var can be removed?

executor/trace.go

XuHuaiyu · 2018-07-23T08:09:40Z

plan/trace.go

+
+	StmtPlan Plan
+
+	ChildSchema expression.Schema


Useless attribute.

XuHuaiyu · 2018-07-23T08:16:42Z

plan/trace.go

+		b.err = errors.New("fail to optimize")
+		return nil
+	}
+	pp, _ := targetPlan.(PhysicalPlan)


We do not need this.

XuHuaiyu · 2018-07-23T08:19:07Z

util/tracing/tracer.go

+)
+
+// TiDBTrace is set as Baggage on traces which are used for snowball tracing.
+const TiDBTrace = "tr"


why not "trace"?

the shorter the better.

tiancaiamao · 2018-07-24T08:35:29Z

distsql/distsql.go

 	if !sctx.GetSessionVars().EnableStreaming {
 		kvReq.Streaming = false
 	}
+
+	for _, key := range kvReq.KeyRanges {


length of KeyRangs is constant.

You sure?

tiancaiamao · 2018-07-24T08:43:22Z

executor/aggregate.go

@@ -64,6 +68,8 @@ type HashAggPartialWorker struct {
 	// chk stores the input data from child,
 	// and is reused by childExec and partial worker.
 	chk *chunk.Chunk
+
+	trace opentracing.Span


IMHO, the trace related information should come with the channel message for a worker.

tiancaiamao · 2018-07-24T08:44:13Z

executor/aggregate.go

@@ -351,12 +359,14 @@ func (w *HashAggPartialWorker) run(ctx sessionctx.Context, waitGroup *sync.WaitG
 		if needShuffle {
 			w.shuffleIntermData(sc, finalConcurrency)
 		}
+		w.trace.Finish()


Will w.trace be nil ?

Nope. All trace is at least a noop trace.

tiancaiamao · 2018-07-24T08:51:16Z

executor/trace.go

+	childrenResults []*chunk.Chunk
+}
+
+// buildTrace builds a TraceExec for future executing. This func will be called


s/func/function

tiancaiamao · 2018-07-24T08:53:26Z

executor/trace.go

+	e.children = make([]Executor, 0, len(pp.Children()))
+	for _, child := range pp.Children() {
+		switch p := child.(type) {
+		case *plan.PhysicalTableReader:


What will happen if we explain trace select ... ?

tiancaiamao · 2018-07-24T08:54:50Z

executor/trace.go

+// Open opens a trace executor and it will create a root snowball span which will be
+// used for the follwoing span in a relationship of `ChildOf` or `FollowFrom`.
+func (e *TraceExec) Open(ctx context.Context) error {
+	e.rootTrace = tracing.NewRecordedTrace("trace_exec", func(sp basictracer.RawSpan) {


After that, global opentracing Tracer is modified?

Yes. I will file another PR to better handle this.

tiancaiamao · 2018-07-24T08:58:04Z

executor/trace.go

+	})
+	ctx = opentracing.ContextWithSpan(ctx, e.rootTrace)
+	for _, child := range e.children {
+		err := child.Open(ctx)


Will some node open multiple times?
The children node open may also call its children's open ? If something like indexReader open multiple time, would there be resource leak?

Usually, it is not. I just copied the code from basicExecutor with some addition about how to set up tracing.

coocood · 2018-07-27T09:49:32Z

@zhexuany
A tree format like the current explain result would be nice.

coocood · 2018-08-09T04:13:53Z

distsql/distsql.go

 	if !sctx.GetSessionVars().EnableStreaming {
 		kvReq.Streaming = false
 	}
+
+	for _, key := range kvReq.KeyRanges {


We don't need to trace every key range.
In index lookup request, there can be thousands of ranges.
This has too much cost for index lookup requests.

zhexuany · 2018-08-29T15:28:35Z

@coocood

Need a test case in the executor package.

Not easy to do so since we remove all tracing.ChildSpanFromContext. Maybe a way is to create dummy executor and then create child span in such executor's Next.

…o tidb_tracing_prototype

coocood · 2018-08-29T16:25:18Z

@zhexuany
Currently, the select statement is optimized before trace executes, we can not trace optimization time.
If we do query optimization in TraceExec, then we can trace the optimization time.
And you can write a test that has trace results.

zhexuany · 2018-08-30T06:07:48Z

@coocood Good point, I will do it in another PR.

coocood · 2018-08-30T08:26:21Z

LGTM

zz-jason · 2018-08-30T08:38:44Z

plan/trace.go

+	retFields := []string{"operation", "duration", "spanID"}
+	schema := expression.NewSchema(make([]*expression.Column, 0, len(retFields))...)
+	schema.Append(buildColumn("", "operation", mysql.TypeString, mysql.MaxBlobWidth))
+


remove this empty line?

zz-jason · 2018-08-30T08:38:59Z

plan/trace.go

+		return nil, errors.New("trace only supports select query")
+	}
+
+	optimizedP, err := Optimize(b.ctx, trace.Stmt, b.is)


s/optimizedP/stmtPlan/?

zz-jason · 2018-08-30T08:40:26Z

plan/trace.go

+// rows result.
+func (b *planBuilder) buildTrace(trace *ast.TraceStmt) (Plan, error) {
+	if _, ok := trace.Stmt.(*ast.SelectStmt); !ok {
+		return nil, errors.New("trace only supports select query")


how about "only the select statement can be traced"?

zz-jason · 2018-08-30T08:41:01Z

executor/trace_test.go

+type testTraceExec struct{}
+
+func (s *testTraceExec) SetupSuite(c *C) {
+


remove this empty line?

zz-jason · 2018-08-30T08:46:20Z

executor/trace_test.go

+
+}
+
+func (s *testSuite) TestTraceExec(c *C) {


Seems this test file does nothing, can we remove the whole file and add it when adding ut.
And I think it's better to add these integration test in mysql-test, not in the ut.

Yes. I leave a todo and I will soon add the actual test in another PR.

zz-jason · 2018-08-30T08:50:55Z

executor/trace.go

+
+// buildTrace builds a TraceExec for future executing. This method will be called
+// at build().
+func (b *executorBuilder) buildTrace(v *plan.Trace) Executor {


It's better to maintain this function in the "builder.go" file, like other functions.

zz-jason · 2018-08-30T08:52:18Z

executor/trace.go

+	e.children = make([]Executor, 0, len(pp.Children()))
+	for _, child := range pp.Children() {
+		switch p := child.(type) {
+		case *plan.PhysicalTableReader, *plan.PhysicalIndexReader, *plan.PhysicalIndexLookUpReader, *plan.PhysicalHashAgg, *plan.PhysicalProjection, *plan.PhysicalStreamAgg, *plan.PhysicalSort:


please split the long code line.

seems these operators are also not supported in this PR?

executor/trace.go

zz-jason · 2018-08-30T08:56:51Z

executor/trace.go

+// Open opens a trace executor and it will create a root trace span which will be
+// used for the following span in a relationship of `ChildOf` or `FollowFrom`.
+// for more details, you could refer to http://opentracing.io
+func (e *TraceExec) Open(ctx context.Context) error {


seems that we can not trace the building and optimization time of a query, only the execution time can be traced?

It is only for now. I will file another PR to support this.

Since the TraceExec has no child, I think we can just leave this function to be empty.

zz-jason · 2018-08-30T16:53:55Z

executor/builder.go

+func (b *executorBuilder) buildTrace(v *plan.Trace) Executor {
+	return &TraceExec{
+		baseExecutor: newBaseExecutor(b.ctx, v.Schema(), v.ExplainID()),
+		stmtNode:     v.StmtNode,


can we store the plain text of the select statement to be traced? We can trace the time spent by parser with the help of this.

zz-jason · 2018-08-30T16:55:23Z

executor/trace.go

+// Open opens a trace executor and it will create a root trace span which will be
+// used for the following span in a relationship of `ChildOf` or `FollowFrom`.
+// for more details, you could refer to http://opentracing.io
+func (e *TraceExec) Open(ctx context.Context) error {


Since the TraceExec has no child, I think we can just leave this function to be empty.

zz-jason · 2018-08-30T18:00:05Z

executor/trace.go

+	}
+
+	// append select executor to trace executor
+	e.children = append(e.children, e.builder.build(pp))


We'd better use another variable to store the executor, for example e.stmtExec.

zz-jason · 2018-08-30T18:00:35Z

executor/trace.go

+	})
+	// we actually don't care when underlying executor started. We only care how
+	// much time was spent
+	for _, child := range e.children {


only need to call e.stmtExec.Open()

zz-jason · 2018-08-30T18:01:33Z

executor/trace.go

+			return errors.Trace(err)
+		}
+	}
+	e.childrenResults = make([]*chunk.Chunk, 0, len(e.children))


only one Chunk is needed for e.stmtExxec

zz-jason · 2018-08-30T18:01:52Z

executor/trace.go

+	// store span into context
+	ctx = opentracing.ContextWithSpan(ctx, e.rootTrace)
+
+	if len(e.children) > 0 {


can be removed.

zz-jason · 2018-08-30T18:02:19Z

executor/trace.go

+			if err := e.children[0].Next(ctx, e.childrenResults[0]); err != nil {
+				return errors.Trace(err)
+			}
+			if e.childrenResults[0].NumRows() != 0 {


e.childrenResults[0].NumRows() == 0

executor/trace.go

zz-jason

LGTM

zz-jason · 2018-08-30T18:22:56Z

/run-all-tests

zhexuany added the status/WIP label Jul 9, 2018

zhexuany requested review from tiancaiamao and zz-jason July 16, 2018 06:27

zhexuany added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed status/WIP labels Jul 16, 2018

tiancaiamao reviewed Jul 16, 2018

View reviewed changes

This was referenced Jul 16, 2018

util/tracing: adding tracing package #7063

Closed

executor: move tidb_reader code to its named files. #7065

Merged

vendor:update vendor for tracing #7067

Merged

zhexuany force-pushed the tidb_tracing_prototype branch 6 times, most recently from 3202bc5 to 1f7bfda Compare July 23, 2018 06:20

XuHuaiyu reviewed Jul 23, 2018

View reviewed changes

tiancaiamao reviewed Jul 24, 2018

View reviewed changes

zhexuany force-pushed the tidb_tracing_prototype branch 2 times, most recently from 371f35f to 96f0e8a Compare July 27, 2018 10:35

coocood reviewed Aug 9, 2018

View reviewed changes

zhexuany force-pushed the tidb_tracing_prototype branch 2 times, most recently from c4295c1 to 4889004 Compare August 15, 2018 02:49

tidb tracing prototype

a8821c8

zhexuany force-pushed the tidb_tracing_prototype branch from 83570ea to a8821c8 Compare August 16, 2018 05:53

add benchmark for noop

92d1764

zhexuany added the status/all tests passed label Aug 29, 2018

zhexuany added 2 commits August 29, 2018 23:30

address comments

70f0d9b

Merge branch 'tidb_tracing_prototype' of github.com:zhexuany/tidb int…

6181ca2

…o tidb_tracing_prototype

zhexuany added 2 commits August 30, 2018 11:02

remove unecessary import

cb366aa

add more tests

46e681e

zhexuany added 2 commits August 30, 2018 14:08

add a todo

e0d7cba

Merge branch 'master' into tidb_tracing_prototype

7657e5e

zz-jason reviewed Aug 30, 2018

View reviewed changes

zhexuany added 5 commits August 30, 2018 17:17

address comments

227626b

remvoe extra line

03c3f7a

move buildTrace to builder.go

507b567

remove extra imports from trace.go

0ccae95

record optimize time in traceExec's Next

e2cb6ff

zz-jason reviewed Aug 30, 2018

View reviewed changes

zhexuany added 2 commits August 31, 2018 01:03

open child in next rather in open

9f5b5bf

fix optimize logic mistake

e49d58a

zz-jason reviewed Aug 30, 2018

View reviewed changes

address comment

da0e06b

zz-jason reviewed Aug 30, 2018

View reviewed changes

executor/trace.go Outdated Show resolved Hide resolved

use == not !=

2c6aa09

zz-jason approved these changes Aug 30, 2018

View reviewed changes

Merge branch 'master' into tidb_tracing_prototype

1468f03

zhexuany merged commit 341dc10 into pingcap:master Aug 30, 2018

zhexuany deleted the tidb_tracing_prototype branch August 30, 2018 21:25

ngaut mentioned this pull request Aug 31, 2018

We need a tutorial on how to use tidb tracing #7565

Closed

		@@ -89,3 +89,7 @@ required = ["github.com/golang/protobuf/jsonpb"]
		name = "github.com/sirupsen/logrus"

		type testTraceExec struct{}

		func (s testTraceExec) SetupSuite(c C) {

*: tidb tracing prototype #7016

*: tidb tracing prototype #7016

Conversation

zhexuany commented Jul 9, 2018 • edited Loading

What have you changed? (mandatory)

What are the type of the changes (mandatory)?

How has this PR been tested (mandatory)?

Does this PR affect documentation (docs/docs-cn) update? (optional)

Refer to a related PR or issue link (optional)

Benchmark result if necessary (optional)

Add a few positive/negative examples (optional)

ngaut commented Jul 11, 2018

zhexuany commented Jul 16, 2018

winoros commented Jul 16, 2018

zhexuany commented Jul 16, 2018

tiancaiamao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhexuany Jul 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coocood commented Jul 27, 2018

Choose a reason for hiding this comment

zhexuany commented Aug 29, 2018

coocood commented Aug 29, 2018

zhexuany commented Aug 30, 2018

coocood commented Aug 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zz-jason left a comment

zhexuany commented Jul 9, 2018 •

edited

Loading

zhexuany Jul 24, 2018 •

edited

Loading