drainer/: Change unreasonable buff size and reduce memory usage

[july2993]

What problem does this PR solve?

the maxBinlogItemCount var will affect the job buffer size in an unreasonable way.

In syncer.sync will work like this

// where will be worker-count goroutine to run this
for {
    get job from "jobChan chan *job"
    if reach some condition {
          execute the pending job(execute sql)
     } else {
         continue
     }
}

note execute the sql may take times, and not consume the job chan.
so the upstream may block at add job into the chan, and also will not add job the other sync job chan.

What is changed and how it works?

avoid any buffer at upstream to much mem and not decrease performance, and don't make buffer size of job chan depent on maxBinlogItemCount

the flowing v1 means f9e4589
v2 means one more commit 3074b3b

before 10:53 version: v2.1.16 maxBinlogItemCount=16
start at 10:53 version: v1 maxBinlogItemCount=16
start at 30 11:17 version:v2.1.16 maxBinlogItemCount=64k (the previous default value)
start at 30 11:47 version: v1 maxBinlogItemCount=0

all using worker-count = 256, txn-batch = 20
note when the drainer restarts, the first 5 minutes will use safe-mode, the event metrics count the changed events.

Another test:

the upstream data one binlog near 60M

start at 19:25: version: v1
start at 19:45: version: v2

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

[july2993]

/run-all-tests

[july2993]

/run-all-tests

[july2993]

/run-all-tests

[ericsyh]

/run-all-tests

[IANTHEREAL]

lgtm

[WangXiangUSTC]

/run-all-tests

[zier-one]

why ci was failed?

[july2993]

why ci was failed?

i don't know, failed at status test, but not related to this pr.
(run ok locally and failed ci at another pr too-2.1)

[zier-one]

LGTM

[lichunzhu]

/run-integration-test

[lichunzhu]

IMO, CI fails because drainer hasn't consumed the pump 8250's storaged binlogs in 15 seconds when closing pump. This problem can be reproduced on my server.

[july2993]

IMO, CI fails because drainer hasn't consumed the pump 8250's storaged binlogs in 15 seconds when closing pump. This problem can be reproduced on my server.

Is it something relate to this pr change? can you fix this?

[lichunzhu]

IMO, CI fails because drainer hasn't consumed the pump 8250's storaged binlogs in 15 seconds when closing pump. This problem can be reproduced on my server.

Is it something relate to this pr change? can you fix this?

I'm not so sure now. I'm working on that now and later I will get a conclusion.

[july2993]

/run-all-tests

[lichunzhu]

/run-all-tests

[july2993]

/run-all-tests

[lichunzhu]

There is still 4s gap between drainer and pump after 15s. I guess when drainer has cache it can cache some binlogs from pump then drainer can still sort and execute the cached binlogs even though this pump is paused. Maybe binlogChanSize shouldn't be set to 0. Or we can set check time in check_status more bigger.

[july2993]

/run-all-tests

[july2993]

/run-all-tests

[july2993]

/run-all-tests

[july2993]

/run-all-tests

[july2993]

@lichunzhu PTAL, the root cause is when runs into the status test, it will start with commit-ts = 0 and fetch data from the oldest point.

[lichunzhu]

LGTM