*: make load data atomic by default

[tina77fritz]

Signed-off-by: Tina Fritz tina77fritz@gmail.com

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:
load data uses several transactions by default which will break transaction atomic property.

What is changed and how it works?

Proposal: xxx

What's Changed:

How it Works:
If tidb_dml_batch_size = 0, load data will atomic and make it default.

Related changes

Check List

Tests

• Unit test
• Manual test

1. load a large file
2. see the log of the server

[2020/07/27 23:07:59.160 +08:00] [INFO] [server.go:388] ["new connection"] [conn=5] [remoteAddr=127.0.0.1:46754]
[2020/07/27 23:08:52.888 +08:00] [INFO] [2pc.go:285] ["[BIG_TXN]"] [con=5] ["table ID"=50] [size=3921322] [keys=135218] [puts=135218] [dels=0] [locks=0] [checks=0] [txnStartTS=418345787592015872]
[2020/07/27 23:08:52.892 +08:00] [INFO] [2pc.go:394] ["2PC detect large amount of mutations on a single region"] [region=44] ["mutations count"=135218]
[2020/07/27 23:08:53.022 +08:00] [INFO] [2pc.go:394] ["2PC detect large amount of mutations on a single region"] [region=49] ["mutations count"=135218]
[2020/07/27 23:08:53.029 +08:00] [INFO] [load_data.go:255] ["commit one task success"] [conn=5] ["commit time usage"=901.4976ms] ["keys processed"=135218] ["tasks processed"=1] ["tasks in queue"=0]

Side effects

• Performance regression
  • Consumes more MEM

Release note

• No release note

[tina77fritz]

/run-check_dev_2

[cfzjywxk]

Maybe it's easy to use much memory loading a lot of data, I think we'd better make it atomic after the memory improvement task #17479. @lysu @jackysp @imtbkcat What do you think?

[tina77fritz]

Maybe it's easy to use much memory loading a lot of data, I think we'd better make it atomic after the memory improvement task #17479. @lysu @jackysp @imtbkcat What do you think?

emm... it will reach the transaction size limit if it is really a large transaction. The user will make a choice himself.

1. enlarge the transaction size limit
2. set tidb_dml_batch_size = xxx

[cfzjywxk]

Maybe it's easy to use much memory loading a lot of data, I think we'd better make it atomic after the memory improvement task #17479. @lysu @jackysp @imtbkcat What do you think?

emm... it will reach the transaction size limit if it is really a large transaction. The user will make a choice himself.

1. enlarge the transaction size limit
2. set tidb_dml_batch_size = xxx

There will be memory usage for the executor itself, for example the data processing and cache may use some memory, and then they will be written into the transaction memory buffer. We could do some loading tests to check memory usage like #15369. Could you help us with this check and the memory usage optimization, many thanks.

[tina77fritz]

Maybe it's easy to use much memory loading a lot of data, I think we'd better make it atomic after the memory improvement task #17479. @lysu @jackysp @imtbkcat What do you think?

emm... it will reach the transaction size limit if it is really a large transaction. The user will make a choice himself.

1. enlarge the transaction size limit
2. set tidb_dml_batch_size = xxx

There will be memory usage for the executor itself, for example the data processing and cache may use some memory, and then they will be written into the transaction memory buffer. We could do some loading tests to check memory usage like #15369. Could you help us with this check and the memory usage optimization, many thanks.

Maybe I have no time to optimize the memory usage for it :)
I just think it is terrible when a DBMS loses the atomic property of its transaction.

[jackysp]

LGTM

[tina77fritz]

I think the memory usage optimization from @bobotu has been finished. Could it continue? @cfzjywxk

[cfzjywxk]

I think the memory usage optimization from @bobotu has been finished. Could it continue? @cfzjywxk

Yes, I think after we change the default value to one transaction, we need to change the load logic from
"preparing all data well in memory, submit this task to commit goroutine and do the batch check and insert, then write into transaction buffer",
into
"load and prepare data and write them into transaction memory buffer at the same time".
Then we will not hold the rows content all in memory and increase the oom risk, and the large transition memory will be controlled and may spill to disk in the future.
Would you like to help us with this? Thanks a lot~

[cfzjywxk]

LGTM

[cfzjywxk]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

@tina77fritz merge failed.

[codecov]

Codecov Report

Merging #18807 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #18807   +/-   ##
===========================================
  Coverage   79.2776%   79.2776%           
===========================================
  Files           546        546           
  Lines        148632     148632           
===========================================
  Hits         117832     117832           
  Misses        21309      21309           
  Partials       9491       9491           

[bobotu]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

/run-all-tests

[jackysp]

/merge

[ti-srebot]

/run-all-tests