Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Multi Batch Write #154

Merged
merged 20 commits into from
Mar 19, 2020
Merged

Improve Multi Batch Write #154

merged 20 commits into from
Mar 19, 2020

Conversation

Little-Wallace
Copy link

This PR is to improve performance of MultiBatchWrite in some case which it may perform worse than PipelinedWrite.

If one thread just calls interface 'DBImpl::Write' and tries to insert into a large WriteBatch, it may block all write thread. So we must block thread after calling std::this_thread::yield for a period of time. And every thread should to its task if there is only one WriteBatch.

@Little-Wallace
add yield api
27663b7 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
fix some format
21e7732 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
refactor some code
0e97437 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
refactor queue
7b98963 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
remove yield api
bcb2a74 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
use move vector to avoid vector allocation
09730cc 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
fix warn
6fd72ed 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
fix comment
811e8a7 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace Little-Wallace added the component/rocksdb Component: RocksDB engine label Mar 16, 2020
@Little-Wallace
reduce run time of ut
ae58725 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
small change
6400197 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
test less write for pass ut
36a2f8f 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
test less write to pass ut
8263ba1 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
use single thread
5eeae76 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
rollback threads
bbe4c45 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
perform like normal pipelined write
42b391e 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
test more batch
aabfe03 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
yield more time
f0a3028 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace
fix bug which may case write hung
7c0ffad 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
yiwu-arbug
yiwu-arbug reviewed Mar 19, 2020
View reviewed changes
Copy link
Collaborator

@yiwu-arbug yiwu-arbug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert formatting changes of unrelated files? Those make it difficult to cherry-pick if needed.

Connor1996
Connor1996 reviewed Mar 19, 2020
View reviewed changes
db/db_impl/db_impl_write.cc Outdated
versions_->SetLastSequence(memtable_write_group.last_sequence);
write_thread_.ExitAsMemTableWriter(&writer, memtable_write_group);
// We only allow leader_thread to finish this WriteGroup because there may
// be another task which is done by thread which is not in this WriteGroup,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// be another task which is done by thread which is not in this WriteGroup,
// be another task which is done by the thread that is not in this WriteGroup,

db/db_impl/db_impl_write.cc Outdated
write_thread_.ExitAsMemTableWriter(&writer, memtable_write_group);
// We only allow leader_thread to finish this WriteGroup because there may
// be another task which is done by thread which is not in this WriteGroup,
// and it would not notify threads in WriteGroup. So we must make someone in
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// and it would not notify threads in WriteGroup. So we must make someone in
// and it would not notify the threads in this WriteGroup. So we must make someone in

@Little-Wallace
fix format
fdf85fa 
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Connor1996
Connor1996 approved these changes Mar 19, 2020
View reviewed changes
Copy link
Member

@Connor1996 Connor1996 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Little-Wallace Little-Wallace merged commit 2938ac5 into tikv:6.4.tikv Mar 19, 2020
@sre-bot sre-bot mentioned this pull request Mar 19, 2020
Closed
yiwu-arbug pushed a commit that referenced this pull request Mar 20, 2020
@Little-Wallace
Improve Multi Batch Write (#154)
3bf273a 
* perform like normal pipelined write

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
@Little-Wallace Little-Wallace deleted the yield_batch branch August 26, 2020 07:16
@Little-Wallace Little-Wallace mentioned this pull request Mar 2, 2022
Closed
@tabokie tabokie mentioned this pull request May 9, 2022
Closed
39 tasks
tabokie pushed a commit to tabokie/rocksdb that referenced this pull request May 12, 2022
@Little-Wallace @tabokie
Improve Multi Batch Write (tikv#154)
c6312bf 
* perform like normal pipelined write

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
tabokie added a commit that referenced this pull request May 12, 2022
@tabokie @Little-Wallace
Implement pipelined commit (#267) (#284)
d3e4138 
* add multibatch write into memtable (#131)

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

Support write multiple WriteBatch into RocksDB together. These WriteBatch will be assigned sequence number in order and pushed into queue. If a thread is waiting for some state, it could steal some job from work queue.

Signed-off-by: tabokie <xy.tao@outlook.com>

* Improve Multi Batch Write (#154)

* perform like normal pipelined write

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: tabokie <xy.tao@outlook.com>

* pass enable_multi_thread_write to BuildDBOptions (#170)

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: tabokie <xy.tao@outlook.com>

* Fix life time of `memtable_write_group` (#171)

* fix life time of memtable_write_group

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: tabokie <xy.tao@outlook.com>

* Commit pipeline when write a WriteBatch for linearizability (#267)

* support commit pipeline

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: tabokie <xy.tao@outlook.com>

* format

Signed-off-by: tabokie <xy.tao@outlook.com>

* remove useless code

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Wallace <bupt2013211450@gmail.com>
tabokie added a commit that referenced this pull request May 25, 2022
@tabokie @Little-Wallace
Implement pipelined commit (#267) (#284)
910417b 
* add multibatch write into memtable (#131)

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

Support write multiple WriteBatch into RocksDB together. These WriteBatch will be assigned sequence number in order and pushed into queue. If a thread is waiting for some state, it could steal some job from work queue.

Signed-off-by: tabokie <xy.tao@outlook.com>

* Improve Multi Batch Write (#154)

* perform like normal pipelined write

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: tabokie <xy.tao@outlook.com>

* pass enable_multi_thread_write to BuildDBOptions (#170)

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: tabokie <xy.tao@outlook.com>

* Fix life time of `memtable_write_group` (#171)

* fix life time of memtable_write_group

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: tabokie <xy.tao@outlook.com>

* Commit pipeline when write a WriteBatch for linearizability (#267)

* support commit pipeline

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Signed-off-by: tabokie <xy.tao@outlook.com>

* format

Signed-off-by: tabokie <xy.tao@outlook.com>

* remove useless code

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Wallace <bupt2013211450@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiwu-arbug yiwu-arbug yiwu-arbug left review comments

@Connor1996 Connor1996 Connor1996 approved these changes

Assignees
No one assigned
Labels
component/rocksdb Component: RocksDB engine
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Little-Wallace @yiwu-arbug @Connor1996