Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): decouple spill task from epoch #17539

Merged
merged 57 commits into from
Jul 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
4fa032a
refactor(storage): remove sync finish event
wenym1 Jun 18, 2024
09950fc
syncing data upload tasks partial succeed
wenym1 Jun 18, 2024
85b2ded
extract poll_syncing_task
wenym1 Jun 18, 2024
892dd29
remove uploader event
wenym1 Jun 18, 2024
2874313
fix compile
wenym1 Jun 19, 2024
7bd9c79
Merge branch 'yiming/remove-sync-finish-event' into yiming/dag-uploader
wenym1 Jun 24, 2024
82a4191
feat(storage): decouple upload task finish order from epoch
wenym1 Jun 25, 2024
f2fa4de
Merge branch 'main' into yiming/remove-sync-finish-event
wenym1 Jun 26, 2024
af22875
Merge branch 'yiming/remove-sync-finish-event' into yiming/dag-uploader
wenym1 Jun 26, 2024
eaac3f9
move to new file
wenym1 Jun 26, 2024
00e9d00
Merge branch 'main' into yiming/remove-sync-finish-event
wenym1 Jun 26, 2024
b0aad6d
Merge branch 'yiming/remove-sync-finish-event' into yiming/dag-uploader
wenym1 Jun 26, 2024
372dfc4
Merge branch 'main' into yiming/dag-uploader
wenym1 Jun 26, 2024
4136e88
track spill table id
wenym1 Jun 27, 2024
5aef226
add check and refactor
wenym1 Jun 28, 2024
e51e186
temp save
wenym1 Jun 28, 2024
d234bbe
refactor(storage): remove global epoch in uploader
wenym1 Jun 28, 2024
37cc33d
Merge branch 'yiming/uploader-no-global-epoch' into yiming/dag-uploader
wenym1 Jun 28, 2024
30a808e
fix test and address comment
wenym1 Jul 2, 2024
bf2cc9c
feat(storage): decouple spill task from epoch
wenym1 Jul 2, 2024
8d6badd
Merge branch 'main' into yiming/uploader-no-global-epoch
wenym1 Jul 8, 2024
ffb9906
fix panic
wenym1 Jul 8, 2024
3808d55
fix
wenym1 Jul 9, 2024
abf2ff8
Merge branch 'yiming/uploader-no-global-epoch' into yiming/dag-uploader
wenym1 Jul 9, 2024
d02775a
print error table id
wenym1 Jul 9, 2024
62b5cd4
ack sync with subset
wenym1 Jul 9, 2024
aa562f8
Merge branch 'yiming/uploader-no-global-epoch' into yiming/dag-uploader
wenym1 Jul 9, 2024
1010dbf
fix clippy
wenym1 Jul 9, 2024
0d30c07
Merge branch 'yiming/uploader-no-global-epoch' into yiming/dag-uploader
wenym1 Jul 9, 2024
1a3f2e7
Merge branch 'main' into yiming/uploader-no-global-epoch
wenym1 Jul 9, 2024
ddab2d3
Merge branch 'yiming/uploader-no-global-epoch' into yiming/dag-uploader
wenym1 Jul 9, 2024
4ace8e4
feat(storage): pass epoch and table id before barrier
wenym1 Jul 9, 2024
72b97a4
Merge branch 'main' into yiming/dag-uploader
wenym1 Jul 9, 2024
403f207
Merge branch 'yiming/dag-uploader' into yiming/decouple-spill-from-epoch
wenym1 Jul 9, 2024
f55b2cc
fmt
wenym1 Jul 9, 2024
6503a4c
loose sanity check
wenym1 Jul 9, 2024
7134288
fix
wenym1 Jul 9, 2024
492c4bc
fix ut
wenym1 Jul 10, 2024
7545c15
fix test
wenym1 Jul 10, 2024
c3674d8
fix test and add panic log
wenym1 Jul 10, 2024
13e28ee
Merge branch 'main' into yiming/pre-start-sync-epoch
wenym1 Jul 11, 2024
5a0e7a8
add log
wenym1 Jul 11, 2024
3c14411
send barrier to all CN
wenym1 Jul 11, 2024
d445196
remove non-empty check
wenym1 Jul 11, 2024
267d0ee
allow initialize at non-checkpoint barrier
wenym1 Jul 11, 2024
cbcdac3
check stopped epoch when not contain epoch
wenym1 Jul 11, 2024
3d47848
skip unconnected empty node
wenym1 Jul 11, 2024
dbf3d3d
Merge branch 'main' into yiming/decouple-spill-from-epoch
wenym1 Jul 11, 2024
62d5335
Merge branch 'yiming/pre-start-sync-epoch' into yiming/decouple-spill…
wenym1 Jul 12, 2024
c69aa7d
spill by epoch table ids
wenym1 Jul 12, 2024
aee91df
fix typo
wenym1 Jul 12, 2024
9c70968
Merge branch 'main' into yiming/pre-start-sync-epoch
wenym1 Jul 13, 2024
3204e70
Merge branch 'yiming/pre-start-sync-epoch' into yiming/decouple-spill…
wenym1 Jul 13, 2024
1bbe7af
fix test
wenym1 Jul 13, 2024
8d7d63b
refactor test_utils
wenym1 Jul 13, 2024
a5614ff
add test
wenym1 Jul 13, 2024
47b7e02
fmt
wenym1 Jul 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -936,7 +936,7 @@ mod tests {
use tokio::sync::oneshot;

use crate::hummock::event_handler::refiller::CacheRefiller;
use crate::hummock::event_handler::uploader::tests::{gen_imm, TEST_TABLE_ID};
use crate::hummock::event_handler::uploader::test_utils::{gen_imm, TEST_TABLE_ID};
use crate::hummock::event_handler::uploader::UploadTaskOutput;
use crate::hummock::event_handler::{HummockEvent, HummockEventHandler, HummockVersionUpdate};
use crate::hummock::iterator::test_utils::mock_sstable_store;
Expand Down
Loading
Loading