Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add batch force commit #4

Draft
wants to merge 81 commits into
base: master
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
eeb5be1
Supports batching in ForceCommit API
noob-se7en Jan 12, 2025
5f5a554
nit
noob-se7en Jan 12, 2025
ca5104a
Refactoring
noob-se7en Jan 14, 2025
434e8a3
nit
noob-se7en Jan 14, 2025
504f3c9
nit
noob-se7en Jan 14, 2025
987bb00
nit
noob-se7en Jan 14, 2025
ff25c5f
nit
noob-se7en Jan 14, 2025
e28ff47
nit
noob-se7en Jan 14, 2025
99a7cee
nit
noob-se7en Jan 14, 2025
255bc34
lint
noob-se7en Jan 14, 2025
3a9e41a
nit
noob-se7en Jan 14, 2025
b2eeb85
fixes lint
noob-se7en Jan 14, 2025
1782207
nit
noob-se7en Jan 14, 2025
90db3b8
Merge branch 'master' of github.com:Harnoor7/pinot into add_batch_for…
noob-se7en Jan 15, 2025
fa418b9
refactoring
noob-se7en Jan 15, 2025
470c6eb
refactoring
noob-se7en Jan 15, 2025
8de7bfc
fixes bug
noob-se7en Jan 15, 2025
4f2d4fc
nit
noob-se7en Jan 15, 2025
50af02e
nit
noob-se7en Jan 15, 2025
09d557e
nit
noob-se7en Jan 15, 2025
32b7fd5
nit
noob-se7en Jan 15, 2025
e334983
nit
noob-se7en Jan 15, 2025
1aecc5a
fix_bug
noob-se7en Jan 15, 2025
5be2722
Adds scheduling logic in controller
noob-se7en Jan 15, 2025
153a897
nit
noob-se7en Jan 15, 2025
430127d
fixes lint
noob-se7en Jan 15, 2025
f20948e
fixes bug
noob-se7en Jan 15, 2025
5012b5f
nit
noob-se7en Jan 15, 2025
c2312d2
nit
noob-se7en Jan 15, 2025
49474f5
fix bug
noob-se7en Jan 16, 2025
ab2220f
Updates foceCommit API to handle Pauseless
noob-se7en Jan 16, 2025
ed90f11
updates metadata
noob-se7en Jan 16, 2025
e88aa2a
fixes lint
noob-se7en Jan 16, 2025
8c8d8d3
adds tests
noob-se7en Jan 16, 2025
1be0316
saves 1 zk call
noob-se7en Jan 16, 2025
55fa6e2
Merge branch 'update_forceCommit_status' of github.com:Harnoor7/pinot…
noob-se7en Jan 16, 2025
3297ddd
updates log
noob-se7en Jan 16, 2025
748d0d3
Adds tests
noob-se7en Jan 16, 2025
095acc0
Addresses PR comments
noob-se7en Jan 17, 2025
36360b8
nit
noob-se7en Jan 17, 2025
d3d42ca
Merge branch 'update_forceCommit_status' of github.com:Harnoor7/pinot…
noob-se7en Jan 17, 2025
262bee0
pulls latest changes for pauseless
noob-se7en Jan 17, 2025
68cdc26
adds unit test
noob-se7en Jan 17, 2025
9f833c6
Merge branch 'update_forceCommit_status' of github.com:Harnoor7/pinot…
noob-se7en Jan 17, 2025
f5d68ae
addresses comment
noob-se7en Jan 17, 2025
bffab6d
Merge branch 'master' of github.com:apache/pinot into add_batch_force…
noob-se7en Jan 17, 2025
a1079c2
Addresses Pr comment
noob-se7en Jan 17, 2025
b8a2e7f
Merge branch 'master' of github.com:apache/pinot into update_forceCom…
noob-se7en Jan 17, 2025
5730a06
addresses PR comments
noob-se7en Jan 17, 2025
c8565d6
nit
noob-se7en Jan 17, 2025
165e7ab
Merge branch 'update_forceCommit_status' of github.com:Harnoor7/pinot…
noob-se7en Jan 17, 2025
0cab772
refactoring
noob-se7en Jan 17, 2025
de04824
Addresses PR comments
noob-se7en Jan 17, 2025
b95a2f6
nit
noob-se7en Jan 17, 2025
857dd6a
attempt to fix test
noob-se7en Jan 23, 2025
2f7e5d9
Merge branch 'master' of github.com:apache/pinot into update_forceCom…
noob-se7en Jan 23, 2025
0b64439
nit
noob-se7en Jan 23, 2025
9e3ddad
Attempts to fix test
noob-se7en Jan 23, 2025
01604e9
Merge branch 'update_forceCommit_status' of github.com:Harnoor7/pinot…
noob-se7en Jan 23, 2025
bb84ae2
Merge branch 'master' of github.com:apache/pinot into add_batch_force…
noob-se7en Jan 24, 2025
71f4ee1
Attempts to fix test
noob-se7en Jan 24, 2025
2408d13
attempt to fix test
noob-se7en Jan 24, 2025
ff67929
Addresses PR comments
noob-se7en Jan 28, 2025
80dda07
Adds timeout and interval query parameters in API
noob-se7en Jan 28, 2025
11299f4
nit
noob-se7en Jan 28, 2025
ad7aec0
fixes lint
noob-se7en Jan 29, 2025
7ea5535
Adds unit test
noob-se7en Jan 29, 2025
5ea7c3f
nit
noob-se7en Jan 29, 2025
6907c8f
Addresses PR comments
noob-se7en Jan 30, 2025
2a61ce4
attempts to fix test
noob-se7en Jan 30, 2025
445efbc
speeds up test
noob-se7en Jan 30, 2025
444fc49
Attempts to fix test
noob-se7en Jan 30, 2025
e7ab323
nit
noob-se7en Jan 30, 2025
5c186d8
addresses PR comments and attempts to fixe test
noob-se7en Jan 31, 2025
6110a5a
nit
noob-se7en Jan 31, 2025
f2fbd4b
fixes lint
noob-se7en Jan 31, 2025
1d5af84
nit
noob-se7en Jan 31, 2025
92c771d
nit
noob-se7en Jan 31, 2025
b14e2af
nit
noob-se7en Jan 31, 2025
de52012
Misc fix and cleanup
Jackie-Jiang Jan 31, 2025
5b78c1b
Add missing empty line
Jackie-Jiang Jan 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
nit
  • Loading branch information
noob-se7en committed Jan 31, 2025
commit 6110a5a3afd332458773d70abba4f091090e2c21
Original file line number Diff line number Diff line change
@@ -1930,8 +1930,8 @@ private void waitUntilPrevBatchIsComplete(String tableNameWithType, Set<String>
List<Set<String>> getSegmentBatchList(IdealState idealState, Set<String> targetConsumingSegments,
int batchSize) {
List<Set<String>> segmentBatchList = new ArrayList<>();
if (batchSize == Integer.MAX_VALUE) {
// Add as many segments to batch as possible
if (batchSize >= targetConsumingSegments.size()) {
// All segments can be added in a single batch.
// No need to divide segments in batches.
segmentBatchList.add(targetConsumingSegments);
return segmentBatchList;