Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: centralize the logic of choosing worker #311

Merged
merged 3 commits into from
Oct 18, 2022
Merged

refactor: centralize the logic of choosing worker #311

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f8bef27
issue-268
QuintinTao Oct 17, 2022
6f0c51f
Merge branch 'main' into issue-268
archerny Oct 17, 2022
d5b3c0c
refactor: centralize the logic of choosing worker
QuintinTao Oct 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions analytic_engine/src/instance/write_worker.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@ impl WorkerLocal {
worker_num: usize,
) -> Result<()> {
let worker_id = self.data.as_ref().id;
if table_id % worker_num != worker_id {
if choose_worker(table_id, worker_num) != worker_id {
return DataNotLegal {
table: table_name,
worker_id,
Expand Down Expand Up @@ -666,7 +666,7 @@ impl WriteGroup {
///
/// Returns the WriteHandle of the worker
pub fn choose_worker(&self, table_id: TableId) -> WriteHandle {
let index = table_id.as_u64() as usize % self.worker_datas.len();
let index = choose_worker(table_id.as_u64() as usize, self.worker_datas.len());
let worker_data = self.worker_datas[index].clone();

WriteHandle { worker_data }
Expand Down Expand Up @@ -989,7 +989,12 @@ impl WriteWorker {
self.local.data.background_notify.notified().await;
}
}

/// Centralize the logic of choosing worker for table into one place.
/// Choose worker by modulo total worker_num
#[inline]
pub fn choose_worker(table_id: usize, worker_num: usize) -> usize {
table_id % worker_num
}
#[cfg(test)]
pub mod tests {
use common_util::runtime;
Expand Down
4 changes: 2 additions & 2 deletions analytic_engine/src/table/data.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ use snafu::{Backtrace, OptionExt, ResultExt, Snafu};
use table_engine::{engine::CreateTableRequest, table::TableId};

use crate::{
instance::write_worker::{WorkerLocal, WriteHandle},
instance::write_worker::{choose_worker, WorkerLocal, WriteHandle},
memtable::{
factory::{FactoryRef as MemTableFactoryRef, Options as MemTableOptions},
skiplist::factory::SkiplistMemTableFactory,
Expand Down Expand Up @@ -568,7 +568,7 @@ impl TableDataSet {
) -> Option<TableDataRef> {
self.table_datas
.values()
.filter(|t| t.id.as_u64() as usize % worker_num == worker_index)
.filter(|t| choose_worker(t.id.as_u64() as usize, worker_num) == worker_index)
.max_by_key(|t| t.memtable_memory_usage())
.cloned()
}
Expand Down