Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inefficiency in CoolModel class #78

Open
hugy718 opened this issue Jul 27, 2022 · 2 comments
Open

Inefficiency in CoolModel class #78

hugy718 opened this issue Jul 27, 2022 · 2 comments
Assignees
Labels
enhancement New feature or request logic-issue problems about the main logic

Comments

@hugy718
Copy link
Collaborator

hugy718 commented Jul 27, 2022

The CoolModel class is currently serve to keep various readstores in memory without loading the the actual chunk data. It looks like a caching layer, then there are a few issue to address.

  • memory management: there should be a fixed capacity to control how many cube/cohort/... are loaded and when loading and processing mark how much space they take, free when no space
  • methods are marked with locked with time-consuming IO inside, we should only lock only the steps that modify the cache internal structure
  • there is a field called currentCube, so it can serve the cohort of a cube at one time. This is not necessary. Multiple workers should be able to concurrently request for cohorts under different cubes with only necessary synchronization
@hugy718 hugy718 added enhancement New feature or request logic-issue problems about the main logic labels Jul 27, 2022
@hugy718
Copy link
Collaborator Author

hugy718 commented Jul 27, 2022

One more thing. If a cube is updated and it has been cached inside CoolModel, then the old version is directly returned when requested. Needs additional checking to load the latest version.

@KimballCai
Copy link
Contributor

@liuchangshiye please solve this problem together with @NLGithubWP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request logic-issue problems about the main logic
Projects
None yet
Development

No branches or pull requests

4 participants