Skip to content

ORC-361: Remove the single thread restriction on the MemoryManagerImpl. #433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
omalley wants to merge 2 commits into from
Closed

ORC-361: Remove the single thread restriction on the MemoryManagerImpl. #433

omalley wants to merge 2 commits into from

Conversation

@omalley
Copy link
Contributor

@omalley omalley commented Oct 4, 2019

The current MemoryManagerImpl assumes that only a single thread is writing ORC files. I've changed the design so that:

  • There is a single MemoryManagerImpl object for all of the threads.
  • The only expensive operations are when new writers are created or closed.
  • There is a new method (checkMemory) that is used to determine if the scale has changed. It uses an atomic long to detect changes without contention.
  • The rows between memory checks is moved into the writer.

@omalley
Copy link
Contributor Author

omalley commented Oct 4, 2019

I simplified the new MemoryManager implementation some more.

@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
6d4b830 
Fixes apache#433

Signed-off-by: Owen O'Malley <omalley@apache.org>
omalley added a commit to omalley/orc that referenced this pull request Oct 7, 2019
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
d1a85e0 
Fixes apache#433

Signed-off-by: Owen O'Malley <omalley@apache.org>
omalley added a commit to omalley/orc that referenced this pull request Oct 8, 2019
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
3c29fef 
Fixes apache#433

Signed-off-by: Owen O'Malley <omalley@apache.org>
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
11db9a6 
Fixes apache#433

Signed-off-by: Owen O'Malley <omalley@apache.org>
@omalley omalley closed this in d657ed4 Oct 8, 2019
omalley added a commit that referenced this pull request Oct 8, 2019
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
20e732f 
Fixes #433

Signed-off-by: Owen O'Malley <omalley@apache.org>
omalley added a commit that referenced this pull request Oct 8, 2019
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
3aed316 
Fixes #433

Signed-off-by: Owen O'Malley <omalley@apache.org>
zratkai pushed a commit to zratkai/orc that referenced this pull request Jun 20, 2024
@omalley @LAToth-W
CDPD-29488 ORC-361: Remove the single thread restriction on the Memor…
b53166f 
…yManagerImpl.

Fixes apache#433

Signed-off-by: Owen O'Malley <omalley@apache.org>
(cherry picked from commit d657ed4)
(cherry picked from commit 54900be)
Change-Id: Ie2afc5fab07bf2a4b1d1562a341665bcfc0b5c78
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@omalley @t3rmin4t0r