Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

Improve memory allocation on native row to column operator #746

Closed
zhouyuan opened this issue Mar 1, 2022 · 0 comments
Closed

Improve memory allocation on native row to column operator #746

zhouyuan opened this issue Mar 1, 2022 · 0 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@zhouyuan
Copy link
Collaborator

zhouyuan commented Mar 1, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently R2C will allocate a 128MB(hard-coded) chunk for the following operation.
https://github.com/oap-project/gazelle_plugin/blob/master/native-sql-engine/core/src/main/scala/com/intel/oap/execution/ArrowRowToColumnarExec.scala#L103

Describe the solution you'd like
This should be improved by estimate/calculate the real memory usage

  • for fixed width type, it's easy to calculate the detailed memory size
  • for string, we should estimate the size and do re-alloc in case the memory is not enough

Describe alternatives you've considered
N/A

Additional context
N/A

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants