Skip to content

Commit

Permalink
Reuse result vector in Alpha reader (facebookincubator#9226)
Browse files Browse the repository at this point in the history
Summary:
X-link: facebookincubator/nimble#26

Pull Request resolved: facebookincubator#9226

Optimize both CPU and memory utilization by reusing result and
intermediate vector memory.

Reviewed By: oerling

Differential Revision: D55219745

fbshipit-source-id: 0ec3e735903dd95ae4f08e439347319785890492
  • Loading branch information
Yuhta authored and Joe-Abraham committed Jun 7, 2024
1 parent 0227a15 commit c15975d
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 0 deletions.
1 change: 1 addition & 0 deletions velox/dwio/common/Reader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ VectorPtr RowReader::projectColumns(
bits::forEachSetBit(
passed.data(), 0, input->size(), [&](auto i) { rawIndices[j++] = i; });
for (auto& child : children) {
child->disableMemo();
child = BaseVector::wrapInDictionary(
nullptr, indices, size, std::move(child));
}
Expand Down
8 changes: 8 additions & 0 deletions velox/vector/ComplexVector.h
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,10 @@ class RowVector : public BaseVector {
/// Note : If the child is null, then it will stay null after the resize.
void resize(vector_size_t newSize, bool setNotNull = true) override;

VectorPtr& rawVectorForBatchReader() {
return rawVectorForBatchReader_;
}

private:
vector_size_t childSize() const {
bool allConstant = false;
Expand Down Expand Up @@ -270,6 +274,10 @@ class RowVector : public BaseVector {
// loadedVector is called, and reset to false when updateContainsLazyNotLoaded
// is called (i.e. some children are likely updated to lazy).
mutable bool childrenLoaded_ = false;

// For some non-selective reader, we need to keep the original vector that is
// unprojected and unfilterd, and reuse its memory.
VectorPtr rawVectorForBatchReader_;
};

// Common parent class for ARRAY and MAP vectors. Contains 'offsets' and
Expand Down

0 comments on commit c15975d

Please sign in to comment.