Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry pick Reduce memory usage of concat (large)utf8 to active_release #411

Merged
merged 2 commits into from
Jun 9, 2021

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Jun 4, 2021

Automatic cherry-pick of e801d4b

Manual cherry-pick of 0cbf85a from #382

* reduce memory needed for concat

* reuse code for str allocation buffer
@alamb
Copy link
Contributor Author

alamb commented Jun 4, 2021

@ritchie46 / @jorgecarleitao should we backport this one? there appears to be an outstanding question: #348 (comment)

@codecov-commenter
Copy link

codecov-commenter commented Jun 4, 2021

Codecov Report

Merging #411 (0cbf85a) into active_release (cd95d7c) will increase coverage by 0.07%.
The diff coverage is 88.84%.

❗ Current head 0cbf85a differs from pull request most recent head d5668ff. Consider uploading reports for the commit d5668ff to get more accurate results
Impacted file tree graph

@@                Coverage Diff                 @@
##           active_release     #411      +/-   ##
==================================================
+ Coverage           82.54%   82.62%   +0.07%     
==================================================
  Files                 162      162              
  Lines               44093    44651     +558     
==================================================
+ Hits                36398    36892     +494     
- Misses               7695     7759      +64     
Impacted Files Coverage Δ
arrow/src/array/array_boolean.rs 90.90% <ø> (ø)
arrow/src/array/array_primitive.rs 92.93% <ø> (ø)
arrow/src/array/array_string.rs 96.05% <ø> (ø)
arrow/src/array/raw_pointer.rs 100.00% <ø> (ø)
arrow/src/buffer/ops.rs 96.96% <ø> (ø)
arrow/src/compute/kernels/aggregate.rs 73.40% <ø> (ø)
arrow/src/compute/kernels/cast_utils.rs 92.50% <ø> (ø)
arrow/src/compute/kernels/comparison.rs 95.84% <ø> (ø)
arrow/src/compute/kernels/length.rs 100.00% <ø> (ø)
arrow/src/csv/writer.rs 82.64% <ø> (ø)
... and 40 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd95d7c...d5668ff. Read the comment docs.

@ritchie46
Copy link
Contributor

ritchie46 commented Jun 5, 2021

@ritchie46 / @jorgecarleitao should we backport this one? there appears to be an outstanding question: #348 (comment)

I addressed the outstanding question in #382. This addresses the concerns for the other kernels. I think that that should be merged in first, and then both can be cherry picked at the same time.

* MutableArrayData::with_capacities

* better pattern matching

* add binary capacities

* add list child data

* add struct capacities

* add panic for dictionary type

* change dictionary capacity enum variant
@alamb
Copy link
Contributor Author

alamb commented Jun 8, 2021

I cherry picked 0cbf85a from #382 as well

@alamb alamb merged commit 5b80035 into active_release Jun 9, 2021
@alamb alamb deleted the cherry_pick_e801d4b9 branch June 9, 2021 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants