Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support RecordBatch with zero columns but non zero row count #1536

Closed
tustvold opened this issue Apr 11, 2022 · 0 comments · Fixed by #1552
Closed

Support RecordBatch with zero columns but non zero row count #1536

tustvold opened this issue Apr 11, 2022 · 0 comments · Fixed by #1552
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Certain queries may only be interested in the number of rows, and not any actual column data. In theory this could be handled as a projection pushdown that selects no columns, but currently there is no way to create a RecordBatch with no columns and a non-zero row count and so this doesn't work.

Describe the solution you'd like

I would like RecordBatch to store the row_count as a struct member, and allow this to be specified in RecordBatchOptions

Describe alternatives you've considered

We could not make this change

Additional context

This problem can also arise when using Hive style partitioning, where the underlying file data may not contain all columns - apache/datafusion#2000
@tustvold tustvold added the enhancement Any new improvement worthy of a entry in the changelog label Apr 11, 2022
@tustvold tustvold mentioned this issue Apr 11, 2022
Closed
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2022
@tustvold
Support empty RecordBatch (apache#1536)
cdbbeb2
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2022
@tustvold
Support empty RecordBatch (apache#1536)
d1dd90d
@tustvold tustvold mentioned this issue Apr 12, 2022
Merged
alamb pushed a commit that referenced this issue Apr 12, 2022
@tustvold
Create RecordBatch With Non-Zero Row Count But No Columns (#1536) (#1552
c9549bb 
)

* Support empty RecordBatch (#1536)

* Placate clippy

* Review feedback

* Fix doc

* Fix create_record_batch_slice_empty_batch test
@alamb alamb changed the title Create RecordBatch With Non-Zero Row Count But No Columns Support RecordBatch with No Columns but Non-Zero Row Count Apr 15, 2022
@alamb alamb added the arrow Changes to the arrow crate label Apr 15, 2022
@alamb alamb changed the title Support RecordBatch with No Columns but Non-Zero Row Count Support RecordBatch with zero columns but non zero row count Apr 15, 2022
@tustvold tustvold mentioned this issue May 6, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Milestone
No milestone
2 participants
@alamb @tustvold