Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Empty Column Projection in ParquetRecordBatchReader #1537

Closed
tustvold opened this issue Apr 11, 2022 · 0 comments · Fixed by #1560
Closed

Support Empty Column Projection in ParquetRecordBatchReader #1537

tustvold opened this issue Apr 11, 2022 · 0 comments · Fixed by #1560
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

If an empty array is passed to ArrowReader::get_record_reader_by_columns it returns an error.

Describe the solution you'd like

It would be advantageous to support an empty projection within ParquetRecordBatchReader, and instead use file metadata to return the requisite number of rows. This will need the ability to create a RecordBatch with no columns and a non-zero row count #1536

Describe alternatives you've considered

We could not add this functionality

Additional context

This would simplify the implementation of Hive based partitioning, amongst other things - apache/datafusion#2000
@tustvold tustvold added the enhancement Any new improvement worthy of a entry in the changelog label Apr 11, 2022
This was referenced Apr 11, 2022
Merged
Merged
@alamb alamb added the parquet Changes to the parquet crate label Apr 15, 2022
@alamb alamb changed the title Support Empty Column Projection in ParquetRecordBatchReader Support Empty Column Projection in ParquetRecordBatchReader Apr 15, 2022
tustvold added a commit to tustvold/arrow-datafusion that referenced this issue Apr 20, 2022
@tustvold
Remove workaround for apache/arrow-rs#1537
8ca40e5
@tustvold tustvold mentioned this issue Apr 20, 2022
Merged
tustvold added a commit to apache/datafusion that referenced this issue Apr 21, 2022
@tustvold
Remove Parquet Empty Projection Workaround (#2289)
110f350 
* Remove workaround for apache/arrow-rs#1537

* Handle batches with no columns in SchemaAdapter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support empty projection in ParquetRecordBatchReader tustvold/arrow-rs
2 participants
@alamb @tustvold