Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support scan empty projection #7920

Merged
merged 7 commits into from
Oct 25, 2023
Merged

support scan empty projection #7920

merged 7 commits into from
Oct 25, 2023

Conversation

haohuaijin
Copy link
Contributor

@haohuaijin haohuaijin commented Oct 24, 2023 •
edited by Dandandan
Loading

Which issue does this PR close?

Closes #3214

Rationale for this change

When I want to do the query like

select random() from test;

I encounter Arrow error: External error: Arrow error: Invalid argument error: must either specify a row count or at least one column. I found that the error was occurring because, after reading a RecordBatch, we will to reproject the RecordBatch use the partition column, and it did not support an empty projection. see below.
https://github.com/apache/arrow-datafusion/blob/ba50a8b178eece7e79b100d0b73bdc9d6d3ec6d5/datafusion/core/src/datasource/physical_plan/file_scan_config.rs#L342

What changes are included in this PR?

  • change file_scan_config.rs and cross_join.rs to support empty RecordBatch
  • change push_down_projection to support scan table with no projection

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Oct 24, 2023
@haohuaijin
Copy link
Contributor Author

very strange, avro.slt did not report an error when I ran cargo test locally.

@alamb
Copy link
Contributor

alamb commented Oct 24, 2023

very strange, avro.slt did not report an error when I ran cargo test locally.

I think that is because avro is not a default feature

You probably need to run cargo test --features=avro to run it locally

alamb
alamb approved these changes Oct 24, 2023
View reviewed changes
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me. Thank you.

cc @Dandandan

Dandandan
Dandandan reviewed Oct 24, 2023
View reviewed changes
datafusion/optimizer/src/push_down_projection.rs
@@ -149,10 +149,6 @@ impl OptimizerRule for PushDownProjection {
{
let mut used_columns: HashSet<Column> = HashSet::new();
if projection_is_empty {
let field = find_small_field(scan.projected_schema.fields()).ok_or(
Copy link
Contributor

@Dandandan Dandandan Oct 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can remove the [LogicalPlan::Values] case as well (https://github.com/apache/arrow-datafusion/pull/7920/files#diff-96a0620f2e490af5e30f17cfb0b6a0070fd99656dd4957966693894c816abe0bL166-L175)?

Copy link
Contributor Author

@haohuaijin haohuaijin Oct 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we can remove it.already fix, thanks.

@Dandandan
Copy link
Contributor

FYI @ch-sc

Dandandan
Dandandan reviewed Oct 24, 2023
View reviewed changes
datafusion/physical-plan/src/joins/cross_join.rs
@@ -347,13 +348,14 @@ fn build_batch(
})
.collect::<Result<Vec<_>>>()?;

RecordBatch::try_new(
RecordBatch::try_new_with_options(
Copy link
Contributor

@Dandandan Dandandan Oct 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we have more operators with empty inputs that use RecordBatch::try_new that should move to use RecordBatch::try_new_with_options (but are not covered in tests).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have thought about this too, but I didn't come up with any more tests😥.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is fine to merge as is and see if we get any regressions (hopefully not), those should be easy to fix in a similar manner.

Dandandan
Dandandan reviewed Oct 24, 2023
View reviewed changes
datafusion/sqllogictest/test_files/subquery.slt
@@ -695,7 +695,7 @@ logical_plan
Projection: __scalar_sq_1.COUNT(*) AS b
--SubqueryAlias: __scalar_sq_1
----Aggregate: groupBy=[[]], aggr=[[COUNT(UInt8(1)) AS COUNT(*)]]
------TableScan: t1 projection=[t1_id]
------TableScan: t1 projection=[]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@Dandandan Dandandan merged commit af2cda9 into apache:main Oct 25, 2023
@Dandandan
Copy link
Contributor

Thank you @haohuaijin !

@Dandandan Dandandan mentioned this pull request Oct 25, 2023
Closed
@haohuaijin haohuaijin deleted the empty_projection branch October 26, 2023 01:43
@matthewgapp matthewgapp mentioned this pull request Jan 11, 2024
Closed
@jiacai2050 jiacai2050 mentioned this pull request Feb 22, 2024
Merged
jiacai2050 added a commit to apache/horaedb that referenced this pull request Feb 23, 2024
@tanruixiang @jiacai2050
deps: bump datafusion (#1445)
62ffffc 
## Rationale
Close #1461 

## Detailed Changes
Bump datafusion to
https://github.com/CeresDB/arrow-datafusion/commits/e21b03154, which is
version 33.

Some important breaking changes:
- apache/datafusion#7920
- apache/datafusion#9109

## Test Plan
CI

---------

Co-authored-by: jiacai2050 <dev@liujiacai.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Dandandan Dandandan Dandandan approved these changes

@Weijun-H Weijun-H Weijun-H requested changes

@alamb alamb alamb approved these changes

Assignees
No one assigned
Labels
core Core DataFusion crate optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Don't scan first column on empty projection
4 participants
@haohuaijin @alamb @Dandandan @Weijun-H