Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat]:fast check has column #5328

Merged
merged 10 commits into from
Mar 3, 2023

Conversation

suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Feb 18, 2023

Which issue does this PR close?

part of #5309

Rationale for this change

What changes are included in this PR?

add had_column in DFSchema, and replace field_from_column(l).is_ok() by has_column(l)

Are these changes tested?

yes

Are there any user-facing changes?

no

@suxiaogang223 suxiaogang223 reopened this Feb 18, 2023
@github-actions github-actions bot added logical-expr Logical plan and expressions optimizer Optimizer rules sql SQL Planner labels Feb 18, 2023
@suxiaogang223
Copy link
Contributor Author

It seems that something went wrong with github's servers in the morning, so I reopen this request

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution @suxiaogang223

I am not sure using Result will ever be faster

If you think this will be faster, we can run the benchmark added in #5256

cargo bench --bench sql_planner

datafusion/common/src/dfschema.rs Outdated Show resolved Hide resolved
datafusion/expr/src/logical_plan/builder.rs Outdated Show resolved Hide resolved
datafusion/expr/src/logical_plan/builder.rs Outdated Show resolved Hide resolved
error should not be returned
suxiaogang223 and others added 2 commits February 20, 2023 19:18
Co-authored-by: Daniël Heres <danielheres@gmail.com>
Co-authored-by: Daniël Heres <danielheres@gmail.com>
datafusion/common/src/dfschema.rs Outdated Show resolved Hide resolved
datafusion/common/src/dfschema.rs Outdated Show resolved Hide resolved
datafusion/common/src/dfschema.rs Show resolved Hide resolved
Copy link
Member

@jackwener jackwener left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR carefully, it looks like it has been polished.
Please fix the conflict, thank you @suxiaogang223

Comment on lines 180 to 185
columns.into_iter().try_for_each::<_, Result<()>>(|c| {
if !schema.has_column(&c) {
missing_cols.push(c);
}
});
Ok(())
})?;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

                columns.into_iter().for_each(|c| {
                    if !schema.has_column(&c) {
                        missing_cols.push(c);
                    }
                });

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @suxiaogang223 -- I agree with @jackwener this is looking great

@github-actions github-actions bot removed the sql SQL Planner label Mar 2, 2023
@alamb
Copy link
Contributor

alamb commented Mar 2, 2023

I changed this PR's description to be part of #5309 as I think there are still several calls such as field_with_name that need the same treatment

I also took the liberty of merging from main to resolve the merge conflict in 8cd7f9b and fixing a new instance of is_ok in
1479746 and pushing to this branch

@alamb alamb merged commit be6efbc into apache:main Mar 3, 2023
@ursabot
Copy link

ursabot commented Mar 3, 2023

Benchmark runs are scheduled for baseline = 1455b02 and contender = be6efbc. be6efbc is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@suxiaogang223
Copy link
Contributor Author

I changed this PR's description to be part of #5309 as I think there are still several calls such as field_with_name that need the same treatment

I also took the liberty of merging from main to resolve the merge conflict in 8cd7f9b and fixing a new instance of is_ok in
1479746 and pushing to this branch

Thank you very much, I have been a little slow to reply recently。I'm happy to work about datafusion。😄

@suxiaogang223 suxiaogang223 deleted the features/fast-check-has-column branch March 3, 2023 14:38
@alamb
Copy link
Contributor

alamb commented Mar 3, 2023

Thank you very much, I have been a little slow to reply recently。I'm happy to work about datafusion。😄

we are happy to have the contribution 🤗

@andygrove andygrove added the enhancement New feature or request label Mar 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request logical-expr Logical plan and expressions optimizer Optimizer rules
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants