Skip to content

Migrate core test to insta part 3 #16978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Oct 23, 2025
Merged

Migrate core test to insta part 3 #16978

merged 15 commits into from
Oct 23, 2025

Conversation

@Chen-Yuan-Lai
Copy link
Contributor

@Chen-Yuan-Lai Chen-Yuan-Lai commented Jul 30, 2025

Which issue does this PR close?

Rationale for this change

Related #16324 #16617

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

No

@github-actions github-actions bot added the core Core DataFusion crate label Jul 30, 2025
@alamb alamb mentioned this pull request Jul 30, 2025
Closed
@Chen-Yuan-Lai Chen-Yuan-Lai force-pushed the migrate_core_test_to_insta_part_3 branch from 16959ba to 5cb1e98 Compare August 3, 2025 17:11
@Chen-Yuan-Lai Chen-Yuan-Lai force-pushed the migrate_core_test_to_insta_part_3 branch from 5cb1e98 to 0e07645 Compare August 4, 2025 09:42
@Chen-Yuan-Lai Chen-Yuan-Lai marked this pull request as ready for review August 4, 2025 14:40
@Chen-Yuan-Lai
Copy link
Contributor Author

Chen-Yuan-Lai commented Aug 4, 2025

@alamb @blaginin, sorry for the long delay, I think the PR is ready to be reviewed :)

@blaginin blaginin self-requested a review August 10, 2025 10:35
alamb
alamb reviewed Aug 10, 2025
View reviewed changes
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this PR @Chen-Yuan-Lai -- unfortuately I am not likely going to be able to find the contiguous time needed to review a PR this large.

I wonder if you would be willing to help make it easier to review, by splitting it up into smaller PRs ?

Also I think the structure of some of these tests is pretty gnarly to begin with (e.g. macros with many parameters). Maybe we can work out a better pattern as part of this work if we are going to rewrite the whole thing anyways

Maybe we can try to port a few tests in one of the files to use iinsta to make sure we are good with the pattern before applying the pattern to the entire thing

For example, a more explicit Test style struct might make the options being tested more efficient.

I know this is a big ask, but I think getting these tests into better shape would really help the project

datafusion/core/tests/physical_optimizer/replace_with_order_preserving_variants.rs Outdated
/// when the flag `prefer_existing_sort` is `true` for bounded cases.
/// * `$PLAN`: The plan to optimize.
/// * `$SOURCE_UNBOUNDED`: Whether the given plan contains an unbounded source.
macro_rules! assert_optimized_in_all_boundedness_situations {
Copy link
Contributor

@alamb alamb Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yeah, I remember this macro -- this is crazy

@Chen-Yuan-Lai
Copy link
Contributor Author

Chen-Yuan-Lai commented Aug 14, 2025

Maybe we can try to port a few tests in one of the files to use iinsta to make sure we are good with the pattern before applying the pattern to the entire thing

For example, a more explicit Test style struct might make the options being tested more efficient.

I know this is a big ask, but I think getting these tests into better shape would really help the project

Sure! I will try it soon

@alamb
Copy link
Contributor

alamb commented Aug 14, 2025

Thank you so much @Chen-Yuan-Lai

@alamb
Copy link
Contributor

alamb commented Sep 29, 2025

Any idea if you will be able to split this PR into smaller pieces @Chen-Yuan-Lai ?

I can try to get an AI agent to do it too if you prefer

@Chen-Yuan-Lai
Copy link
Contributor Author

Chen-Yuan-Lai commented Sep 29, 2025

@alamb Sure, feel free to use an AI agent to split this PR! Sorry for the long delay - I've been short on time. Thanks for offering to help with this!

@blaginin
Merge branch 'main' into migrate_core_test_to_insta_part_3
c733ea5 
# Conflicts:
#	datafusion/core/tests/physical_optimizer/enforce_distribution.rs
@blaginin
Copy link
Collaborator

blaginin commented Sep 29, 2025

Resolved one minor conflict

@blaginin
Copy link
Collaborator

blaginin commented Sep 29, 2025

I can try to get an AI agent to do it too if you prefer

happy to do the split too!

@alamb
Copy link
Contributor

alamb commented Oct 1, 2025

I can try to get an AI agent to do it too if you prefer

happy to do the split too!

that would be great too -- thank you !

@alamb alamb mentioned this pull request Oct 3, 2025
Merged
@alamb
Copy link
Contributor

alamb commented Oct 3, 2025

I ported part of the sort_enforcement tests here:

@blaginin
Merge branch 'main' into migrate_core_test_to_insta_part_3
e6bbd84 
# Conflicts:
#	datafusion/core/tests/physical_optimizer/enforce_sorting.rs
#	datafusion/core/tests/physical_optimizer/partition_statistics.rs
@blaginin
Fix line to remove
4e2de24
@blaginin
Copy link
Collaborator

blaginin commented Oct 6, 2025

i'll chunk this into more prs

@alamb
Copy link
Contributor

alamb commented Oct 6, 2025

i'll chunk this into more prs

😍

@alamb
Copy link
Contributor

alamb commented Oct 6, 2025

I'll prioritize getting them reviewed

@alamb alamb marked this pull request as draft October 6, 2025 18:55
@blaginin blaginin self-assigned this Oct 7, 2025
This was referenced Oct 8, 2025
Merged
Merged
@blaginin
Merge branch 'main' into migrate_core_test_to_insta_part_3
a0b3fea 
# Conflicts:
#	datafusion/core/tests/physical_optimizer/replace_with_order_preserving_variants.rs
@blaginin blaginin mentioned this pull request Oct 20, 2025
Merged
@blaginin blaginin marked this pull request as ready for review October 23, 2025 18:45
@blaginin
Copy link
Collaborator

blaginin commented Oct 23, 2025

🚀

@blaginin blaginin added this pull request to the merge queue Oct 23, 2025
Merged via the queue into apache:main with commit d127973 Oct 23, 2025
28 checks passed
@alamb
Copy link
Contributor

alamb commented Oct 23, 2025

Amazing -- thanks for pushing it over the line @blaginin

tobixdev pushed a commit to tobixdev/datafusion that referenced this pull request Nov 2, 2025
@Chen-Yuan-Lai @alamb
@blaginin @tobixdev
Migrate core test to insta part 3 (apache#16978)
72ce59e 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Part of apache#15791 .

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
Related apache#16324 apache#16617 

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
No

---------

Co-authored-by: Ian Lai <Ian.Lai@senao.com>
Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: blaginin <dmitrii@blaginin.me>
github-merge-queue bot pushed a commit that referenced this pull request Nov 5, 2025
@blaginin @claude @alamb
Complete migrating enforce_distrubution tests to insta (#18185)
acdd263 
- Closes #15791
- Closes #15178 🥳

- Surpasses part of #16978

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
codetyri0n pushed a commit to codetyri0n/datafusion that referenced this pull request Nov 11, 2025
@blaginin @claude
@alamb @codetyri0n
Complete migrating enforce_distrubution tests to insta (apache#18185)
40e15f9 
- Closes apache#15791
- Closes apache#15178 🥳

- Surpasses part of apache#16978

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alamb alamb alamb left review comments

@blaginin blaginin blaginin approved these changes

Assignees

@blaginin blaginin

Labels

core Core DataFusion crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Chen-Yuan-Lai @alamb @blaginin