Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Third batch of updating sql tests to use assert_batches_eq #1241

Conversation

matthewmturner
Copy link
Contributor

Which issue does this PR close?

Continued work on #743 task 2.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

@github-actions github-actions bot added the datafusion Changes in the datafusion crate label Nov 4, 2021
@houqp houqp added the development-process Related to development process of DataFusion label Nov 5, 2021
@alamb
Copy link
Contributor

alamb commented Nov 8, 2021

Thank you @matthewmturner -- I think the reason this is failing is that the github executors have 2 cores, but your machine had 8, but the physical explain plans depend on the number of cores. I'll fix up the tests (as I have a vested interest in pushing this project along)

@matthewmturner
Copy link
Contributor Author

@alamb noted, thanks!

it might be a silly question, but would you be able to provide color on why the physical explain plan depend on the number of cores?

@alamb
Copy link
Contributor

alamb commented Nov 9, 2021

it might be a silly question, but would you be able to provide color on why the physical explain plan depend on the number of cores?

It is not silly at all @matthewmturner -- the core reason is that the target concurrency defaults to the number of cores -- so when DataFusion goes through the exercise is splitting the input into several output datastreams (partitons), it tries to make just enough output streams to keep all the cores busy.

@alamb alamb merged commit e852b42 into apache:master Nov 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Changes in the datafusion crate development-process Related to development process of DataFusion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants