Skip to content

Improve ExternalSorter ResourcesExhausted Error Message#20226

Merged
alamb merged 2 commits intoapache:mainfrom
erenavsarogullari:improve_external_sorter_error_message_when_resource_exhausted
Feb 12, 2026
Merged

Improve ExternalSorter ResourcesExhausted Error Message#20226
alamb merged 2 commits intoapache:mainfrom
erenavsarogullari:improve_external_sorter_error_message_when_resource_exhausted

Conversation

@erenavsarogullari
Copy link
Member

@erenavsarogullari erenavsarogullari commented Feb 9, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

When there is not enough memory to continue external sort, either increasing the memory limit or decreasing sort_spill_reservation_bytes need to be applied. It can be useful to guide the user with clearer error message by highlighting required configs for the consistency because expected settings are as follows:

SET datafusion.runtime.memory_limit = '10G'
SET datafusion.execution.sort_spill_reservation_bytes = 10485760

Current:

Not enough memory to continue external sort. Consider increasing the memory limit, 
or decreasing sort_spill_reservation_bytes

New:

Not enough memory to continue external sort. Consider increasing the memory limit config: 
'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.

Are these changes tested?

A new UT case has been added to cover this use-case and it has been tested locally successfully.

For updated snapshot files:
Following integration tests have been successful by DataFusion Test Pipeline:

test test_cli_top_memory_consumers::case_1 ... ok
test test_cli_top_memory_consumers::case_2 ... ok
test test_cli_top_memory_consumers::case_3 ... ok

Ref: https://github.com/apache/datafusion/actions/runs/21811797863/job/62925363536?pr=20226

Are there any user-facing changes?

Yes, improving legacy ExternalSorter ResourcesExhausted Error Message.

@github-actions github-actions bot added the physical-plan Changes to the physical-plan crate label Feb 9, 2026
@erenavsarogullari erenavsarogullari force-pushed the improve_external_sorter_error_message_when_resource_exhausted branch 5 times, most recently from ba97e75 to 64eaa60 Compare February 9, 2026 03:56
@erenavsarogullari erenavsarogullari force-pushed the improve_external_sorter_error_message_when_resource_exhausted branch 2 times, most recently from 9539d82 to e306fe7 Compare February 10, 2026 17:36
Copy link
Contributor

@kosiew kosiew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erenavsarogullari

Thanks for working on this.

Left some non-blocking comments.

@erenavsarogullari
Copy link
Member Author

Thanks @kosiew and @martin-g for the reviews. All comments have been addressed.

@erenavsarogullari erenavsarogullari force-pushed the improve_external_sorter_error_message_when_resource_exhausted branch from 010cb74 to 080bf49 Compare February 12, 2026 05:29
@erenavsarogullari erenavsarogullari force-pushed the improve_external_sorter_error_message_when_resource_exhausted branch from 080bf49 to ac88bbb Compare February 12, 2026 05:34
@alamb alamb added this pull request to the merge queue Feb 12, 2026
@alamb
Copy link
Contributor

alamb commented Feb 12, 2026

🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve ExternalSorter ResourcesExhausted Error Message

4 participants