Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ success: false
exit_code: 1
----- stdout -----
[CLI_VERSION]
Error: Not enough memory to continue external sort. Consider increasing the memory limit, or decreasing sort_spill_reservation_bytes
Error: Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
caused by
Resources exhausted: Failed to allocate

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ success: false
exit_code: 1
----- stdout -----
[CLI_VERSION]
Error: Not enough memory to continue external sort. Consider increasing the memory limit, or decreasing sort_spill_reservation_bytes
Error: Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
caused by
Resources exhausted: Additional allocation failed for ExternalSorter[0] with top memory consumers (across reservations) as:
Consumer(can spill: bool) consumed XB, peak XB,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ success: false
exit_code: 1
----- stdout -----
[CLI_VERSION]
Error: Not enough memory to continue external sort. Consider increasing the memory limit, or decreasing sort_spill_reservation_bytes
Error: Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
caused by
Resources exhausted: Additional allocation failed for ExternalSorter[0] with top memory consumers (across reservations) as:
Consumer(can spill: bool) consumed XB, peak XB,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,8 @@ async fn automatic_usage_example() -> Result<()> {
println!("✓ Expected memory limit error during data processing:");
println!("Error: {e}");
/* Example error message:
Error: Not enough memory to continue external sort. Consider increasing the memory limit, or decreasing sort_spill_reservation_bytes
Error: Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit',
or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
caused by
Resources exhausted: Additional allocation failed with top memory consumers (across reservations) as:
ExternalSorterMerge[3]#112(can spill: false) consumed 10.0 MB, peak 10.0 MB,
Expand Down
20 changes: 18 additions & 2 deletions datafusion/physical-plan/src/sorts/sort.rs
Original file line number Diff line number Diff line change
Expand Up @@ -819,7 +819,8 @@ impl ExternalSorter {
match e {
DataFusionError::ResourcesExhausted(_) => e.context(
"Not enough memory to continue external sort. \
Consider increasing the memory limit, or decreasing sort_spill_reservation_bytes"
Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', \
or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'."
),
// This is not an OOM error, so just return it as is.
_ => e,
Expand Down Expand Up @@ -1736,6 +1737,21 @@ mod tests {
"Assertion failed: expected a ResourcesExhausted error, but got: {err:?}"
);

// Verify external sorter error message when resource is exhausted
let config_vector = vec![
"datafusion.runtime.memory_limit",
"datafusion.execution.sort_spill_reservation_bytes",
];
let error_message = err.message().to_string();
for config in config_vector.into_iter() {
assert!(
error_message.as_str().contains(config),
"Config: '{}' should be contained in error message: {}.",
config,
error_message.as_str()
);
}

Ok(())
}

Expand All @@ -1756,7 +1772,7 @@ mod tests {

// The input has 200 partitions, each partition has a batch containing 100 rows.
// Each row has a single Utf8 column, the Utf8 string values are roughly 42 bytes.
// The total size of the input is roughly 8.4 KB.
// The total size of the input is roughly 820 KB.
let input = test::scan_partitioned_utf8(200);
let schema = input.schema();

Expand Down