feat(sort): auto-tune buffer sizing from available memory #8959

mattsu2020 · 2025-10-20T08:45:16Z

Summary

add a direct libc dependency so sort can query sysconf-style memory data on Unix targets
auto-compute the main buffer size using both file size hints and available memory and fall back to sane defaults when either is missing
update chunk growth and external-sort buffers to follow the new heuristic and avoid wasteful allocations

Testing

cargo check --workspace

github-actions · 2025-10-20T09:08:01Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

codspeed-hq · 2025-10-20T09:08:55Z

CodSpeed Performance Report

Merging #8959 will degrade performances by 3.08%

_{Comparing mattsu2020:sort-memory-functions (72201b2) with main (4c68e1e)}

Summary

❌ 1 regression
✅ 105 untouched
⏩ 73 skipped¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	`BASE`	`HEAD`	Change
❌	`sort_long_line[160000]`	1.8 ms	1.9 ms	-3.08%

73 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Added 'sysconf' to the CSpell jargon dictionary to include the system configuration function term, preventing false positives in spell-checking for technical code.

sylvestre · 2025-10-20T09:25:47Z

i added comments in the other PR

github-actions · 2025-10-20T09:46:09Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

Moved buffer size calculation functions (e.g., automatic_buffer_size, file_size_hint) to a new buffer_hint module for better code organization and modularity. Removed the Linux-specific nix dependency as the memory hint functionality is now handled internally without external crates.

Updated the `physical_memory_bytes_unix` function to explicitly cast `pages` and `page_size` to `u128` before multiplication, ensuring safe arithmetic and preventing potential overflow. Also added "libc" dependency to `fuzz/Cargo.lock` to support the changes.

…ory_bytes_unix Reformatted the multi-line `#[cfg(...)]` attribute and reordered imports in `physical_memory_bytes_unix()` for better code clarity and consistency.

…bytes_unix Minor code style cleanup to improve readability and consistency in the buffer hint module.

…y_bytes The explicit return is redundant in Rust, as the last expression in a block is implicitly returned, improving code style and adherence to idiomatic practices.

Corrected "overcommitting" to "overcommit" in the comment for accurate spelling.

mattsu2020 · 2025-10-20T10:51:17Z

https://github.com/nix-rust/nix/blob/e1e630f1b942a222f59c42cee63af16b2a925f15/src/unistd.rs#L3138C1-L3141C40
Based on the nix source code, it depends on libc since it may not be usable in some environments.

github-actions · 2025-10-20T11:05:11Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

src/uu/sort/src/buffer_hint.rs

Added standard copyright and license header to the buffer_hint.rs file in the sort utility to comply with project licensing requirements and ensure proper attribution.

github-actions · 2025-10-21T07:41:34Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

src/uu/sort/src/buffer_hint.rs

Removal of unnecessary variables Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>

src/uu/sort/src/buffer_hint.rs

sylvestre · 2025-10-21T11:27:54Z

src/uu/sort/src/sort.rs

+    #[cfg(target_os = "linux")]
+    {
+        match get_rlimit() {
+            Ok(limit) => limit.saturating_div(4).clamp(32, 256),


please explain what it is doing

The batch size is dynamically adjusted based on the number of file descriptors. Comments and processing have been modified for clarity.

Added a comment in the `physical_memory_bytes` function for non-Unix or Redox targets to clarify why `None` is returned, improving code readability and documenting the absence of a portable API for detecting total physical memory.

src/uu/sort/Cargo.toml

Updated cfg conditions in `physical_memory_bytes` to explicitly handle Linux and Android targets. Refined `physical_memory_bytes_unix` with improved error handling for sysconf calls, ensuring safer memory page and size calculations while removing fallback libc implementation for non-Linux/Android Unix systems.

github-actions · 2025-10-21T11:55:51Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

… batch size Removed the libc crate dependency from Cargo.toml and Cargo.lock to reduce unnecessary dependencies. Refactored default_merge_batch_size() in sort.rs to use named constants (LINUX_BATCH_DIVISOR, LINUX_BATCH_MIN, LINUX_BATCH_MAX) for better code readability and maintainability, while preserving the same dynamic batch size calculation logic based on file descriptor limits.

Reordered the imports in the `use nix::unistd` statement to place `SysconfVar` before `sysconf`, improving code readability and adhering to alphabetical import ordering conventions.

github-actions · 2025-10-21T12:41:40Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-22T05:20:41Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-22T05:53:48Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

sylvestre · 2025-10-22T11:32:35Z

@mattsu2020 sorry but why are you rebasing your pr very often?

mattsu2020 · 2025-10-22T11:36:33Z

@mattsu2020 sorry but why are you rebasing your pr very often?

I wasn't sure if it was keeping up with the latest updates, so I ended up updating it frequently.
I'll be careful to avoid making it difficult to review.

github-actions · 2025-10-22T11:49:40Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

sylvestre · 2025-10-22T12:01:26Z

Yeah, please let the reviewer do that :) thanks

mattsu2020 and others added 2 commits October 20, 2025 17:39

feat(sort): auto-tune buffer sizing from available memory

810d0e6

Merge branch 'uutils:main' into sort-memory-functions

e197e64

mattsu2020 mentioned this pull request Oct 20, 2025

Improve sort buffer sizing heuristics and honor explicit --buffer-size #8833

Merged

docs: add 'sysconf' to jargon wordlist

7160f59

Added 'sysconf' to the CSpell jargon dictionary to include the system configuration function term, preventing false positives in spell-checking for technical code.

mattsu2020 added 6 commits October 20, 2025 19:14

refactor(sort): improve readability of cfg attribute for physical_mem…

e4d46f4

…ory_bytes_unix Reformatted the multi-line `#[cfg(...)]` attribute and reordered imports in `physical_memory_bytes_unix()` for better code clarity and consistency.

style(buffer_hint): remove unnecessary blank line in physical_memory_…

280e127

…bytes_unix Minor code style cleanup to improve readability and consistency in the buffer hint module.

refactor(sort): remove unnecessary return statement in physical_memor…

ddb36bc

…y_bytes The explicit return is redundant in Rust, as the last expression in a block is implicitly returned, improving code style and adherence to idiomatic practices.

fix: correct typo in buffer_hint.rs comment

bfa172e

Corrected "overcommitting" to "overcommit" in the comment for accurate spelling.