Skip to content

Commit 6b1a3bc

Browse files
committed
Optimize read_to_end.
This patch makes `read_to_end` use Vec's memory-growth pattern rather than using a custom pattern. This has some interesting effects: - If memory is reserved up front, `read_to_end` can be faster, as it starts reading at the buffer size, rather than always starting at 32 bytes. This speeds up file reading by 2x in one of my use cases. - It can reduce the number of syscalls when reading large files. Previously, `read_to_end` would settle into a sequence of 8192-byte reads. With this patch, the read size follows Vec's allocation pattern. For example, on a 16MiB file, it can do 21 read syscalls instead of 2057. In simple benchmarks of large files though, overall speed is still dominated by the actual I/O. - A downside is that Read implementations that don't implement `initializer()` may see increased memory zeroing overhead. I benchmarked this on a variety of data sizes, with and without preallocated buffers. Most benchmarks see no difference, but reading a small/medium file with a pre-allocated buffer is faster.
1 parent 481b42b commit 6b1a3bc

File tree

1 file changed

+3
-6
lines changed

1 file changed

+3
-6
lines changed

Diff for: src/libstd/io/mod.rs

+3-6
Original file line numberDiff line numberDiff line change
@@ -366,16 +366,13 @@ fn append_to_string<F>(buf: &mut String, f: F) -> Result<usize>
366366
fn read_to_end<R: Read + ?Sized>(r: &mut R, buf: &mut Vec<u8>) -> Result<usize> {
367367
let start_len = buf.len();
368368
let mut g = Guard { len: buf.len(), buf: buf };
369-
let mut new_write_size = 16;
370369
let ret;
371370
loop {
372371
if g.len == g.buf.len() {
373-
if new_write_size < DEFAULT_BUF_SIZE {
374-
new_write_size *= 2;
375-
}
376372
unsafe {
377-
g.buf.reserve(new_write_size);
378-
g.buf.set_len(g.len + new_write_size);
373+
g.buf.reserve(32);
374+
let capacity = g.buf.capacity();
375+
g.buf.set_len(capacity);
379376
r.initializer().initialize(&mut g.buf[g.len..]);
380377
}
381378
}

0 commit comments

Comments
 (0)