MADV_DONTNEED causing lots of minor page faults #31885
Comments
keeperofdakeys
changed the title
keeperofdakeys
changed the title
|
It is the jemalloc that calls madvise. To avoid it you can switch to system #![feature(alloc_system)]
extern crate alloc_system;The root cause, are costly allocations (deallocations) coming from BufReader. Though, in my opinion the default buffer size used by BufReader is quite |
|
Thanks a lot, using the system allocator does resolve the problem. Limiting the size of BufReader also sounds like a good idea, especially for my case, where most of the files I'm reading are quite small. Other measurements by perf suggest that nearly 50% of the running time was just page faults. If there are many other Rust programs out there doing such frequent big allocations, they may see similar hits. Is this jemalloc behaviour optimal? |
|
There used to be a comment next to the default buffer size constant but it seems to have been lost at some point: https://github.com/rust-lang/rust/pull/9091/files#diff-b131eeef531ad098b32f49695a031008R60. The rationale was never that great when I added it, and I'd be fine dropping it to 4k or 8k which I think is somewhat more standard? |
|
It's also worth noting that there's no reason to use BufReader at all in a context like this: https://github.com/keeperofdakeys/Process-Query/blob/master/src/procrs/pid/mod.rs#L51, since |
|
I'd be happy to make a PR to change the default buffer size then. Should a warning also be added to the API docs regarding jemalloc and large buffer sizes? As for my code, using read_to_end directly on the File is causing reads of a single byte. I just did some measurements, and this seems to have a detrimental impact. No BufReader, 50ms, ~280000 syscalls (lots of read(1) calls) So at least for my case, a properly configured BufReader makes a huge difference. |
|
I noticed a PR to update jemalloc (#31889), and I think this will explain why the beta version did not show the problem. Nightly and stable are using jemalloc 3, the beta is still using jemalloc 4. If jemalloc 4 handles big buffers better (not calling madvise a lot), would it still be worth changing the default buffer size? |
|
I'm not sure where read requests with a 1 byte buffer could be coming from - this is the implementation of read_to_end: https://github.com/rust-lang/rust/blob/master/src/libstd/io/mod.rs#L340-L375. It'll start with 16 byte reads and then ramp up exponentially to 64k reads. |
|
I am using the .bytes iterator in one place, I might have mistook one file being read with read(1) for all files being read with it. That's also why I started using BufReader originally. |
|
Ah yeah, .bytes() will read one byte at a time. |
The 64k capacity was picked by me a couple of years ago in the initial implementation of buffered IO adaptors: https://github.com/rust-lang/rust/pull/9091/files#diff-b131eeef531ad098b32f49695a031008R62. 64K was picked for symmetry with libuv, which we no longer use. 64K is *way* larger than the default size of any other language that I can find. C, C++, and Java default to 8K, and Go defaults to 4K. There have been a variety of issues filed relating to this such as rust-lang#31885. Closes rust-lang#31885
…ichton The 64k capacity was picked by me a couple of years ago in the initial implementation of buffered IO adaptors: https://github.com/rust-lang/rust/pull/9091/files#diff-b131eeef531ad098b32f49695a031008R62. 64K was picked for symmetry with libuv, which we no longer use. 64K is *way* larger than the default size of any other language that I can find. C, C++, and Java default to 8K, and Go defaults to 4K. There have been a variety of issues filed relating to this such as rust-lang#31885. Closes rust-lang#31885
Drop the default buffer size to 8K The 64k capacity was picked by me a couple of years ago in the initial implementation of buffered IO adaptors: https://github.com/rust-lang/rust/pull/9091/files#diff-b131eeef531ad098b32f49695a031008R62. 64K was picked for symmetry with libuv, which we no longer use. 64K is *way* larger than the default size of any other language that I can find. C, C++, and Java default to 8K, and Go defaults to 4K. There have been a variety of issues filed relating to this such as #31885. Closes #31885
I've got a program that is experiencing lots minor of page faults (>12000 in >100 ms runtime), doubling its runtime. I suspect the issue is because of excessive madvise(..., ..., MADV_DONTNEED) calls. The following excerpts from perf stat, strace output, and perf trace output show the evidence. The behaviour occurs on a nightly from one week ago, and the current stable. However I can't replicate this on an older beta I have (rustc 1.7.0-beta.3), where no such madvise calls, or minor page faults occur.
I can't find any madvise calls in the rust source code, so I'm a bit stumped in how to further investigate this issue.
perf stat output:
strace output:
perf trace output:
The text was updated successfully, but these errors were encountered: